The public relies on journalists to learn about and share academic research. Public knowledge can be undermined, however, when academics try to influence what research journalists cover or limit the “acceptable debate” about an issue.
This influence can be achieved through “citation cartels,” where sympathetic researchers cite and reference one another and ignore or dismiss the high-quality research of others that reach different conclusions. Citation cartels belittle research they disagree with, rather than refute it.
Citation cartels can advance researchers’ careers substantially: Cartel members receive more media exposure and get more academic citations. Universities consider citation numbers as evidence of research productivity and influence, which affects hiring and promotion decisions.
A prime example of this process is media coverage of standardized testing.
I have long complained that education journalism tends to be slanted against standardized testing in books criticizing “testing culture” and American meritocracy, and in PBS documentaries. Journalist Anya Kamenetz recently published a book that thoroughly covered the opinions and evidence against standardized testing, speaking with dozens of experts and advocates. Yet, she spoke with not a single expert who could have provided counter-evidence or references. Not one.
She could have found counterpoints in the Education Writers Association’s SourceSearch database, which gives journalists a directory of experts to interview. I have also published source lists for the “other side” of the standardized testing issue in books and online.
In fairness, it is true that the public relations personnel at private test development firms often hedge their language with journalists for fear of alienating critics who could be their customers. Most testing companies also impose gag orders on their staff. And, naturally, many of the most knowledgeable testing advocates work inside testing firms. But, journalists on other beats often consult “unnamed sources” inside organizations for information. I have never witnessed it in stories on educational testing research and policy.
The results of the journalist’s one-sided sourcing are one-sided policy recommendations and a one-sided understanding of the issue by the public.
Perhaps the author was biased on the issue, but I discern from her network of contacts that she was instead stuck inside a bubble. She talked to certain prominent experts who steered her to other sources within their respective cartels. To get the other side of the story would have required going outside the usual networks and approaching non-favored experts that would take more time to discover.
The time constraints faced by journalists mean they can’t do their own review of the relevant research; it can take years to really know a research literature. So, journalists must rely on experts in the field. When recommending other sources, strategic scholars will identify someone within the cartel, regardless of their expertise in the topic. When recommending other research, they will mention only that conducted within their cartel.
The consequence of journalistic reliance on citation cartels is obvious—a misinformed public. When journalists repeat a dismissive review, they further the interest of the cartel, not the public.
Another example of cartel activity is when the journalist Matt Barnum recently argued for the elimination of high school graduation examinations. Ever since the federal government harmonized graduation rate statistics in 2010, high schools have increased their graduation rates, sometimes by lowering standards. One might surmise that high school “exit exams” could check that degradation. According to education research citation cartels, however, high school exit exams only hurt low-achieving students—they fail the exam and then drop out of school, drastically lowering their earning potential—and provide no countervailing benefits.
Barnum’s preferred sources can say exit exams only hurt low-achieving students because they ignore or dismiss most contrary evidence. The research they dismiss shows that exit exams can improve academic achievement overall—students learn more.
Other research they ignore shows that unqualified high school graduates tend to drop out of college, often after accumulating substantial student loan debt. Still other research reveals a stark difference in international test performance between countries with high school exit exams and those without. The public needs to know the consequences of removing exit exams, which requires journalists to consider all the relevant research.
When cartels don’t ignore other research, they attack it with a dismissive review. An academic literature review summarizes previous research conducted on a particular topic. With a dismissive literature review, a researcher assures the public that either no research exists on the topic, or what research exists is so poorly done that it is not worth citing. Dismissive reviews can be useful, but often, and perhaps usually, they are not.
Dismissive reviews may be the quickest route to claim expertise for someone who lacks genuine expertise. It says, “I looked and there’s nothing there.” Dismissive reviews may also help an academic obtain research grants because funders are more likely to fund new projects where they perceive a need.The consequence of journalistic reliance on citation cartels is obvious—a misinformed public.
Typically, prominent dismissive reviewers, whose claim to fame is their allegedly superior research skill, provide no evidence or analysis whatsoever for their research literature claims. I’ve cataloged hundreds of instances. Some common phrases simply dismiss previous work (e.g., “paucity of research” or “this is the first study”) and others denigrate it (e.g., “prior research is simply anecdotal” or “this is the first rigorous study using sophisticated analytical methods”).
Literature reviews can be trustworthy; meta-analysts, those researchers who specialize in research literature summaries, typically provide thorough descriptions of where and how they look for source material. Meta-analyst claims are easily verified because another researcher can retrace their steps, but dismissive reviews from cartels are accepted on faith. Meta-analysts expand society’s knowledge by citing all the relevant research even when they disagree with it. That approach encourages debate. Scholars in citation cartels, however, reference strategically, thus boosting their own professional profiles and those of their friends and colleagues.
Perhaps some journalists simply refuse to believe that their sources can be anything but sincere and independent. But journalists, as the focus of much cartel behavior, could do much to deter them.
First, they could refrain from printing dismissive reviews. Journalists do not know if and when a dismissive review is true, and they do not have the time to check entire research literature to verify them. Second, they could investigate citation cartel operations. Lastly, they could expand the sources they talk to beyond their comfortable bubbles. Journalists need to contact a better variety of experts and academics to get the full picture that the public deserves.
Foundation and government research funders could help break up the cartels, too. They could demand that applicants for grant funds back up their dismissive review claims in the same manner a meta-analyst would, by specifying the details of their literature search. And they could deny further funding to applicants who previously submitted false dismissive reviews. Finally, if foundations were to make their grant applications public, other researchers could see the research literature review claims made by applicants, and rebut the erroneous ones.
Academic integrity and public knowledge can be threatened by ideological cartels on the left and the right. Education journalists need to hold themselves to a high standard to fight off cartel influence, and funders need to avoid funding them. In that way, truth is the final arbiter of “acceptable debate,” and the public can make informed decisions about American education.
Richard P. Phelps is founder of the Nonpartisan Education Group and editor of its Review.