The Vanderbilt/WashU report on the problem of ideological scholarship in the humanities and social sciences continues to generate both applause and condemnation. The report’s strongest summary conclusion runs as follows:
“Every field we have studied shows some signs of [...] a deterioration in scholarly standards fueled by the substitution of political criteria for properly scholarly criteria in the assessment of research and a more general repudiation of long-standing ideals of rigor and objectivity.”
Among critics, one particular line of attack has been simply to reject this claim as either right-wing hysteria or at least an irresponsibly unsubstantiated assertion.
For instance, from academic commentator John K. Wilson in an Inside Higher Ed essay:
“There is virtually no evidence offered in the report for their sweeping conclusions about the entirety of the humanities. Instead, the report declares that it has a series of secret ‘internal reports’ on various fields that prove what it claims.
No sound scholar publishes their conclusions and declares that all the evidence to support those claims is found in secret ‘internal reports.’ Whether you agree with the conclusions or not, it is a bad argument when you refuse to show your evidence.”
A particularly scornful dismissal from sociologist Julian Go (unlinked as Go’s X account has since closed):
And from public policy professor John Holbein:
These are not the only examples, but they suffice to illustrate the point.
Now, I do not think these critics are all of a kind, but rather fall on a range. At one end are those who seek to defend the sort of research critiqued in the report, not on the merits, but rather through a strategy of “deny, deflect, disparage.” Those at the other end simply want our conversations about these matters to be more empirically grounded. To the former group, there isn’t much to say that will be constructive. To the latter, I reply: OK, fair enough.
Admittedly, for those of us who have long been observing, documenting, and decrying ideological scholarship, these dismissals can feel like a bit of gaslighting. Higher ed reformers have published books, articles, and edited volumes, blogged, tweeted, held conferences, and founded organizations, all dedicated to calling out and addressing this very problem. Given all these receipts, it is tempting to dismiss the dismissals of ideological scholarship as an act of willful ignorance.
Yet if we’re honest, we must admit that none of these efforts add up to a point estimate, a hypothesis test, or any other concrete empirical statement of the sort the report’s critics demand.
If a skeptic asks us just how serious or extensive the problem of ideological scholarship is, really, then we can regale them with tales of slanted sloppy scholarship and outlandish activist absurdities for days on end. But suppose they interrupt us and demand, “OK, OK, I get the gist. It sounds bad. But what’s your evidence?” We might respond in some puzzlement: “What are you talking about? I’ve been sharing evidence this entire time!” “No,” the skeptic counters, “you’ve been giving me anecdotes. I want something hard. What percent of scholarly claims are compromised by ideology? What proportion of published research falls below appropriate scholarly standards?” At this we might hesitate, stammer a little, and admit we can’t produce a figure. To which our skeptic, shaking their head ruefully, might sigh, “You really had me going there. Come find me when you have a spreadsheet in hand.”
The trouble is, there is no straightforward answer we can give to the skeptic’s questions. Even if measuring ideological scholarship were a merely empirical task, it would be a formidable one (though with the ever-expanding capabilities of LLM’s, perhaps not out of reach). A Scopus search for original articles in the social sciences and arts/humanities turns up 76,128 results from 2024 alone, or just over 200 publications per day.1 That would be a lot of material to sift through even if we knew just what we were looking for.
But of course, we don’t know just what we’re looking for. There is no standard operationalization across disciplines for a) what constitutes proper scholarly criteria, and b) what counts as pernicious ideological influence. These are essentially contested matters that differ across subfields, often require insider familiarity to assess, operate as moving targets, and are informed at least as much by tacit norms or judgments as they are established and articulated principles.2 Part of the logic behind academic disciplines and peer review is that communally organized human judgment and expertise are necessary because criteria of evaluation are too subtle and rarified to be codified. In some cases, authors do us a favor by outright saying things like (as quoted in the report) “all truth is subjective” and “science itself is not neutral but a sociopolitical and ideological endeavor,” making the evaluation of ideological scholarship easy.3 But in most cases ideological scholarship is likely to be covert, unintentional, or a matter of degree.
In short, we do not have a generally accepted empirical standard for determining whether any given publication is ideological, let alone the entirety of the corpus that makes up “the social sciences and the humanities.”
That is not to say establishing such a standard hasn’t been tried.
Criteria for Ideological Compromise
As the academic heterodoxy movement of the past decade has gained steam, a few scholars have offered tests or procedures for identifying when scholarship has been compromised by ideology. I’ll cover three of them here, though I expect there are others with which I am not familiar.
The first comes from psychologists and heterodox stalwarts Lee Jussim and Nathan Honeycutt. In a 2023 article in the journal Psychology Learning and Teaching, they provide a series of tests to determine when a research study is operating as propaganda rather than science. Their criteria run as follows:
Test 0: Does the study vindicate some political narrative?
[If not, it can’t very well be political propaganda.]
Test 1: Did they misinterpret or misrepresent their results in ways that unjustifiably advance a particular politicized narrative?
Test 2: Do the authors systematically ignore papers and studies inconsistent with their ideology-affirming conclusions?
Test 3: Did they leap to ideology-affirming conclusions based on weak data?
A study that “passes” 0 and at least one of the remaining tests might be judged ideological. A study that passes multiple tests would be particularly egregious.
More recently in Theory and Society, Callie Burt has proposed the adoption of Ideological Epistemic Analysis (IEA) to identify ideological compromise within research agendas. She does not offer a set of concrete steps like Jussim and Honeycutt, but rather emphasizes several sites in the research process where such compromises are likely to occur:
Research questions asked (and unasked)
The framing of the research
Methods (especially measures)
Interpretation
Canonization
Still more recently, again in Theory and Society, Ashley Rubin specifies some criteria for inappropriate politicization of research practice, as follows:
Avoiding questions that might contradict our political commitments
Setting out to prove a conclusion
Ignoring contradictory evidence
Cherry-picking evidence
Declining to publish sound studies that contradict our political commitments
Ignoring plausible but politically disagreeable interpretations for our findings
Using moralized or partisan language in peer-reviewed studies where more neutral language is available
Explicit endorsements in peer-reviewed work of politicians, political parties, or policies4
Any of these practices can be deemed a violation of scientific scholarly standards in favor of ideological considerations—that is, ideological scholarship.
It is noteworthy that each of these formulations is quite recent, and two of them are from the same journal. This concentration raises the (hopeful, in my view) prospect that the effort to demonstrate ideological slant in scholarship is turning into a research agenda in its own right, perhaps operating similarly to metascience, or its kooky older cousin, science and technology studies.
Nevertheless, I have my doubts that we are anywhere near being able to produce a point estimate or hypothesis test. Even if the above criteria can provide us some general sense of what we’re looking for, any large-scale analysis still has to overcome a major barrier: because the logics of each subfield are distinct and not fully commensurable, any evaluation of ideological distortion requires both considerable insider expertise and an irreducible element of judgment.
The Problem of Standardizing Standards
When introducing my students to the often disorienting experience of reading scholarly literature, I suggest they approach it as if they were walking into the middle of a long-standing conversation. They should not expect to recognize every reference or pick up on every shade of meaning. I guide them as much as possible, providing context, unpacking central principles, and talking through difficult passages. Still, it is a substantial achievement if a student can dimly grasp the main thread. There is a reason that graduate training is a multi-year process. Mastering some meaningful portion of a scholarly conversation enough to participate in and shape it is a major undertaking.
Given both the long-term expansion and hyperspecialization of the higher education sector, we have branched off into innumerable separate conversations, increasingly disconnected from each other, and each with their own distinctive subject matter, internal logic, methods, norms, and sets of standards. These represent not just specialized bodies of knowledge but skills, habits, and sensibilities developed over a long period of time.
This cacophony of fragmented conversations serves as a major impediment to any effort to assess the extent of ideological influence within any given subfield, let alone across them. Even if we can apply the criteria listed above to each subfield in theory, that application will not be one-size-fits-all.
Consider the Jussim/Honeycutt tests. Let us grant that Test 0, at least, is relatively straightforward—even an informed nonspecialist should be able to tell if a given research claim vindicates a partisan narrative. Beyond that, the waters are murky. In order to judge whether a researcher has misinterpreted or misrepresented their results (Test 1), we would need a strong grasp of their evidence, and of its appropriate standards of evaluation in the subfield. To identify whether they ignored literature inconvenient to their political goals (Test 2), we need some mastery of that literature ourselves. To know whether they leapt to conclusions based on weak data (Test 3), we need knowledge of what constitutes strong data within that discipline. This sort of analysis requires both substantial preexisting expertise and a heavy investment of time and intellectual energy. In any given subfield, there are likely to be relatively few people qualified to apply these tests, and fewer still who are motivated to do so.
A further barrier to establishing ideological compromise is that the criteria listed above will typically be contestable. For instance, applying Burt’s IEA, a critic might be able to argue that the authors of a given body of work asked the wrong questions, developed slanted framings, used inappropriate methods or biased measures, applied unsound interpretations to their results, and rested their argument on work that was undeservedly canonized. Except in truly egregious cases, however, the original author and their allies will likely be able to mount plausible defenses on some or all of these points (that their question was the right one, that their framing was defensible, etc.). There will be a component of judgment as to whether a given study or literature has, in fact, failed to meet scholarly standards or shown evidence of ideological compromise.
In combination, these concerns raise major difficulties for efforts to identify ideological scholarship. If one must be an expert (likely one of a small number) in a subfield to conduct an adequate analysis of ideological compromise, and if claims of ideological scholarship are contested by other relevant experts, then the most an outside observer will be able to glean is that there is a controversy. Establishing scholarship as ideological in an authoritative, generally accepted way is a high bar.5
That is not to say such critiques are not worthwhile or cannot be done well. The authors mentioned above have put the kind of analysis they propose into practice, in my view to great effect. Rubin exhaustively documented the replacement of theory with normative claims in law and punishment studies. Burt’s IEA highlighted ideological distortions propping up problematic ideas about the nature of sex and gender in transgender research. Jussim has levied a sustained critique of claims about stereotype threat and self-fulfilling prophecies, which he shows to be overblown or distorted. In related efforts, John Iceland and Eric Silver have demonstrated the narrow and tautological nature of research on systemic racism. In some of my own work, I have argued that claims about “Christian nationalism” as a widespread and radical ideology fail many of the criteria laid out above.
Such efforts represent valuable contributions in both demonstrating ideological compromise of high-profile research agendas and, just as importantly, unpacking how that ideological influence operates. They worked because their authors had deep familiarity with the relevant literatures, the nature of the concepts, the structure of the arguments, and the links between theory and data. They were thus able to articulate based on close reading and careful argumentation precisely where ideological distortions had slipped in and driven the research off course.
But the factors that make these critiques effective are precisely the reasons they are difficult to scale. After all, these major expenditures of expertise, intellectual labor, and professional risk work to highlight ideological compromise in only a few narrow slices of the humanities and social sciences. Suppose we deliver a stack of such careful, extensively sourced reviews to our hypothetical skeptic. “You asked for receipts? Here you go!” we announce triumphantly. At this, the skeptic scoffs. “I ask you for evidence and you bring me more anecdotes. These reviews cover, what, a couple hundred studies? There were some 1500 new publications in the humanities and social sciences just last week!”
Even with some criteria and effective models for what we want to achieve, it is still unclear how we can reach a point estimate or a hypothesis test.
Toward an Ideological Audit
I promised in the introduction to The Next Turtle Down that some of the ideas would be half-baked. It’s time I made good on that promise. I firmly believe that ideological scholarship is a real phenomenon and a major problem. I don’t know how to demonstrate that as an authoritative empirical conclusion that would render it difficult for good-faith and bad-faith actors alike to demand: “Where’s the evidence?”
Fortunately for everyone involved, I’m not the only person thinking about this. The missing internal reports informing the Vanderbilt/WashU Statement that angered John K. Wilson are expected to be released in the coming weeks at the individual authors’ discretion. This will not only offer some grounds on which to evaluate the document’s central claim, but hopefully offer some idea of what systematic evidence for ideological compromise can look like.
In the meantime, I’ll conclude here with a few thoughts on how we might advance the larger project of an ideological audit of academic scholarship.
A Mosaic of Case Studies
In the section above, I suggested that although deep dives into particular problematic research agendas can be enlightening, they’re difficult to scale, in part because they’re more likely to be professionally punished than rewarded.
But suppose that were to change. Through the efforts of Jussim, Honeycutt, Burt, Rubin, and others, we have some criteria and models for how to conduct an assessment of ideological scholarship. Publication outlets may be scarce, but Theory and Society, at least, actively seeks out this sort of analysis. If the Vanderbilt/WashU report is taken seriously in larger circles, perhaps other journals will follow suit. If such ideological assessment became a viable path to professional advancement undertaken by more insiders, we might accumulate a larger body of evidence regarding the nature and prevalence of ideological compromise.
Given the partial incommensurability of subfield evaluations, this still would not give us a point estimate. But it might provide a kind of mosaic of case studies that add up to a larger, more coherent picture of our situation.
AI-Assisted Auditing
As discussed above, the barriers to conducting an ideological audit on a given body of literature include not just scarcity of expertise or possibility of professional risks, but demands on time and energy. To critique the literature, one must know the literature. Given likely pushback or accusations of cherry-picking, an effective critique should be an extensive one. Many who might possess the expertise and see the value in such a project may nonetheless lack the bandwidth to carry it out.
The rapidly advancing capabilities of LLM’s might change that calculation. As with many things AI, I’m a bit nervous about this because it would be easy to abuse with sloppy analysis or baked-in confirmation bias. I would like to think an effective audit still requires a high degree of human knowledge of a subfield and careful reasoning to assess the extent and nature of any ideological compromise. Not everything can be offloaded to the machines.
Still, once we’ve developed a sense of what the problems are and a process for picking them out, and crafted the prompts necessary to automate that process, LLM’s might facilitate much more thorough audits in less time.
By way of analogy, qualitative researchers have a concept of saturation wherein additional interviews or analyses are no longer producing new insights. In the process I’m envisioning, scholars performing an ideological audit could go through the literature until reaching saturation in their assessment of ideological compromise. At that point, an LLM might perform that assessment on the remaining literature, flagging any discrepancies that warrant further attention, but otherwise rendering the burden much more manageable.
Triangulating Evidence
Due to the difficulties of providing close, direct analysis of scholarship itself, claims about ideological slant in the academy typically rely on more surface or proxy approaches. For instance, James Manzi recently used LLM’s to analyze 600,000 social scientific abstracts between 1960 and 2024 to evaluate prevalence, direction, and trends in ideological content. Similarly, David Rozado has examined the use of particular ideology-laden terminology (e.g., white supremacy, Islamophobia) in abstracts over time. These broader but shallower treatments may not be sufficient to demonstrate a departure from scholarly standards, but can provide some corroboration for more direct evidence. Other efforts include the tracking of faculty partisan ratios, or Shields and Muravchik’s widely shared study showing one-sided treatment of controversial issues in university course syllabi.
None of these forms of evidence definitively establish the validity of the Vanderbilt/WashU report’s claim. They do not add up to anything like an ideological compromise coefficient. But taken together, they might still sway the open-minded skeptic.
Going Where the Ball Is
The logic of an audit is that while it may be random or routine in some cases, attention is best focused on the areas with the greatest risk and/or consequences. A financial audit, for instance, occurs either where transactions look the most suspicious or where the most money is involved.
Similarly, we do not need to examine 80,000+ studies per year to determine whether scholarly standards in the humanities and social sciences are compromised by ideological criteria. I would instead propose three considerations for allocating such efforts:
Scholarly prestige: Ideological scholarship is more harmful to the scholarly enterprise when it appears in higher-ranking venues, gathers higher citation counts, or is funded by more impressive grants.
Public recognition: Ideological scholarship is more harmful when it gets more media attention or exerts influence on policy.
Politicized subject matter: Regardless of prestige and recognition, scholarship is more likely to be perniciously ideological when its subject matter is adjacent to partisan or cultural conflict. Research agendas on ostensible psychological differences between liberals and conservatives, for instance, should receive a high degree of scrutiny.
Though already implicit in this essay, it is worth affirming an element of the report’s recommendations: an ideological audit of the kind I am describing can and should only be carried out by those who are scholarly insiders themselves. The effort should be first and foremost internal. In part, this is a matter of qualification—only other scholars possess the ability to perform such an assessment.6 In part, it is a matter of purpose. The goal of this sort of project is to restore scholarly standards of objectivity and rigor, and insiders are the people best suited to appreciate what that looks like and why it matters.
At the same time, it is incumbent upon those scholars to be both rigorous in their assessment and clear in their conclusions. This means being willing to state outright when they think the work of their colleagues or the claims prevalent within their own disciplines have fallen short. This is not for purposes of shaming or score-settling, but rather because a norm that is lauded in the abstract but ignored in particular cases is not a norm at all. A durable course correction requires a full reckoning with where we have gone off track.
This is certainly a substantial undercount given Scopus’ limited coverage and inconsistency of search terms. But it is sufficient to establish the point that there is a lot of published scholarship in the humanities and social sciences.
On the role of tacit norms, practices, or assumptions, see Kuhn’s The Structure of Scientific Revolutions and Polanyi’s Personal Knowledge.
Though to be precise, such propositions are not in themselves evidence of a lack of rigor — they might, in principle, be advanced rigorously. But they do represent a devaluation of rigor.
She grants that in some cases research findings may have a direct bearing on policy, but suggests scholarly commentary in this case should bear on likely effects rather than offering outright endorsements.
As it should be. Though I would like ideological scholarship to be recognized as a serious problem, the fact remains that such an attribution ultimately amounts to an accusation of wrongdoing, which in a spirit of charity should require strong warrant.
As the report notes, there could be exceptions to this, as in a hypothetical where the field of astronomy drifts into astrology and thus the relevant experts are themselves compromised. Even here, though, assessment or intervention is best carried out by in adjacent and less compromised fields.
Some thoughts: First, this is fundamentally a question of priors and null hypothesis. If any institution asks you to trust them, the default answer is usually "no". If a company presents you a bunch of fancy numbers, we expect those to be biased towards whatever is convenient for the company. If any non-western state does, we frequently expect those to be outright doctored. My green friends will automatically distrust anything from any field they don't like, be it the automobile industry or legacy energy providers, unless they're pro-green, of course. If certain scientists want to influence my opinion, it's on them to win my trust. I don't need to prove anything.
This is even more true for state-financed scientists, which is the majority. If you want public money for your experiments, you need to have the public's trust. That's how it ought to work. To be clear, I can understand the frustration with some in my view common yet foolish attitudes, such as distrust of GMOs, or vaccines, or open markets and a long list of other things. But a democracy is not stable if large sums of money get spent over and over again in a way the public does not approve off. There WILL be a correction coming, and the more you fight it, the harder it will get. Trust in science is in many countries at an all-time-low, and in my view rightfully so.
Which leads me to my second point: In many cases, people really don't even give a shit anymore to pretend to be unbiased. If scientists openly say that investigating certain things is unethical because we could get the wrong results, science is dead. I personally know a head of neurology who had a wave of strokes and other issues during the beginning of the covid vaccinations, the majority of which where older people who had just taken the vaccine. He didn't report a single one of those as possibly connected to the vaccine, despite himself thinking that they probably were, on the simple logic that older people were still better-off on net with the vaccination, and that reporting this might keep some percentage from vaccinating themselves. In the short term he is right, but in the long term this obliterates all trust. And when I talk with colleagues on this, a decent percentage will straight up admit they'd do the same. I also think these vaccination were with overwhelming likelihood on-net positive, but how am I supposed to argue that against covid-sceptic friends if even I know the numbers on this are probably bullshit?
And Julian Go is of course correct here; The social sciences never debunked anything. That also doesn't mean the racists were correct, however.
The actual debunkings came from a much more mature fields: Biology. It's easy to show that the large, fundamental and categorical differences between races that early 20th century race scientists wanted to prove are implausible and simply do not fit the data. Social scientists of course would like to claim that glory for themselves, but they simply had nothing to do with it. From the start, they've always been more activist than scientist.
And this leads me to the final, fundamental problem right now in science. Because there is not just one science. There are mature fields with a good feedback mechanism to reality, such as physics, which have their struggles, but there is relatively little worry of bias. And there are young fields with an almost nonexistent feedback mechanism, such as the social sciences.
Even inside a field, you can have substantial differences; For example, I work in human (medical) genetics. When I have a dystonia patient with a family history and I find a rare mutation in a gene known to be connected to movement disorders, and based on this, we treat him with a drug and he is healed, then there is little argument about the relevance of this mutation. The coincidence of his treatment working would be so astronomical as to be implausible. On the other hand, if I find that obese people happen to have some variant in a higher proportion, than this might be directly connected to obesity, or indirectly, or just spuriously, or not at all. There is no feedback, so I can't properly test it.
In social sciences, everything is like this. And the social scientist will usually openly admit to all sorts of things that shouldn't be investigated, which scientists should be barred for daring to get wrong results, and with which convictions they went into the field in the first place. Not rarely, they'll literally call themselves "activist-scholars". And then act surprised when people don't trust their results.
Fantastic article, and something I've thought about for quite a while since I run into these skeptics of ideological scholarship all the time.
I also think there's a deeper irony here though, because the very people demanding evidence for these supposedly "sweeping conclusions" are generally the same ones who accept social justice-y conclusions hook, line, and sinker at the lowest levels of logical and methodological rigor.