Lab Studies Predict Real Behavior
False Assumption: Controlled lab experiments in social psychology reliably demonstrate effects that occur and matter in the real world.
Summaries Written by FARAgent (AI) on February 11, 2026 · Pending Verification
For decades, social psychology sold a simple promise: put people in a controlled setting, isolate a variable, and you can see the machinery of real behavior at work. That view had respectable reasons behind it. Laboratory experiments offered random assignment, cleaner causal inference, and a way to test ideas that would be messy or impossible to study in everyday life. Researchers and journalists treated famous findings, on self-control, priming, conformity, prejudice, relationship dynamics, as more than curiosities; they were often presented as proofs of how people actually behave outside the lab. A reasonable person could look at that record, and at the prestige of the journals publishing it, and conclude that the lab was a small stage on which the larger world could be glimpsed.
Over time, though, evidence began to complicate that confidence. Even within the field, figures such as William McGuire had long argued that experiments were often only "proofs of concept" under narrow conditions, not direct forecasts of everyday life. Then came the replication crisis: large replication efforts in the 2010s found that many published psychology effects were weaker than advertised or failed to reappear at all. Researchers such as Eli Finkel and Michael Inzlicht, among others, began asking whether elegant lab effects, including some in self-control and social behavior, were too fragile, too small, or too dependent on student samples and artificial tasks to tell us much about weight loss, voting, marriage, or workplace conduct. Field results and naturalistic studies often produced smaller effects than the lab had suggested.
The assumption has not disappeared, because the supporting case has not disappeared either. Lab studies still do something valuable: they can identify mechanisms, rule out some causal stories, and generate findings that later hold up in field experiments or large observational work. The current debate is narrower and less triumphant than it once was. Growing evidence suggests that many celebrated lab effects do not travel well on their own, while an influential minority of researchers argue that the problem is not experimentation itself but overclaiming, weak measurement, and a publishing culture that rewarded neat stories over durable ones.
Status: A small but growing and influential group of experts think this was false
People Involved
- Michael Inzlicht, a social psychology professor at the University of Toronto, published a 2015 blog post titled 'Reckoning with the Past' in which he publicly critiqued the field's replicability and validity issues after years of growing doubts. The post, hosted by Simine Vazire who had earlier hosted his self-critical guest post, detailed how he had come to view many lab effects as unreliable predictors of real behavior and described the personal toll of his shift toward skepticism. He lost close friendships and mentors, faced professional isolation, and experienced burnout that left him dreaming of early retirement. His writings gained coverage in The Atlantic, The Globe and Mail, and Undark Magazine, moving him from the mainstream to the replication-crisis camp. [1]
- Eli Finkel, a psychology professor at Northwestern University, reached similar conclusions years before Inzlicht and warned that replicability alone was a low bar compared with validity concerns for real-world relevance. He argued that many lab findings failed to bridge the gap between what could happen under controlled conditions and what actually did happen amid everyday complexities. His early skepticism helped set the stage for broader questioning within the field. [1]
- Walter Mischel, a key figure in social psychology, had long championed situationism over stable personality traits through influential work such as the marshmallow test. His research contributed to the widespread dismissal of personality psychology as a serious pillar of the discipline, nearly driving it to extinction in some quarters. Yet during the replication crisis many of his own findings came under scrutiny and failed to hold up consistently. [1]
- Craig A. Anderson, a researcher who promoted the idea of high external validity, published meta-analyses in 1999 comparing lab and field studies across 38 topics and reported substantial agreement between the two. His work was widely cited as evidence that controlled experiments reliably translated to real-world outcomes. Later replications challenged the breadth of his conclusions. [4]
- Greg Mitchell, a social psychologist at the University of Virginia School of Law, conducted a larger-scale replication of Anderson's approach examining 217 comparisons and found major variations, including 30 outright reversals between lab and field results. His analysis showed that correspondence was often poor in social psychology and gender-difference studies performed especially badly under realistic conditions. He concluded that external validity had to be assessed case by case rather than assumed. [4]
- Richard H. Thaler, an economist who later won the 2017 Nobel Prize, catalogued empirical anomalies that contradicted rational choice theory by observing real behavior such as the endowment effect. His early papers faced rejection and seminar hostility, yet he persisted in integrating psychological insights from Daniel Kahneman, Amos Tversky, and Herbert Simon's bounded rationality. Over time his work shifted the field toward behavioral economics. [6]
▶ Supporting Quotes (20)
“Ten years ago, I sat at my computer and hit “publish” on a blog post that would permanently alter my relationship with my field. The post, titled Reckoning with the Past, wasn't my first critique of social psychology.”— Ten Years a Skeptic
“Maybe I should have heeded the wise words of my friend Eli Finkel, psychology professor at Northwestern University, who came to similar conclusions, years before I did.”— Ten Years a Skeptic
“Which is more than I can say about the work of Walter Mischel, one of the chief architects of personality's near-death experience. For all his brilliance, Mischel's own research didn't escape the replication crisis unscathed.”— Ten Years a Skeptic
“The post, titled Reckoning with the Past, wasn't my first critique of social psychology... But this post was different. This was my coming-out moment as a full-blown skeptic in social psychology.”— Ten Years a Skeptic
“For all his brilliance, Mischel's own research didn't escape the replication crisis unscathed. There's probably a Greek tragedy in there somewhere: the scholar who tried to bury personality psychology saw his own legacy become diminished under the weight of methodological reform.”— Ten Years a Skeptic
“Maybe I should have heeded the wise words of my friend Eli Finkel, psychology professor at Northwestern University, who came to similar conclusions, years before I did.”— Ten Years a Skeptic
“Ten years ago, I sat at my computer and hit “publish” on a blog post that would permanently alter my relationship with my field.”— Ten Years a Skeptic
“Maybe I should have heeded the wise words of my friend Eli Finkel, psychology professor at Northwestern University, who came to similar conclusions, years before I did.”— Ten Years a Skeptic
“For all his brilliance, Mischel's own research didn't escape the replication crisis unscathed.”— Ten Years a Skeptic
“I had already dipped my toes in those waters with an earlier piece called Check Yourself Before You Wreck Yourself (a guest post on Simine Vazire’s blog)”— Ten Years a Skeptic
“in 1999, Craig A. Anderson and his colleagues compared laboratory and field research on 38 topics in 21 meta-analyses (or analyses of numerous other studies), and found a lot of agreement between the results of the two.”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“Greg Mitchell, a social psychologist at the University of Virginia School of Law, wanted to know if these findings hold up in a bigger sample—and whether there were differences among different kinds of psychological research.”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“his role in exposing and making sense of empirical anomalies in orthodox economics”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“exploring the economic implications of Kahneman’s work with the late Amos Tversky (Tversky and Kahneman 1974) on the predictable effects of bias-inducing heuristics on the quality of decision-making. He also pioneered the use within economics of their Prospect Theory approach”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“His contributions build on work by two previous Nobel Laureates, Herbert Simon (in 1978)”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“"The people at the rally are not a random or representative sample of the electorate," says Lynn Vavreck... "These are strategic and well-planned events. This isn't just happening."”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“"Some of the crowd size is people who come because they love Bernie and they want to hear this message and they keep doing it and they follow him around just like [fans of] the Grateful Dead or Phish."”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“Adam Grant, the superstar organizational psychologist who was one of the main characters in Part 1, is also worried about undue skepticism of psychological findings.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“Nosek is a careful, measured speaker when he talks with the media, and yet here’s what he said to Nature’s Tom Chivers about social priming in 2019: “I don’t know a replicable finding. It’s not that there isn’t one, but I can’t name it.””— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“James Manzi is a doctoral student in Sociology at the University of Oxford. This study analyzes approximately 600,000 English-language social science abstracts published between 1960 and 2024”— The Ideological Trajectory of the Social Sciences
Organizations Involved
Social psychology as a discipline had long promoted the view that controlled lab experiments reliably demonstrated effects that mattered outside the laboratory, an assumption that contributed to the replication crisis and eventually toppled some of the field's most prominent hierarchies and elder statesmen. Departments at Michigan, Ohio State, Stanford, Waterloo, and Yale lost their earlier unquestioned authority as failures accumulated. The crisis also revived interest in personality psychology, which proved more replicable than many situational lab effects. [1]
The Association for Psychological Science published Mitchell's large-scale replication study in its journal Perspectives on Psychological Science, which helped spread awareness of the limits of lab external validity across subfields. The study examined far more comparisons than earlier work and documented frequent mismatches between lab and field outcomes. Its appearance in a respected outlet gave the critique institutional visibility. [4]
Top journals such as the Quarterly Journal of Economics published one in five articles with less than a 50 percent chance of replication despite rigorous selection processes, showing that even high-prestige outlets enforced no strict quality standards on replicability. Funding agencies continued to support such work because grants were tied to publication records rather than later verification. [5]
The Journal of Economic Behavior and Organization provided an early platform for Thaler's seminal paper on consumer choice anomalies, allowing behavioral critiques of rational choice theory to reach academic audiences. Similar journals in psychology, including Group Processes & Intergroup Relations, published unreplicated priming studies without adequate checks for questionable research practices. [6][12]
Social science disciplines including economics and political science maintained left-of-center ideological means while remaining less extreme than fields such as sociology or gender studies, yet still shaped research output at scale through hiring, peer review, and topic selection. An analysis of 600,000 abstracts from 1960 to 2024 found every discipline leaned left on a fixed U.S. ideological spectrum. [13]
▶ Supporting Quotes (7)
“I started seeing much of our cherished research as, frankly, worthless. And my skepticism, some would say cynicism, has only deepened over the years”— Ten Years a Skeptic
“in a new paper published in Perspectives on Psychological Science, a journal of the Association for Psychological Science”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“Even the crème de la crème of economics journals barely manage a ⅔ expected replication rate. 1 in 5 articles in QJE scores below 50%, and this is a journal that accepts just 1 out of every 30 submissions.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“For this stuff to be financed, generated, published, and eventually rewarded requires the complicity of funding agencies, journal editors, peer reviewers, and hiring/tenure committees. Given the current structure of the machine, ultimately the funding agencies are to blame.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“Thaler’s seminal (1980) article in the very first issue of the Journal of Economic Behavior and Organization (JEBO)”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“published in 2008 in Group Processes & Intergroup Relations”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“Disciplines concerned with public policy (“policy-proximal disciplines), such as economics and political science, tend to be less fanatically leftist than disciplines concerned more with feelings, such as psychology, sociology, and gender studies (“policy-distal disciplines”).”— The Ideological Trajectory of the Social Sciences
The Foundation
For decades the assumption that controlled lab experiments in social psychology reliably demonstrate effects that occur and matter in the real world rested on several lines of reasoning that struck many thoughtful observers as persuasive at the time. Clean experimental controls eliminated extraneous variables and produced clear causal inferences, as in studies showing that self-control manipulations reduced snack consumption in the lab; these results were taken as credible evidence for real-world applications such as weight loss programs. William McGuire's perspectivism framed lab experiments as proofs of concept that showed what could happen under specific conditions, which many interpreted as support for broader relevance even if the effects might be modest. Meta-analyses by researchers such as Craig Anderson aggregated dozens of studies across 38 topics and reported substantial agreement between lab and field findings, reinforcing the belief that results generalized reliably. Rational choice theory in economics appeared robust because Milton Friedman's as-if methodology emphasized predictive success over realistic assumptions, yet anomalies such as the endowment effect, where people demanded more to sell an item than they would pay to buy an identical one, suggested the model missed important behavioral regularities. Even the visible size of political rallies seemed to signal broad enthusiasm and predict electoral success, as commentators noted the contrast between Bernie Sanders' large crowds and smaller events for other candidates. [1][4][6][7]
A growing body of evidence has increasingly questioned these claims without yet overturning them in the mainstream. Inzlicht and Marina Milyavskaya tracked participants in the real world over weeks and found that lab measures of self-control did not predict success on everyday goals such as weight loss amid competing life demands. The Open Science Collaboration's large replication project examined 97 psychology studies with significant effects and found that only 36 percent replicated successfully, with average effect sizes roughly half the original. Many Labs projects reported even lower replication rates around 25 percent for social psychology, and Brian Nosek stated that no known social priming finding had proven reliably replicable. Mitchell's expanded analysis of 217 lab-field comparisons uncovered 30 reversals and showed poor correspondence especially in social psychology and gender-difference research, suggesting that external validity could not be assumed but required case-by-case evaluation. An analysis of 600,000 social science abstracts using large language models revealed that roughly 90 percent leaned left on a fixed ideological spectrum, with every discipline showing a left-of-center mean each year; prominent papers such as one on hegemonic masculinity scored strongly left while others landed in moderate territory. Studies claiming policy implications often relied on small samples, barely significant p-values, correlational data, or arbitrary statistical specifications that prediction markets easily flagged as low in replicability. [1][4][5][9][12][13]
▶ Supporting Quotes (23)
“Say we run an experiment showing that when people exercise self-control in a lab, they eat fewer cookies or chips or whatever tempting snack we put in front of them. Hooray! Hypothesis confirmed. Time to write that self-help book about how self-control is the key to weight loss, right? Not so fast.”— Ten Years a Skeptic
“I discovered William McGuire's work on perspectivism. His insights radically shifted my thinking about what our experiments can tell us—or more precisely, what they don't tell us. Lab experiments, I came to realize, are merely proofs of concept”— Ten Years a Skeptic
“Say we run an experiment showing that when people exercise self-control in a lab, they eat fewer cookies or chips... That lab study... tells us remarkably little about whether self-control helps people lose weight in the messy reality... Marina Milyavskaya and I actually tested the power of momentary self-control in the real world... self-control didn't predict success at all.”— Ten Years a Skeptic
“I discovered William McGuire's work on perspectivism... Lab experiments... are merely proofs of concept, demonstrations that something could happen under specific conditions. But that's a far cry from showing that these effects matter in the real world.”— Ten Years a Skeptic
“an explosion of vignette studies paired with self-reports... they often fail to answer the real-world questions... a psychology not of actual behaviour, but of how people imagine they might behave.”— Ten Years a Skeptic
“Say we run an experiment showing that when people exercise self-control in a lab, they eat fewer cookies or chips or whatever tempting snack we put in front of them. Hooray! Hypothesis confirmed. Time to write that self-help book about how self-control is the key to weight loss, right? Not so fast.”— Ten Years a Skeptic
“Lab experiments, I came to realize, are merely proofs of concept, demonstrations that something could happen under specific conditions. But that's a far cry from showing that these effects matter in the real world.”— Ten Years a Skeptic
“personality psychology emerged as one of our field's true pillars of strength, producing research that actually holds up under scrutiny.”— Ten Years a Skeptic
“Craig A. Anderson and his colleagues compared laboratory and field research on 38 topics in 21 meta-analyses ... and found a lot of agreement between the results of the two.”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“In social psychology and other sub-disciplines, experimenters are partial to “psychological realism”: “They create a world in the lab and try to activate the same feeling or thoughts inside the lab but not with the same stimuli as outside the lab.””— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“there's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“When they do a correlational analysis but give "policy implications" as if they were doing a causal one, they're not walking around the garden, they're doing the landscaping of forking paths.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“The title of Thaler’s paper alludes to Friedman’s classic (1953) paper on the methodology of positive economics, in which it is argued that the realism of a theory’s predictions is what matters, not the realism of its assumptions.”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“developing a theory of ‘mental accounting’ and using it to understand the ways that consumers respond to different kinds of pricing strategies”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“If you only considered crowd size at rallies for Bernie Sanders and Hillary Clinton, you might wonder how Clinton has won so many big states. Sanders draws massive, enthusiastic crowds, while Clinton's rallies often seem tiny and subdued by comparison.”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“Since last summer, when Sanders' huge rallies got him noticed by the national media, there's been a lot of discussion of an "enthusiasm gap" between Sanders and Clinton. But in a recent Gallup poll, it was Clinton, not Sanders, who had the lead in enthusiasm among supporters.”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“Numerous research methodologies have shown that U.S. college and university faculty tend to be on the political left... however, there has been relatively limited comprehensive empirical analysis of the ideological orientation of that output itself.”— The ideological orientation of academic social science research 1960–2024
“There is evidence that direct biases in classroom instruction, grading, and graduate-level admissions are either unobserved or at most small effects; however, mechanisms by which ideology might influence research have recently been demonstrated in controlled experiments.”— The ideological orientation of academic social science research 1960–2024
“A highly controlled laboratory experiment may have excellent internal validity, but the artificial nature of the lab setting might mean the findings do not apply well to the real world”— The Real-World Approach and Its Problems: A Critique of Ecological Validity
“holding a warm drink makes people act more warmly toward others, claims one finding, and being exposed to stimuli connected to the elderly makes people walk slower, claims another.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“the researchers asked college students to sit and think about maybe not getting to take a class they wanted to take.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“roughly 90 percent of politically relevant social science articles leaned left 1960–2024, and the mean political stance of every social science discipline was left-of-center every year during the period.”— The Ideological Trajectory of the Social Sciences
“e.g., 2005’s “Hegemonic Masculinity” in Gender and Society ranks as an 8 on a 0 to 10 scale of leftism, while 1972’s “The Agenda-Setting Function of Mass Media” ... in Public Opinion Quarterly is a middle of the road 5.”— The Ideological Trajectory of the Social Sciences
How It Spread
The assumption spread through standard academic channels that rewarded clean lab results and large-scale meta-analyses published in respected journals, creating an impression of consensus on external validity. Vignette studies using self-reports proliferated because they were cheap and allowed large samples, even though they measured imagined rather than actual behavior. Meta-analyses that aggregated many studies reinforced the belief that lab findings generalized, and these reviews appeared in psychological journals that carried institutional weight. [1][4]
Flawed papers reached wide audiences regardless of journal prestige because there was no correlation between impact factor and later replicability; citations continued to treat retracted or non-replicable work as valid for decades, embedding the results in literature reviews and textbooks. Thaler's memorable examples of everyday irrationality appeared in books and curricula, while the Nobel Prize and academic appointments helped move behavioral economics into the mainstream. National media amplified the notion that rally crowd sizes indicated broad support by providing extensive visual coverage of large events. [5][6][7]
Psychologists promoted lab findings through TED talks, popular books, and organizational trainings as quick fixes for problems such as prejudice, creating what some called a Primeworld view of subtle cues shaping behavior. A meta-analysis cited as rigorous evidence against color-blind approaches consisted of 77 percent correlational studies and relatively weak experiments. The leftward ideological orientation documented in social science abstracts spread through journal publication, citations, and policy influence, with sociology showing a stable period in the 1970s and 1980s before a steady further shift left after the Berlin Wall fell. [12][13]
▶ Supporting Quotes (14)
“The post went viral—or as viral as things get in academia—caught the attention of The Atlantic, The Globe and Mail, and MIT’s Undark Magazine, and, almost overnight, it changed how I was seen within the field.”— Ten Years a Skeptic
“The post went viral—or as viral as things get in academia—caught the attention of The Atlantic, The Globe and Mail, and MIT’s Undark Magazine.”— Ten Years a Skeptic
“The post went viral—or as viral as things get in academia—caught the attention of The Atlantic, The Globe and Mail, and MIT’s Undark Magazine, and, almost overnight, it changed how I was seen within the field.”— Ten Years a Skeptic
“I’ve noticed an explosion of vignette studies paired with self-reports, where participants are asked how they think they’d respond to hypothetical scenarios. These studies have their place... but I’m concerned that our push for larger sample sizes has unintentionally encouraged overreliance on these quick and therefore inexpensive methods.”— Ten Years a Skeptic
“Mitchell replicated the Anderson study with 217 lab-field comparisons from 82 meta-analyses, in such areas as industrial-organizational (I-O), social, consumer, and developmental psychology.”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“there's no association between statistical power and impact factor, and journals with higher impact factor have more papers with erroneous p-values.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“studies that replicate are cited at the same rate as studies that do not. ... even after retraction the vast majority of citations are positive, and those positive citations continue for decades after retraction.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“his persuasive use of memorable everyday examples of how real people do not always behave as rational choice theory asserts they should: more than anyone else, Thaler has raised the public profile and policy impact”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“Richard H. Thaler was awarded the 2017 Nobel Memorial Prize in Economic Sciences for his contributions to behavioural economics”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“Early on, when Sanders was only polling single digits and big crowds started showing up, they were seen as proof of the appeal of his insurgent message.”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“characterizing the political content of hundreds of thousands of academic texts, especially in a consistent and fine-grained way, has historically been prohibitively expensive and time-intensive when relying on human coders.”— The ideological orientation of academic social science research 1960–2024
“psychologists in the TED Talk world of sexy translation and application have not racked up an impressive track record”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“he touted a meta-analysis lead-authored by Lisa Leslie as constituting “an extensive body of rigorous research” highlighting the potential perils of color blindness, despite the fact that 77% of the included papers were correlational”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“sociology, unsurprisingly moved left during the 1960s, then stabilized during the Sociobiology era of 1975-1985, then moved steadily left through 2024.”— The Ideological Trajectory of the Social Sciences
Resulting Policies
Academic institutions responded to the replication crisis by adopting preregistration, open data sharing, and larger sample sizes as standard practices, although vignette studies with self-reports continued to proliferate because they remained inexpensive. Hiring, tenure, and grant decisions still relied heavily on publication counts and journal prestige without systematic checks for replicability, directing resources toward work that later proved unreliable. [1][5]
Policymakers embraced libertarian paternalism and nudge techniques that rested on behavioral findings challenging strict rational choice assumptions, incorporating insights from Thaler and others into government programs. Organizations adopted diversity, equity, and inclusion trainings and interventions based on priming studies and critiques of color-blind messaging, even though many of the underlying experiments later failed to replicate. [6][12]
Policy-proximal fields such as economics and political science, which maintained left-of-center means, continued to influence public policy with research that reflected those ideological patterns, showing only brief moderation between 1970 and 1990 before shifting further left after 1990. [13]
▶ Supporting Quotes (5)
“Sample sizes, for example, are way up... What seemed radical in 2015—preregistration, open data, open methods—is now standard practice... I’ve noticed an explosion of vignette studies paired with self-reports”— Ten Years a Skeptic
“all these things depend on their peers. When criticising bad research it's easier for everyone to blame the forking paths rather than the person walking them.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“developing the concept of ‘libertarian paternalism’ with Cass Sunstein and exploring its practical implications for policy, especially via the ‘nudge’ technique”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“thanks to the interventions offered by wise behavioral scientists.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“policy-proximal disciplines generally showed limited rightward moderation between roughly 1970 and 1990 ... Economics, political science, and public administration (“policy-proximal disciplines”) all became less leftwing in the 1970s.”— The Ideological Trajectory of the Social Sciences
Harm Caused
Critics such as Inzlicht experienced personal costs including lost friendships, severed mentorships, and professional isolation that contributed to burnout and thoughts of early retirement. The field as a whole devoted resources to studies that later proved worthless, while the push for larger samples raised costs and created funding inequities that disadvantaged smaller labs and researchers without major grants. [1]
Billions of dollars were spent on non-replicating research that distorted scientific agendas and produced false policy implications for interventions in economics, psychology, and management. Journals sometimes protected high-profile researchers despite clear evidence of problems, eroding public trust and prolonging the circulation of questionable claims. [5]
Misled perceptions about the meaning of rally sizes kept candidates such as Bernie Sanders in races longer than delegate math justified, affecting party resources and unity during primary campaigns. Ideological skew in topic selection and framing led to asymmetric research agendas that favored certain questions over others. [7][8]
Overhyped studies prompted organizations to invest time and money in interventions that later showed little real-world effect, while the broader leftward shift in disciplines such as psychology and gender studies directed resources toward ideologically driven work rather than balanced inquiry. [12][13]
▶ Supporting Quotes (11)
“The personal costs were painful. I lost a close friend and mentor... For much of the past decade, I felt like a shadow of my former self... The costs of running such massive studies have created new inequities”— Ten Years a Skeptic
“I lost a close friend and mentor... The personal toll... I felt like a shadow of my former self... There were dark moments when I dreamed of walking away entirely.”— Ten Years a Skeptic
“The costs of running such massive studies have created new inequities, making it harder for researchers without deep pockets... Hell, it’s even hard for labs with deep pockets, like mine.”— Ten Years a Skeptic
“I lost a close friend and mentor—someone I deeply admired and looked up to—because they saw the replication movement as an act of treachery, a betrayal of the giants whose shoulders we stood on.”— Ten Years a Skeptic
“The costs of running such massive studies have created new inequities, making it harder for researchers without deep pockets or access to major funding to run all the studies we want.”— Ten Years a Skeptic
“Rewarding impact without regard for the truth inevitably leads to disaster. ... basing further research on false results”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“Paolo Macchiarini "left a trail of dead patients" but was protected for years by his university. Andrew Wakefield's famously fraudulent autism-MMR study took 12 years to retract.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“So far, Hillary Clinton has received 2.7 million more votes nationwide than Sanders and leads significantly in pledged delegates.”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“Recent randomized experiments have been used to delineate a causal relationship between faculty ideology and academic research... research teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs.”— The ideological orientation of academic social science research 1960–2024
“if you publish a study and overhype it before the evidence is truly in, and as a result excitement, further research, and real-world interventions are generated, and then your finding turns out not to hold up to rigorous replication attempts, you have wasted a huge amount of people’s time, money, and other resources.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“all disciplines showed leftward movement between 1990 and 2024.”— The Ideological Trajectory of the Social Sciences
Downfall
The replication crisis began to expose the limits of many lab effects when large projects such as the Open Science Collaboration found that only 36 percent of 97 significant psychology findings replicated, with average effect sizes halved. Many Labs efforts reported even lower rates around 25 percent for social psychology, and Brian Nosek noted that no social priming finding had proven reliably replicable. [9][12]
Inzlicht's 2015 blog post 'Reckoning with the Past' served as a public turning point, prompting him to declare himself a full skeptic and accelerating methodological reforms such as preregistration and open practices across the field. His later real-world study with Marina Milyavskaya tracked self-control over weeks and found no predictive link to goals such as weight loss, shifting attention from replicability alone to validity concerns. Personality psychology, once nearly dismissed, emerged as more robust than many situational lab effects, helping flatten earlier hierarchies at departments including Michigan, Ohio State, Stanford, Waterloo, and Yale. [1]
Mitchell's large-scale replication of 217 lab-field comparisons documented 30 reversals and poor performance in social psychology and gender research, leading him to argue that external validity required case-by-case assessment rather than blanket assumptions and to recommend more field studies. Replication Markets run through DARPA's SCORE program estimated average replicability at about 54 percent, with prediction markets spotting low-quality work even when experts did not always agree. No clear improvement appeared after 2011, as expected replication rates remained flat through 2018. [4][5]
Thaler's documentation of anomalies such as the endowment effect and his application of Prospect Theory challenged the empirical dominance of rational choice theory, despite initial journal rejections and seminar hostility; the eventual Nobel recognition helped legitimize behavioral economics. Clinton's stronger performance in statewide contests despite smaller rallies, along with earlier examples involving Howard Dean and Ron Paul, illustrated that crowd size did not reliably predict electoral success. An analysis of 600,000 abstracts using large language models traced a leftward ideological shift that accelerated after the Berlin Wall fell and surged again in the mid-2010s, raising questions about balance in social science research. [6][7][13]
▶ Supporting Quotes (16)
“Marina Milyavskaya and I actually tested the power of momentary self-control in the real world... What we found—much to our surprise and perhaps chagrin—was that self-control didn't predict success at all. The lab effects, so clean and promising, simply evaporated in the noise of real life.”— Ten Years a Skeptic
“personality psychology emerged as one of our field's true pillars of strength, producing research that actually holds up under scrutiny... What seemed radical in 2015—preregistration, open data, open methods—is now standard practice.”— Ten Years a Skeptic
“although I still see reliability and replicability as the field’s biggest problems, I now worry about validity a lot more... The replication crisis evolved into a renaissance.”— Ten Years a Skeptic
“with that one piece, I had effectively switched allegiances: publicly moving from the mainstream of social psychology to what some might call the replication-crisis camp.”— Ten Years a Skeptic
“Marina Milyavskaya and I actually tested the power of momentary self-control in the real world. We followed people over weeks and months, tracking their moment-to-moment self-control and their progress toward various goals, including weight loss. What we found—much to our surprise and perhaps chagrin—was that self-control didn't predict success at all.”— Ten Years a Skeptic
“The prestigious programs that once dominated social psychology—Michigan, Ohio State, Stanford, Waterloo, Yale—no longer hold the same unquestioned authority.”— Ten Years a Skeptic
“in a relatively small but concerning number of comparisons, 30 of the 217, the results in the field were the opposite of those in the lab. Of these reversals, the majority came from social psychology. Laboratory studies of gender differences fared particularly poorly when results were tested under more realistic conditions.”— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“Concludes Mitchell: “It’s not really helpful to think in broad terms about external validity. It is a concept you have to take finding by finding, setting by setting.””— Great Results in the Psych Lab—But Do They Hold Up in the Field?
“The average replication probability in the market was 54%; while the replication results are not out yet (250 of the 3000 papers will be replicated), previous experiments have shown that prediction markets work well.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“In reality there has been no discernible improvement.”— What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers
“exposing the empirical shortcomings of rational choice theory and developing the Kahneman and Tversky perspective to make sense of a wide range of anomalies”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“the paper had earlier been rejected by several well-established journals”— Richard H. Thaler: A Nobel Prize for Behavioural Economics
“If big crowds meant big wins at the polls, former Vermont Gov. Howard Dean and former Texas Rep. Ron Paul would both be president. But election after election, crowd size has been an unreliable predictor of winning.”— Campaign Mystery: Why Don't Bernie Sanders' Big Rallies Lead To Big Wins?
“Of 97 original studies with significant effects, 36% replicated successfully, with effect sizes averaging half the original”— Estimating the reproducibility of psychological science
““Combining across journals, 14 of 55 (25%) of social psychology effects replicated,” which was worse than the already-dire rate observed in other areas of psychology.”— Adam Grant vs. Coleman Hughes, Part 2: Causation Does Not Imply. . . Anything
“the 1960s saw the social sciences move dramatically to the left (higher on this graph), then stabilize in mid-1970s to late 1980s, then rise steadily after the collapse of the Muscovite empire ... Then leftism exploded during the late Great Awokening. It may (or may not) have diminished in 2024.”— The Ideological Trajectory of the Social Sciences
Sources
-
[1]
-
[4]
-
[5]
-
[6]
-
[7]
-
[8]
-
[9]
-
[10]
-
[11]
-
[12]
-
[13]
Related False Assumptions