Replicability under the microscope: Major study highlights the value of open and transparent research

Published in the journal Nature, the study “Investigating the replicability of the social and behavioral sciences” has reignited an important discussion in economics and the wider social sciences: how confident should we be in the results of individual research studies? 

The large-scale international project brought together 292 researchers from 199 institutions across the world. Their task was ambitious: to test whether findings from 164 published studies in disciplines such as economics, business, political science, psychology, sociology and education could be reproduced when researchers repeated the analyses using new or independent data.

The answer was mixed. Roughly half of the findings could be replicated successfully. At the same time, the effects observed in the replication studies were typically much smaller than those reported in the original research papers.

For businesses, policymakers and economists outside academia, the findings carry an important message. Scientific research remains one of the strongest tools for understanding society and markets, but single studies should rarely be treated as definitive truths.

Among the authors is Professor Jesper N. Wulff from the Department of Economics and Business Economics at Aarhus University. He describes the project as an unprecedented attempt to investigate whether results in the social sciences can actually be recreated when other researchers repeat the work under transparent and rigorous conditions.

“The original studies are not necessarily wrong,” Jesper N. Wulff explains. “But the results show that scientific findings are often more uncertain and context-dependent than headlines may suggest.”

Putting findings to the test

The replication effort formed part of the SCORE programme – short for Systematizing Confidence in Open Research and Evidence. Unlike many earlier replication projects, the researchers designed the studies in advance, preregistered their methods, subjected the plans to peer review beforehand and ensured that the 
analyses were transparent and statistically robust.

The researchers selected published papers from leading journals between 2009 and 2018. They then attempted to reproduce the original findings either with newly collected data or with alternative existing datasets.

One of the studies examined by Jesper N. Wulff focused on executive pay. The original paper had concluded that chief executives receive higher salaries when they lead firms operating in industries with high barriers to entry, such as pharmaceuticals or construction.

Jesper N. Wulff revisited the data and supplemented it with additional information. But the results did not support the same clear conclusion as the original paper.

“The original study identified a strong positive effect,” he says. “In our replication, we found that the evidence was far more uncertain. We could not confidently conclude one way or the other.”

That pattern appeared repeatedly throughout the Nature study. Even when replication studies pointed in the same direction as the original research, the measured effects were frequently weaker. On average, the replication effects were reduced by more than half, and in roughly 70 per cent of cases the replicated effect size was smaller than originally reported.

The study also highlights how difficult it is to define what counts as a successful replication. Depending on the statistical criteria used, the estimated replication rate ranged from 29 per cent to 75 per cent.

This matters because scientific findings are often presented in simplified terms outside academia. In practice, many research results depend on assumptions, methods, data choices and interpretation. Small methodological differences can sometimes change the outcome substantially.

The value of openness

According to Jesper N. Wulff, one of the clearest lessons concerns transparency.

“Current rates of replicability and reproducibility leave room for improvement,” he says. “Any insights that can make the path to reliable findings more reliable will accelerate progress. Looking back at previous work is as necessary as looking ahead, and rigorous practices to do so demonstrate the scientific method at work.”

The study therefore argues for broader use of open science practices. One recommendation receiving particular attention is preregistration – a process where researchers publicly specify in advance what they intend to test and how they will analyse their data.

Jesper N. Wulff compares the current situation to a dart game where the target is drawn after the dart has landed.

“It would strengthen research quality if researchers preregistered their hypotheses and analytical strategies beforehand,” he says. “Otherwise, there is a risk of chasing striking patterns in the data afterwards instead of testing planned hypotheses.”

^{Professor Jesper N. Wulff, Department of Economics and Business Economics, Aarhus BSS}

Another issue is access to underlying research material. In many cases, external researchers cannot fully inspect the data behind published studies, making critical evaluation more difficult.

“That lack of openness is a major challenge,” Jesper N. Wulff says. “Science becomes stronger when other researchers are able to scrutinise and test the results independently.”

Importantly, the researchers behind the Nature paper stress that failed replications should not be interpreted as proof that entire fields are fundamentally flawed. Replication outcomes can be influenced by differences in datasets, methods, timing, populations and even random statistical variation.

Instead, the findings underline that scientific knowledge develops cumulatively over time. Confidence in a claim should grow gradually as multiple studies point in the same direction across different contexts and datasets.

For economists and business professionals, the implications are highly relevant. Research findings increasingly influence policy decisions, investment strategies, labour market reforms and corporate governance practices. But the Nature study suggests that decision-makers should exercise caution when relying too heavily on individual headline-grabbing studies.

The researchers argue that uncertainty is not a weakness of science but one of its defining features. Replication serves as a mechanism for self-correction, helping researchers separate robust patterns from findings that may be more fragile or context-specific.

“A 100 per cent replication rate is not necessarily the goal. It might even imply that the work is extremely conservative and does not push the boundaries of knowledge into the unknown. The point is not to eliminate uncertainty, but to handle it rigorously.”

^{Professor Jesper N. Wulff, Department of Economics and Business Economics, Aarhus BSS}

The study also demonstrates that the replicability challenge is not confined to one discipline. Economics, psychology, sociology and political science all showed relatively similar replication rates. No single field clearly outperformed the others.

For Jesper N. Wulff, the broader lesson is ultimately about how society interprets evidence.

“We should probably place less weight on isolated studies and more emphasis on whether many independent studies arrive at similar conclusions,” he says. “That may take years, but it is the best way to build reliable knowledge.”

The Nature publication represents one of the most comprehensive efforts yet to examine how reliable published social science findings are. Its conclusion is not that social science is broken, but that uncertainty and nuance deserve a more prominent place in how research is communicated and applied.

For economists and professionals outside academia, the message is clear: evidence-based decisions remain essential, but evidence is rarely absolute. In the social sciences, communicating nuance, uncertainty and the limitations of a study is just as important as communicating the result itself.

Facts

We strive to comply with Universities Denmark’s principles for good research communication. For this reason, we provide the following information as a supplement to this article: