Do social science studies published in Nature and Science replicate?

The results of efforts to replicate 21 experimental social science studies published in Nature and Science between 2010 and 2015 are reported in Nature Human Behaviour this week. The primary replication method used in the study found that one out of the four Nature papers and seven of the seventeen Science papers evaluated did not replicate.

Understanding the extent to which scientific findings can be trusted is of fundamental importance. Recent attempts to replicate published research from across various fields and journals found that a substantial number of studies do not replicate. To ensure replication rates are not being underestimated, however, higher-powered replication studies with larger sample sizes have been called for.

Colin Camerer, Brian Nosek and colleagues conducted high-powered replications of the 21 experimental social science studies - using sample sizes around five times larger than the original sample sizes. They find that 62% of the replications show an effect in the same direction as the original studies. This effect varies between 57% and 67%, depending on how replicability is determined. The authors suggest that the original studies are likely to have contained both false positives and inflated effect sizes. Based on a survey of 400 social scientists, the authors also report a strong correlation between the research community’s expectations of each study’s replicability and the reality.

In an associated collection of eight Correspondences, non-replicating study authors discuss why their results did not replicate, providing their views into potential sources of irreproducibility. In an accompanying News & Views, Malcolm Macleod argues that replication studies are necessary for establishing a solid foundation for the social sciences, and can generate important new research into the parameters that could lead to more robust scientific conclusions.

doi:10.1038/s41562-018-0405-5

Return to research highlights