The reproducibility crisis

A result is reproducible when someone else can follow the same steps and get the same result. Science depends on it. Yet many published results are hard to reproduce.

What the surveys found

In 2016, the journal Nature asked 1,576 researchers about reproducibility. More than 70% said they had tried and failed to reproduce another scientist's experiments, and more than half had failed to reproduce their own. 52% said there was a "significant crisis", and another 38% a "slight crisis".

In 2015, the Open Science Collaboration repeated 100 published psychology studies. 97% of the original studies had reported a statistically significant result. In the repeats, only 36% did.

Why it happens

• Small samples, which give noisy results that look larger than they are.
• Trying many analyses and reporting only the one that "worked" (you will meet this as p-hacking in the next module).
• Missing details: the paper does not say exactly what was done.
• Code and data not shared, so nobody can check.
• Publication bias, which rewards surprising results.

Machine learning has it too

ML results can fail to reproduce because of unreported random seeds, different software versions, test data leaking into training, or a baseline that was not tuned as carefully as the new method. Conferences like NeurIPS now ask authors to fill in a reproducibility checklist.

What you can do from day one

• Write down every step, setting and version.
• Share your code and data when you are allowed to.
• Run experiments more than once, with different seeds, and report the spread.
• Report what did not work as well as what did.

RememberAPJ-RAISE publishes papers together with their code and data for exactly this reason: so that anyone can check them.