Bottom Lines and Reproducibility

I have a thing about reproducibility. I’m not the only one as this post on In The Pipeline shows.  Nothing annoys me more than someone spending their time and someone else’s money to carry out an experiment then not taking the care and attention necessary to record the details in a manner that allows another to reproduce the result obtain. People who don’t interpret their spectra properly and by default never prove the outcome of their reactions either way come a close second in annoyance.

I’m afraid I fall in to the skeptical camp when it comes to scientific literature. I feel I’ve written about this quite a lot in the past (http://www.possibilitiesendless.com/2013/08/frustrations-of-reproducibility/ for one). I’m sure that a lot of what is published is reproducible by the person who did it in the first place. But I’m also sure that that researcher probably had a good number of habits that probably weren’t recorded as part of the procedure and I think sometimes that will affect the outcome.  Some chemists use argon atmospheres by default but may refer to it more generically. Other details, modes of addition, method of cleaning glassware (or the impurities left in the glassware!)…the little things sometimes matter. I’ve also had a few procedures where doing the ‘established technique’ results in disaster. For example, I have one reaction where the product has to crash out as the reflux cools. Generally people would remove the solvent under reduced pressure if the product failed to crash out but in this instance that produces goo, not the desired (selectively insoluble) product.

As quoted in the post below, I’ve tried to reproduce really really interesting results and haven’t been able to. This has been likely due to some key details being missed from the records kept.  It makes me quite cross on a single-student-single-result basis, but fortunately that result didn’t get published because it was not reproducible. I suppose the question I should be asking is to what lengths should we go to ensure work is reproducible? The ultimate solution is to have someone verify the work independently, but the resources required to do so are ridiculous.  After that it comes down to the scruples of the researchers involved, they’ll probably record the minimum level of detail required by their supervisor. It isn’t reasonable to expect research groups to duplicate each and every result, but it is reasonable to expect research groups to produce original data to support their results. So my stop-gap measure would be to keep really good records, and make sure you have a filing system to support retrieval of the raw data (instrument native format) if requested.

2 thoughts on “Bottom Lines and Reproducibility

Comments please!