Python Data Analysis Oopsie

Back in the day, we used to talk about the need for provenance for digitally enabled science [4, 5]. Much of the initial attention focused on data provenance, what data was used and how it was manipulated to get the reported conclusions.

Data is important, but provenance must also include everything you might need to reproduce the results [3].  And this potentially could include anything and everything.

This fall, researchers from Hawaii report an example of just how crucial this challenge has become. They found that a widely used Python library produces different results depending on the platform it runs on [1].  Uh, oh!

Let’s be clear here.  The computational code is identical on each platform and basically correct. No errors are reported, you get an answer that is 99% the same.  But on some versions of Windows, Macosx and Linux, you get different answers.  In fact, two different versions of Macosx give different answers.  Ouch!

Even more interesting, the problem is said to be due to a utility that searches for files by name, which apparently returns the files in slightly different order on different platforms [2].  For many purposes that probably doesn’t matter, but in this case, the chemistry computation gives different answers depending on the order of the data.  Sigh.

(Technical note:  yes, boys and girls, sort order is not a fixed, universal concept.  On most systems, it is a “localization” setting, because it depends on the language and character encodings used, which vary in different cultures.)

Chemists must now review published studies to check which ones might be affected by these issues.   Good luck with that.

“This simple glitch in the original script calls into question the conclusions of a significant number of papers on a wide range of topics in a way that cannot be easily resolved from published information because the operating system is rarely mentioned.” ([1], p. 8452)

Knowing the algorithm used doesn’t answer the question, nor does the fact that Python was used, or even that it was an apple or windows or linux machine.  You have to know the exact version of the operating system used.  More precisely, the version of the operating system actually used when the computation ran, five years ago.

And, by the way, the offending subroutine could have been used by many other libraries, and might or might not produce problematic results.  So, in principle, you need to know if any of the software used a buggy version of the glob utility.  Again, in the configuration that ran whenever it ran, long in the past.

This, my friends, is what you need automated provenance capture for!

---

1. Jayanti Bhandari Neupane, Ram P. Neupane, Yuheng Luo, Wesley Y. Yoshida, Rui Sun, and Philip G. Williams, Characterization of Leptazolines A–D, Polar Oxazolines from the Cyanobacterium Leptolyngbya sp., Reveals a Glitch with the “Willoughby–Hoye” Scripts for Calculating NMR Chemical Shifts. Organic Letters, 21 (20):8449-8453, 2019/10/18 2019. https://doi.org/10.1021/acs.orglett.9b03216
2. Sean Gallagher, esearchers find bug in Python script may have affected hundreds of studies, in ArsTechnical. 2019. https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/
3. Carole Goble, What is Reproducibility? The R* Brouhaha, in First International Workshop on Reproducible Open Science. 2016: Hannover, Germany. http://repscience2016.research-infrastructures.eu/img/CaroleGoble-ReproScience2016v2.pdf
4. Carole Anne Goble and D. De Roure, myExperiment: social networking for workflow-using e-scientists, in Proceedings of the 2nd workshop on Workflows in support of large-scale science. 2007, ACM: Monterey, California, USA. http://doi.acm.org/10.1145/1273360.1273361
5. James D. Myers, Alan R. Chappell, Matthew Elder, Al Geist, and Jens Schwidder, Re-Integrating The Research Record. Computing in Science and Engineering, 5 (3):44-50, May/June 2003. http://ieeexplore.ieee.org/document/1196306/