Reproducibility and replicability of computer simulations

Keynote talk at ACM REP ’24 on June 20th, 2024

• Read the transcript
• Or get just the slides

My digital garden on the topic of “Science in the Digital Era” contains several short essays on reproducibility, replicabilty, and related issues. A good starting point is the essay on Computational Reproducibility. From there, follow the links and backlinks.

My book Computation in Science explains the fundamentals of computing that computational scientists should know. Chapter 6 is dedicated to reproducibility.

In the following, I list additional material on the topics covered in my talk.

Reproducibility

• Dealing With Software Collapse, Comp. Sci. Eng. 21, 104-108 (2019) (preprint) analyzes the typical scientific software stack and the reasons why it tends to collapse. This is the main cause of computational irreproducibility.

The ten-year reproducibility challenge run by the journal ReScience C in 2020, has lead to a nice collection of reproducibility reports in which the authors report on trying to re-run their own published code that was at least ten years old. It includes several contributions that use Guix to retrofit reproducibility to the old code.

My recorded talk on “Guix as a tool for computational science”, the Guix Ten Year Anniversay colloquium in 2022, discusses best practices for using Guix for reproducible computational science.

The first session of the new MOOC “Reproducible Research II: Practices and tools for managing computations and data” will run until September 12. Make it your personal summer school on computational reproducibility!

Computational models

• The Nature of Computational Models, Comp. Sci. Eng. 25, 61-66 (2023) (preprint) provides and overview of various kinds of computational models and how they relate to traditional mathematical models.
• Computational science: shifting the focus from tools to models (Open Access) argues that models are more important for science than computational tools and that we need to free our models from their current imprisonment in the source code for tools.

Digital Scientific Notations

• Scientific notations for the digital era compares software source code to traditional scientific notations (in particular mathematics) and proposes a new type of scientific notation that combines the best of both worlds. In Digital Science, science writer Mark Buchanan provides a concise summary.
• Scientific communication in the digital age is a short comment in Physics Today about the same topic.

Leibniz

• the code for the current prototype
• a static rendering of all the Wiki pages of the Leibniz prototype
• a four-minute demo video, featuring the Lotka-Volterra equations
• Verifiability in computer-aided research: the role of digital scientific notations at the human-computer interface, PeerJ Computer Science 4:e158, summarizes the experience gained from the first Leibniz prototype, based on a static publishing system that resembles traditional scientific publications.

If you are interested in the history of the Leibniz project, see this page in my Digital Garden.

Trustworthy computational science

Reproducibility and replicability are key ingredients to trustworthy science using computers. But they are not enough. In the preprint “Establishing trust in automated reasoning” I explore what else can and should be done.

I am still looking for a journal to which I could submit a paper like this one. If you have ideas, please contact me!