1.3.4: Reproducibility checklist#
Reproducibility is the standard the whole of Section 1.3 has been building towards: the marks of a good model (1.3.1), the strategies for keeping it tractable (1.3.2), and the pitfalls to avoid (1.3.3) all point to the same test. A model is reproducible when another researcher, working from the documentation alone, can recreate its results. That matters because every item that follows is something a reader needs in order to trust the work, cite it, or build on it: documentation in a standard format, recorded seeds and parameters, separate verification and validation evidence, replications with reported uncertainty, and an honest statement of limitations are what turn a model from a private artefact into a reproducible scientific result [1][2].
The checklist below gathers those items into one self-audit, to run before sharing or publishing a model.
Item five on that list, recording the random seed, is what makes a stochastic model exactly reproducible: the model draws its randomness from a seeded random-number generator, so fixing the seed repeats a run identically, while a new seed produces a different but equally valid one. In the activity below, the same model run twice from one seed traces identical trajectories, while a change of seed makes them diverge.
seed= to the model and drawing every random choice from self.random, never from the global random module.The lesson to carry away is that the seed is the one number a reader needs in order to reproduce a stochastic result exactly, which is precisely why checklist item five insists on recording it.
A high score is easier to award yourself than to earn, because reproducibility is harder to enforce in your own writing than to state as a principle: gaps tend to hide in phrases that sound perfectly reasonable until someone tries to rebuild the work from them. A description like “agents were initialised randomly” or “we ran the model once” reads smoothly but withholds the seed, the distribution, or the replication that a reader would need. We are usually blind to these gaps in our own prose, so the quickest way to learn to spot them is to audit someone else’s methods first, where nothing sounds familiar enough to slip past. In the short methods description below, the task is to find every phrase that would stop a reader reproducing or checking the work.
The gaps that were easy to miss all shared one trait: they sounded fine while withholding a detail a reader needs to rebuild the work. The three questions below turn that same reproducibility lens on subtler cases, where the gap is not vague wording but a choice about seeds and update order that a write-up can get quietly wrong.
Reflection questions
A published model uses a single fixed random seed for all reported results and calls this “reproducible”. What is right and what is wrong about that claim?
You are replicating a published model, but the paper does not state the update order (synchronous or asynchronous). You try both and get different results. What does this tell you, and what would you do?
You have finished a model and written it up, but a colleague who has never seen it says they cannot tell how many replications you ran or which parameter values you used. Which items on the checklist above does this reveal are missing, and why does each one block reproduction?
Suggested answers
A fixed seed means the specific run reported can be reproduced exactly, which is useful for debugging. But a single run of a stochastic model is not, on its own, a result: it says little about the distribution of outcomes. The claim is valid but incomplete: the reported run can be reproduced, yet it may not represent the model’s typical behaviour [2].
It tells you the original description is incomplete, because update order affects results and must be documented [1]. In practice you would contact the authors, report both versions and the discrepancy in your replication, and treat the original result as ambiguous until clarified, exactly the kind of gap a standard documentation format is meant to prevent.
It reveals that the number of replications per configuration (checklist item 9) and the documented parameter values with their sources (item 4) are missing. Without the replication count, a reader cannot tell whether the reported outcome is one run or an average, so they cannot reproduce the reported spread; without the parameter values, they cannot rebuild the model at all, only a different model that happens to share its description [1].
Each answer works the same problem from a different angle; the takeaways below distil what they have in common into the points worth carrying out of this section.
Returning to the question this section opened with, whether someone who had never seen your model could reproduce it from the documentation alone: the checklist and the two drills are what make that answer “yes”. With the good practices of Section 1.3 in place, the remaining decision is practical, namely which software and modelling environment to build the model in, and that is where 1.4: Selecting modelling and simulation tools turns next.
Key takeaways#
Key takeaways
A result is reproducible only if another researcher can recreate it from the documentation alone.
Record the question, assumptions, parameters, seeds, code, verification and validation evidence, replications, uncertainty, and limitations.
Reproducibility is widely regarded as the baseline that turns a model into citable scientific evidence.