# 1.3.3: Common pitfalls Even modellers who know the principles fall into a predictable set of traps. Recognising them is the first step to avoiding them. Consider each in turn: ::::{tab-set} :::{tab-item} Implicit assumptions An assumption that is left implicit cannot be examined or challenged by others, so making every assumption explicit is a basic requirement of transparent, reproducible modelling [^1]. *The fix*: write down each assumption, then ask what would happen to the conclusions if it were wrong; where the answer is "a great deal", test it with a sensitivity analysis [^2]. *Example*: assuming all agents are identical when differences between them are part of what drives the result. ::: :::{tab-item} Unjustified complexity Adding a mechanism because it seems realistic, rather than because the simpler model fails without it, adds parameters to estimate and more that can go wrong, without necessarily making the model more useful [^2]. *The fix*: for each feature you consider adding, check whether the model without it already fails to address the question; if it does not, leave the feature out [^2]. ::: :::{tab-item} Incomplete documentation Incomplete model descriptions, missing parameter values, and undocumented code are recognised reasons that published simulation results are often hard to reproduce [^1]. *The fix*: document the model in a standard format (such as the ODD protocol), write the documentation as you build, and record parameter values and random seeds for every reported result [^1]. *Example*: stating that entities "were initialised randomly" without the distribution, range, or seed, which a reader cannot reproduce. ::: :::{tab-item} Confusing verification and validation As Section 1.2.4 set out, verification (did we build the model right?) and validation (did we build the right model?) answer different questions; treating a verified, bug-free model as automatically valid conflates the two [^3]. *The fix*: treat them as two separate questions, each needing its own evidence, verification that the code matches the design, and validation that the model is adequate for its purpose [^3]. ::: :::{tab-item} Over-fitting Tuning parameters so closely to one dataset that the model loses generality is over-fitting; such a model can match its calibration data yet perform poorly on new data or slightly different conditions [^4]. *The fix*: calibrate and test on separate data, and prefer the simplest parameter set that gives an acceptable fit over a more complex one that gives a perfect fit [^4]. ::: :::: Before examining any single trap in depth, it helps to practise telling them apart. In the activity below, decide which of the five pitfalls each described mistake illustrates. ```{raw} html :file: ../learning_activities_and_widgets/learning_activities/activity_1_3_3a_name_the_pitfall.html ``` ## 1.3.3.1: Spotting the riskiest assumption Of these five traps, the first (hidden assumptions) is the hardest to catch, because the riskiest assumptions are invisible by design, so the rest of this section gives you practice at spotting them. Every model rests on assumptions, but not all of them carry the same risk. The dangerous ones are usually the quiet, taken-for-granted simplifications, precisely because they are easy to overlook yet can decide the result: that all agents are identical, that everyone mixes at random, that people have perfect knowledge. Learning to identify the assumption a conclusion hinges on is what tells you where to direct a sensitivity analysis, rather than testing everything or nothing. In the activity below, identify, for each description, the assumption most likely to change the result if it were wrong. ```{raw} html :file: ../learning_activities_and_widgets/learning_activities/activity_1_3_3b_spot_the_assumption.html ``` Apply the same habit to the other four pitfalls as you build your own models: name the risk before it bites. Taken together, avoiding all five comes down to a short set of working rules, summarised below. Documenting a model well enough to sidestep these pitfalls is a discipline in itself, and Section 1.3.4 turns it into a concrete reproducibility checklist. ## Key takeaways ```{admonition} Key takeaways :class: important - Make every assumption explicit, then test the ones the conclusions depend on. - Add complexity only when the simpler model fails; unjustified detail adds cost and risk without necessarily adding value. - Document the model in a standard format, with parameter values and seeds, so results can be reproduced. - Keep verification and validation separate, and guard against over-fitting by testing on data the model was not calibrated to. ``` ## References [^1]: Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., et al. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. *Journal of Artificial Societies and Social Simulation*, 23(2), 7. [^2]: Railsback, S. F., & Grimm, V. (2019). *Agent-based and individual-based modeling: A practical introduction* (2nd ed.). Princeton University Press. [^3]: David, N., Fachada, N., & Rosa, A. C. (2017). Verifying and validating simulations. In B. Edmonds & R. Meyer (Eds.), *Simulating social complexity: A handbook* (2nd ed., pp. 173–204). Springer. [^4]: Law, A. M. (2015). *Simulation modeling and analysis* (5th ed.). McGraw-Hill Education.