Good modeling practices

Some thoughts I’ve been collecting, primarily oriented toward system dynamics modeling in Vensim, but relevant to any modeling endeavor:

  • Know why you’re building the model.
    • If you’re targeting a presentation or paper, write the skeleton first, so you know how the model will fill in the answers as you go.
  • Organize your data first.
    • No data? No problem. But surely you have some reference mode in mind, and some constraints on behavior, at least in extreme conditions.
    • In Vensim, dump it all into a spreadsheet, database, or text file and import it into a data model, using the Model>Import data… feature, GET XLS DATA functions, or ODBC.
    • Don’t put data in lookups (table functions) unless you must for some technical reason; they’re a hassle to edit and update, and lousy at distinguishing real data points from interpolation.
  • Keep a lab notebook. An open word processor while you work is useful. Write down hypotheses before you run, so that you won’t rationalize surprises.
  • Keep your models organized, and keep versions as you go.
    • The simple way: keep all the files associated with a model in one directory. Save each major model improvement with a new number (mymodel v16.mdl). Periodically copy the whole directory and start fresh, so you don’t have too many models and ancillary files in one place.
    • The sophisticated way: use Subversion
  • Enter and balance units as you go. They prevent a lot of errors. Really.
  • Document as you go, at least for nontrivial equations. Otherwise you’ll hate yourself in the morning, when you have to do it all at once.
  • Use transparent variable names with a clear directional interpretation.
  • Keep your diagrams organized.
    • Clean up spaghetti from time to time.
    • Use color and styles to identify key constants, driving data, etc.
  • Build graphs to make comparison of your outputs with data and expectations easy.
  • Test new structures in submodels, not in their full-feedback context. That way you can easily perform equilibrium and extreme conditions tests without the overhead of a full model.
  • Look at the output of every variable from time to time. If you can’t do that, your model is probably too big.
  • Keep a “todo” list in your model, or in your lab notebook. Include puzzles and anomalies to be solved later.
  • Run frequently. Build large structures incrementally so that what you have is always tested and understood.
  • Avoid nested IF…THEN logic and other confusing, discrete structures where possible.
  • Break up long equations into digestible chunks. If it won’t fit in the equation editor’s text field, no human can understand it.
  • Never embed nontrivial constants – especially those with implicit units – in equations. Pull them out as separate constants.
    • Bad: Price = Variable_Cost*1.234
    • Better: Price = Variable_Cost*(1+Markup)
    • OK: Time_Constant = 1/Fractional_Rate
  • Test those nontrivial constants for sensitivity.
    • You can perform many tests manually, in Synthesim, by wiggling sliders.
    • Automate brute force tests of every parameter in your model using the optimizer.
  • Test decision inputs with particular thoroughness.
    • Try every control at its extremes, and in combination with others.
    • Try perturbing your decisions, looking for improved performance or to determine whether your simulated agents are systematically biased or too dumb.
  • Test everything in extreme conditions. Synthesim overrides are a handy way to do this – freeze a stock at its initial level, and voila – you’ve broken all the feedback loops involving it.
  • Build checksums for complex array operations (e.g., shares that sum to 1). Set limits on the variables concerned, so that you get a runtime warning if the checksum is violated.
  • Write Reality Checks to watch your checksums and automate other kinds of tests. You’re likely to perform a test only once manually, but it’s easy to repeat if automated as an RC. Also, RC’s don’t slow down conventional model runs.
  • Rules were made to be broken.

Questions? Additions? Comment!

Leave a Reply

Your email address will not be published. Required fields are marked *