Tag: conference

Writing a good system dynamics paper II

It’s SD conference paper review time again. Last year I took notes while reviewing, in an attempt to capture the attributes of a good paper. A few additional thoughts:

  • No model is perfect, but it pays to ask yourself, will your model stand up to critique?
  • Model-data comparison is extremely valuable and too seldom done, but trivial tests are not interesting. Fit to data is a weak test of model validity; it’s often necessary, but never sufficient as a measure of quality. I’d much rather see the response of a model to a step input or an extreme conditions test than a model-data comparison. It’s too easy to match the model to the data with exogenous inputs, so unless I see a discussion of a multi-faceted approach to validation, I get suspicious. You might consider how your model meets the following criteria:
    • Do decision rules use information actually available to real agents in the system?
    • Would real decision makers agree with the decision rules attributed to them?
    • Does the model conserve energy, mass, people, money, and other physical quantities?
    • What happens to the behavior in extreme conditions?
    • Do physical quantities always have nonnegative values?
    • Do units balance?
  • If you have time series output, show it with graphs – it takes a lot of work to “see” the behavior in tables. On the other hand, tables can be great for other comparisons of outcomes.
  • If all of your graphs show constant values, linear increases (ramps), or exponentials, my eyes glaze over, unless you can make a compelling case that your model world is really that simple, or that people fail to appreciate the implications of those behaviors.
  • Relate behavior to structure. I don’t care what happens in scenarios unless I know why it happens. One effective way to do this is to run tests with and without certain feedback loops or sectors of the model active.
  • Discuss what lies beyond the boundary of your model. What did you leave out and why? How does this limit the applicability of the results?
  • If you explore a variety of scenarios with your model (as you should), introduce the discussion with some motivation, i.e. why are the particular scenarios tested important, realistic, etc.?
  • Take some time to clean up your model diagrams. Eliminate arrows that cross unnecessarily. Hide unimportant parameters. Use clear variable names.
  • It’s easiest to understand behavior in deterministic experiments, so I like to see those. But the real world is noisy and uncertain, so it’s also nice to see experiments with stochastic variation or Monte Carlo exploration of the parameter space. For example, there are typically many papers on water policy in the ENV thread. Water availability is contingent on precipitation, which is variable on many time scales. A system’s response to variation or extremes of precipitation is at least as important as its mean behavior.
  • Modeling aids understanding, which is intrinsically valuable, but usually the real endpoint of a modeling exercise is a decision or policy change. Sometimes, it’s enough to use the model to characterize a problem, after which the solution is obvious. More often, though, the model should be used to develop and test decision rules that solve the problem you set out to conquer. Show me some alternative strategies, discuss their limitations and advantages, and describe how they might be implemented in the real world.
  • If you say that an SD model can’t predict or forecast, be very careful. SD practitioners recognized early on that forecasting was often a fool’s errand, and that insight into behavior modes for design of robust policies was a worthier goal. However, SD is generally about building good dynamic models with appropriate representations of behavior and so forth, and good models are a prerequisite to good predictions. An SD model that’s well calibrated can forecast as well as any other method, and will likely perform better out of sample than pure statistical approaches. More importantly, experimentation with the model will reveal the limits of prediction.
  • It never hurts to look at your paper the way a reviewer will look at it.

Writing an SD Conference Paper

It’s review time for SD conference papers again. As usual, there’s a lot of variance in quality: really good stuff, stuff that isn’t SD, and good ideas imprisoned in a bad presentation. A few thoughts on how to write a good conference paper, in no particular order:

  • Read a bunch of good SD papers, by browsing the SD Review, Dynamica, Desert Island Dynamics, or past conference plenary papers. You could do a lot worse than picking one as a model for your paper.
  • Start with: What’s the question? Why do we care? Who’s the audience? How will they be influenced? What is their prevailing mental model, and how must it change for things to improve? (If your paper is a methods paper, not a model paper, perhaps the relevant questions are different, but it’s still nice to know why I’m reading something up front.)
  • If you have a model,
    • Make sure units balance, stocks and flows are conserved, structure is robust in extreme conditions, and other good practices are followed. When in doubt, refer to Industrial Dynamics or Business Dynamics.
    • Provide a high-level diagram.
    • Describe what’s endogenous, what’s exogenous, and what’s excluded.
    • Provide some basic stats – What’s the time horizon? How many state variables are there?
    • Provide some data on the phenomena in question, or at least reference modes and a dynamic hypothesis.
    • Discuss validation – how do we know your model is any good?
    • Discuss “Which Policy Run is Best, and Who Says So?” (See DID for the reference).
    • Provide the model in supplementary material, if at all possible.
    • Use intelligible and directional variable names.
    • Clearly identify the parameter changes used to generate each run.
    • Change only one thing at a time in your simulation experiments (or more generally, use scientific method).
    • Explore uncertainty.
    • If your output shows interesting dynamics (or weird discontinuities and other artifacts), please explain.
    • Most importantly, clearly explain why things are happening by relating behavior to structure. Black-box output is boring. Causal loop diagrams or simplified stock-flow schematics may be helpful for explaining the structure of interest.
  • If you use CLDs, Read Problems with Causal Loop Diagrams and Guidelines for Drawing Causal Loop Diagrams and Chapter 5 of Business Dynamics.
  • Archetypes are a compact way to communicate a story, but don’t assume that everyone knows them all. Don’t shoehorn your problem into an archetype; if it doesn’t fit, describe the structure/behavior in its own right.
  • If you present graphs, label axes with units, clearly identify each series, etc. Follow general good practice for statistical graphics. I like lots of graphs because they’re information-rich, but each one should have a clear purpose and association with the text. Screenshots straight out of some modeling packages are not presentation-quality in my opinion.
  • I don’t think it’s always necessary to follow the standard scientific journal article format, it could even be boring, but when in doubt it’s not a bad start.
  • If your English is not the best (perhaps even if it is), at least seek help editing your abstract, so that it’s clear and succinct.
  • Ask yourself whether your paper is really about system dynamics. If you have a model, is it dynamic? Is it behavioral? Does it employ an operational description of the system under consideration? If you’re describing a method, is it applicable to (possibly nonlinear) dynamic systems? If you’re describing a process (group modeling, for example), does it involve decision making or inquiry into a dynamic system? I welcome cross-disciplinary papers, but I think pure OR papers (say, optimizing a shop-floor layout) belong at OR conferences.
  • Do a literature search, especially of the SD Review and SD bibliography, but also of literature outside the field, so that you can explain how the model/method relates to past work in SD and to different perspectives elsewhere. Usually it’s not necessary to report all the gory details of other papers though.
  • Can’t think of a topic? Replicate a classic SD model or a model from another field and critique it. See Christian Erik Kampmann, “Replication and revision of a classic system dynamics model: Critique of ‘Population Control Mechanisms’ System Dynamics Review 7(2), 1991. Or try this.
  • Rejected anyway? Don’t feel bad. Try again next year!