There’s an old joke:
Q: Why are the debates so bitter in academia?
A: Because the stakes are so low.
The stakes are actually very high when models intersect with policy, I think, but sometimes academic debates come across as needlessly petty. That joke came to mind when a colleague shared this presentation abstract:
Pathologies of System Dynamics Models or “Why I am Not a System Dynamicst”
by Dr. Robert Axtell
So-called system dynamics (SD) models are typically interpreted as a summary or aggregate representation of a dynamical system composed of a large number of interacting entities. The high dimensional microscopic system is abstracted-notionally if not mathematically-into a ‘compressed’ form, yielding the SD model. In order to be useful, the reduced form representation must have some fidelity to the original dynamical system that describes the phenomena under study. In this talk I demonstrate formally that even so-called perfectly aggregated SD models will in general display a host of pathologies that are a direct consequence of the aggregation process. Specifically, an SD model can exhibit spurious equilibria, false stability properties, modified sensitivity structure, corrupted bifurcation behavior, and anomalous statistical features, all with respect to the underlying microscopic system. Furthermore, perfect aggregation of a microscopic system into a SD representation will generally be either not possible or not unique.
Finally, imperfectly aggregated SD models-surely the norm-can possess still other troublesome features. From these purely mathematical results I conclude that there is a definite sense in which even the best SD models are at least potentially problematical, if not outright mischaracterizations of the systems they purport to describe. Such models may have little practical value in decision support environments, and their use in formulating policy may even be harmful if their inadequacies are insufficiently understood.
In a technical sense, I agree with everything Axtell says.
However, I could equally well give a talk titled, “pathologies of agent models.” The pathologies might include ungrounded representation of agents, overuse of discrete logic and discrete time, failure to nail down alternative hypotheses about agent behavior, and insufficient exploration of sensitivity and robustness. Notice that these are common problems in practice, rather than problems in principle, because in principle one would always prefer a disaggregate representation. The problem is that we don’t build models in principle; we build them in practice. In reality resources – including data, time, computing, statistical methods, and decision maker attention – are limited. If you want more disaggregation, you’ve got to have less of something else.
Clearly there are times when an aggregate approach could be misleading. To leap from the fact that one can demonstrate pathological special cases to the idea that aggregate models are dangerous strikes me as a gross overstatement. Is the danger of aggregating agents really any greater than the danger of omitting feedback by reducing scope in order to enable modeling disaggregate agents? Hopefully this talk will illuminate some of the ways that one might think about whether a situation is dangerous or not, and therefore make informed choices of method and tradeoffs between scope and detail.
Also, models seldom inform policy directly; their influence occurs through improvement of mental models. Agent models could have a lot to offer there, but I haven’t seen many instances where authors developed the lingo to communicate insights to decision makers at their level. (Examples appreciated – any links?) That relegates many agent models to the same role as other black-box models: propaganda.
It’s strange that Axtell is picking on SD. Why not tackle economics? Most economic models have the same aggregation issues, plus they assume equilibrium and rationality from the start, so any representational problems with SD are greatly amplified. Plus the economic models are far more numerous and influential on policy. It’s like Axtell is bullying the wimpy kid in the class, because he’s scared to take on the big one who smokes at recess and shaves in 5th grade.
The sad thing about this confrontational framing is that SD and agent based modeling are a match made in heaven. At some level disaggregate models still need aggregate representations of agents; modelers could learn a lot from SD about good representation of behavior and dynamics, not to mention good habits like units checking that are seldom followed. At the same time, SD modelers could learn a lot about emergent phenomena and the limitations of aggregate representations. A good example of a non-confrontational approach, recognizing shades of gray:
Heterogeneity and Network Structure in the Dynamics of Diffusion: Comparing Agent-Based and Differential Equation Models
Hazhir Rahmandad, John Sterman
When is it better to use agent-based (AB) models, and when should differential equation (DE) models be used? Whereas DE models assume homogeneity and perfect mixing within compartments, AB models can capture heterogeneity across individuals and in the network of interactions among them. AB models relax aggregation assumptions, but entail computational and cognitive costs that may limit sensitivity analysis and model scope. Because resources are limited, the costs and benefits of such disaggregation should guide the choice of models for policy analysis. Using contagious disease as an example, we contrast the dynamics of a stochastic AB model with those of the analogous deterministic compartment DE model. We examine the impact of individual heterogeneity and different network topologies, including fully connected, random, Watts-Strogatz small world, scale-free, and lattice networks. Obviously, deterministic models yield a single trajectory for each parameter set, while stochastic models yield a distribution of outcomes. More interestingly, the DE and mean AB dynamics differ for several metrics relevant to public health, including diffusion speed, peak load on health services infrastructure, and total disease burden. The response of the models to policies can also differ even when their base case behavior is similar. In some conditions, however, these differences in means are small compared to variability caused by stochastic events, parameter uncertainty, and model boundary. We discuss implications for the choice among model types, focusing on policy design. The results apply beyond epidemiology: from innovation adoption to financial panics, many important social phenomena involve analogous processes of diffusion and social contagion. (Paywall; full text of a working version here)
Details, in case anyone reading can attend – report back here!
Thursday, October 21 at 6:00 – 8:00 PM ** New Time **
Networking 6:00 – 6:45 PM (light refreshments) Presentation 6:45 – 8:00 PM Free and open to the public
** NEW Location **
Booz Allen Hamilton – Ballston-Virginia Square
3811 N. Fairfax Drive, Suite 600
Arlington, VA 22203
(703) 816-5200
Between Virginia Square and Ballston Metro stations, between Pollard St.
and Nelson St.
On-street parking is available, especially on 10th Street near the Arlington Library.
There will be a Booz Allen representative at the front of the building until 7:00 to greet and escort guests, or call 703-627-5268 to be let in.
RSVP by e-mail to Nicholas Nahas, nahas_nicholas@bah.com<mailto:nahas_nicholas@bah.com>, in order to have a rough count of attendees prior to the meeting. Come anyway even if you do not RSVP.
By METRO:
Take the Orange Line to the Ballston station. Exit Metro Station, walk towards the IHOP (right on N. Fairfax) continue for approximately 2-3 blocks. Booz Allen Hamilton (3811 N. Fairfax Dr. Suite 600) is on the left between Pollard St. and Nelson St.
OR Take the Orange Line to the Virginia Square station. Exit Metro Station and go left and walk approximately 2-3 blocks. Booz Allen Hamilton (3811 N. Fairfax Dr. Suite 600) is on the right between Pollard St. and Nelson St.
