Category: SystemDynamics

Modelers: you're not competing

Well, maybe a little, but it doesn’t help.

From time to time we at Ventana encounter consulting engagements where the problem space is already occupied by other models. Typically, these are big, detailed models from academic or national lab teams who’ve been working on them for a long time. For example, in an aerospace project we ran into detailed point-to-point trip generation models and airspace management simulations with every known airport and aircraft in them. They were good, but cumbersome and expensive to run. Our job was to take a top-down look at the big picture, integrating the knowledge from the big but narrow models. At first there was a lot of resistance to our intrusion, because we consumed some of the budget, until it became evident that the existence of the top-down model added value to the bottom-up models by placing them in context, making their results more relevant. The benefit was mutual, because the bottom-up models provided grounding for our model that otherwise would have been very difficult to establish. I can’t quite say that we became one big happy family, but we certainly developed a productive working relationship.

I think situations involving complementary models are more common than head-to-head competition among models that serve the same purpose. Even where head-to-head competition does exist, it’s healthy to have multiple models, especially if they embody different methods. (The trouble with global climate policy is that we have many models that mostly embody the same general equilibrium assumptions, and thus differ only in detail.) Rather than getting into methodological pissing matches, modelers should be seeking the synergy among their efforts and making it known to decision makers. That helps to grow the pie for all modeling efforts, and produces better decisions.

Certainly there are exceptions. I once ran across a competing vendor doing marketing science for a big consumer products company. We were baffled by the high R^2 values they were reporting (.92 to .98), so we reverse engineered their model from the data and some slides (easy, because it was a linear regression). It turned out that the great fits were due to the use of 52 independent parameters to capture seasonal variation on a weekly basis. Since there were only 3 years of data (i.e. 3 points per parameter), we dubbed that the “variance eraser.” Replacing the 52 parameters with a few targeted at holidays and broad variations resulted in more realistic fits, and also revealed problems with inverted signs (presumably due to collinearity) and other typical pathologies. That model deserved to be displaced. Still, we learned something from it: when we looked cross-sectionally at several variants for different products, we discovered that coefficients describing the sales response to advertising were dependent on the scale of the product line, consistent with our prior assertion that effects of marketing and other activities were multiplicative, not additive.

The reality is that the need for models is almost unlimited.  The physical sciences are fairly well formalized, but models span a discouragingly small fraction of the scope of human behavior and institutions. We need to get the cost of providing insight down, not restrict the supply through infighting. The real enemy is seldom other models, but rather superstition, guesswork and propaganda.

Ben Franklin, systems thinker

I find that many great thinkers are systems thinkers, even if they don’t use the lingo of feedback. Here’s a great example, in which Ben Franklin anticipates the American revolution, describing forces that could bring it about:

TO THE COMMITTEE OF CORRESPONDENCE IN MASSACHUSETTS

London, May 15, 1771.

GENTLEMEN,

I have received your favour of the 27th of February, with the journal of the House of Representatives, and copies of the late oppressive prosecutions in the Admiralty Court, which I shall, as you direct, communicate to Mr. Bollan, and consult with him on the most advantageous use to be made of them for the interest of the province.

I think one may clearly see, in the system of customs [import taxes] to be exacted in America by act of Parliament, the seeds sown of a total disunion of the two countries, though, as yet, that event may be at a considerable distance. The course and natural progress seems to be, first, the appointment of needy men as officers, for others do not care to leave England; then, their necessities make them rapacious, their office makes them proud and insolent, their insolence and rapacity make them odious, and, being conscious that they are hated, they become malicious; their malice urges them to a continual abuse of the inhabitants in their letters to administration, representing them as disaffected and rebellious, and (to encourage the use of severity) as weak, divided, timid, and cowardly. Government believes all; thinks it necessary to support and countenance its officers; their quarrelling with the people is deemed a mark and consequence of their fidelity; they are therefore more highly rewarded, and this makes their conduct still more insolent and provoking.

The resentment of the people will, at times and on particular incidents, burst into outrages and violence upon such officers, and this naturally draws down severity and acts of further oppression from hence. The more the people are dissatisfied, the more rigor will be thought necessary; severe punishments will be inflicted to terrify; rights and privileges will be abolished; greater force will then be required to secure execution and submission; the expense will become enormous; it will then be thought proper, by fresh exactions, to make the people defray it; thence, the British nation and government will become odious, the subjection to it will be deemed no longer tolerable; war ensues, and the bloody struggle will end in absolute slavery to America, or ruin to Britain by the loss of her colonies; the latter most probable, from America’s growing strength and magnitude.

….

I do not pretend to the gift of prophecy. History shows, that, by these steps, great empires have crumbled heretofore; and the late transactions we have so much cause to complain of show, that we are in the same train, and that, without a greater share of prudence and wisdom, than we have seen both sides to be possessed of, we shall probably come to the same conclusion….

With great esteem and respect, I have the honour to be, &c.

B. FRANKLIN.

This translates readily into a rich causal loop diagram (click the image to enlarge):

Franklin anticipates the revolution

My CLD here is basically a direct translation of the letter. That makes it sound a little more like a cycle of events, and less like interaction of quantities that can vary, than I would like. I think it could be refined somewhat by aggregating related concepts and rearranging a few links. For example, war is really just an escalation of violence, so one could simplify by treating the level of violence more generically.

The interesting thing about this diagram is that it’s all positive loops. Presumably the “prudence and wisdom” that Franklin noted would have created negative loops that would have stabilized the situation. What were they?

I bet a lot of the same dynamics are in the DOD Afghanistan counterinsurgency diagram.

Thanks to Dan Proctor for the original letter & idea.

The Vensim CLD is here if you want to play: franklin.mdl

Football physics & perception

SEED has a nice story on perception of curving shots in football (soccer).

The physics of the curving trajectory is interesting. In short, a light spinning ball can transition from a circular trajectory to a tighter spiral, surprising the goalkeeper.

What I find really interesting, though, is that goalkeepers don’t anticipate this.

But goalkeepers see hundreds of free kicks in practice on a daily basis. Surely they’d eventually adapt to bending shots, wouldn’t they?

… Elite professionals from some of the top soccer clubs in the world were shown simulations of straight and bending free kicks, which disappeared from view 10 to 12.5 meters from the goal. They then had to predict the path of the ball. The players were accurate for straight kicks, but they made systematic errors on bending shots. Instead of taking the curve into account, players tended to assume the ball would continue straight along the path it was following when it disappeared. Even more surprisingly, goalkeepers were no better at predicting the path of bending balls than other players. …

I think the interesting question is, could they be trained to anticipate this? It’s fairly easy for the goalie to observe the early trajectory of a ball, but due to the nonlinear transition to a new curvature, that’s not helpful. To guess whether the ball might suddenly take a wicked turn, one would have to judge its spin, which has to be much harder. My guess is that prediction is difficult, so the only option is to take robust action. In the case of the famous Carlos shot, one might guess that the goalie should have moved to cover the pole, even if he judged that the ball would be wide. (But who am I to say? I’m a lousy soccer player – I find 9 year olds to be stiff competition.)

SEED has another example:

I wrote about a similar problem on my blog earlier this year: How baseball fielders track fly balls. Researchers found that even when the ball is not spinning, outfielders don’t follow the optimum path to the ball—instead they constantly update their position in response to the ball’s motion.

At first this sounds like a classic lion-gazelle pursuit problem. But there’s one key difference: in pursuit problems I’ve seen, the opponent’s location is known, so the questions are all about physics and (maybe) strategic behavior. In soccer and baseball, at least part of the ball’s state (spin, for example) is at best poorly observed by the receiver. Therefore trajectories that appear to be suboptimal might actually be robust responses to imperfect measurement.

The problems faced by goalies and outfielders are in some ways much like those facing managers: what do you do, given imperfect information about a surprisingly nonlinear world?

SD & ABM: Don't throw stones; build bridges

There’s an old joke:

Q: Why are the debates so bitter in academia?

A: Because the stakes are so low.

The stakes are actually very high when models intersect with policy, I think, but sometimes academic debates come across as needlessly petty. That joke came to mind when a colleague shared this presentation abstract:

Pathologies of System Dynamics Models or “Why I am Not a System Dynamicst”

by Dr. Robert Axtell

So-called system dynamics (SD) models are typically interpreted as a summary or aggregate representation of a dynamical system composed of a large number of interacting entities. The high dimensional microscopic system is abstracted-notionally if not mathematically-into a ‘compressed’ form, yielding the SD model. In order to be useful, the reduced form representation must have some fidelity to the original dynamical system that describes the phenomena under study. In this talk I demonstrate formally that even so-called perfectly aggregated SD models will in general display a host of pathologies that are a direct consequence of the aggregation process. Specifically, an SD model can exhibit spurious equilibria, false stability properties, modified sensitivity structure, corrupted bifurcation behavior, and anomalous statistical features, all with respect to the underlying microscopic system. Furthermore, perfect aggregation of a microscopic system into a SD representation will generally be either not possible or not unique.

Finally, imperfectly aggregated SD models-surely the norm-can possess still other troublesome features. From these purely mathematical results I conclude that there is a definite sense in which even the best SD models are at least potentially problematical, if not outright mischaracterizations of the systems they purport to describe. Such models may have little practical value in decision support environments, and their use in formulating policy may even be harmful if their inadequacies are insufficiently understood.

In a technical sense, I agree with everything Axtell says.

However, I could equally well give a talk titled, “pathologies of agent models.” The pathologies might include ungrounded representation of agents, overuse of discrete logic and discrete time, failure to nail down alternative hypotheses about agent behavior, and insufficient exploration of sensitivity and robustness. Notice that these are common problems in practice, rather than problems in principle, because in principle one would always prefer a disaggregate representation. The problem is that we don’t build models in principle; we build them in practice. In reality resources – including data, time, computing, statistical methods, and decision maker attention – are limited. If you want more disaggregation, you’ve got to have less of something else.

Clearly there are times when an aggregate approach could be misleading. To leap from the fact that one can demonstrate pathological special cases to the idea that aggregate models are dangerous strikes me as a gross overstatement. Is the danger of aggregating agents really any greater than the danger of omitting feedback by reducing scope in order to enable modeling disaggregate agents? Hopefully this talk will illuminate some of the ways that one might think about whether a situation is dangerous or not, and therefore make informed choices of method and tradeoffs between scope and detail.

Also, models seldom inform policy directly; their influence occurs through improvement of mental models. Agent models could have a lot to offer there, but I haven’t seen many instances where authors developed the lingo to communicate insights to decision makers at their level. (Examples appreciated – any links?) That relegates many agent models to the same role as other black-box models: propaganda.

It’s strange that Axtell is picking on SD. Why not tackle economics? Most economic models have the same aggregation issues, plus they assume equilibrium and rationality from the start, so any representational problems with SD are greatly amplified. Plus the economic models are far more numerous and influential on policy. It’s like Axtell is bullying the wimpy kid in the class, because he’s scared to take on the big one who smokes at recess and shaves in 5th grade.

The sad thing about this confrontational framing is that SD and agent based modeling are a match made in heaven. At some level disaggregate models still need aggregate representations of agents; modelers could learn a lot from SD about good representation of behavior and dynamics, not to mention good habits like units checking that are seldom followed. At the same time, SD modelers could learn a lot about emergent phenomena and the limitations of aggregate representations. A good example of a non-confrontational approach, recognizing shades of gray:

Heterogeneity and Network Structure in the Dynamics of Diffusion: Comparing Agent-Based and Differential Equation Models

Hazhir Rahmandad, John Sterman

When is it better to use agent-based (AB) models, and when should differential equation (DE) models be used? Whereas DE models assume homogeneity and perfect mixing within compartments, AB models can capture heterogeneity across individuals and in the network of interactions among them. AB models relax aggregation assumptions, but entail computational and cognitive costs that may limit sensitivity analysis and model scope. Because resources are limited, the costs and benefits of such disaggregation should guide the choice of models for policy analysis. Using contagious disease as an example, we contrast the dynamics of a stochastic AB model with those of the analogous deterministic compartment DE model. We examine the impact of individual heterogeneity and different network topologies, including fully connected, random, Watts-Strogatz small world, scale-free, and lattice networks. Obviously, deterministic models yield a single trajectory for each parameter set, while stochastic models yield a distribution of outcomes. More interestingly, the DE and mean AB dynamics differ for several metrics relevant to public health, including diffusion speed, peak load on health services infrastructure, and total disease burden. The response of the models to policies can also differ even when their base case behavior is similar. In some conditions, however, these differences in means are small compared to variability caused by stochastic events, parameter uncertainty, and model boundary. We discuss implications for the choice among model types, focusing on policy design. The results apply beyond epidemiology: from innovation adoption to financial panics, many important social phenomena involve analogous processes of diffusion and social contagion. (Paywall; full text of a working version here)

Details, in case anyone reading can attend – report back here!

Thursday, October 21 at 6:00 – 8:00 PM ** New Time **

Networking 6:00 – 6:45 PM (light refreshments) Presentation 6:45 – 8:00 PM Free and open to the public

** NEW Location **

Booz Allen Hamilton – Ballston-Virginia Square

3811 N. Fairfax Drive, Suite 600

Arlington, VA 22203

(703) 816-5200

Between Virginia Square and Ballston Metro stations, between Pollard St.

and Nelson St.

On-street parking is available, especially on 10th Street near the Arlington Library.

There will be a Booz Allen representative at the front of the building until 7:00 to greet and escort guests, or call 703-627-5268 to be let in.

RSVP by e-mail to Nicholas Nahas, nahas_nicholas@bah.com<mailto:nahas_nicholas@bah.com>, in order to have a rough count of attendees prior to the meeting. Come anyway even if you do not RSVP.

By METRO:

Take the Orange Line to the Ballston station. Exit Metro Station, walk towards the IHOP (right on N. Fairfax) continue for approximately 2-3 blocks. Booz Allen Hamilton (3811 N. Fairfax Dr. Suite 600) is on the left between Pollard St. and Nelson St.

OR Take the Orange Line to the Virginia Square station. Exit Metro Station and go left and walk approximately 2-3 blocks. Booz Allen Hamilton (3811 N. Fairfax Dr. Suite 600) is on the right between Pollard St. and Nelson St.

Brian Eno, meet Stafford Beer

Brian Eno reflects on feedback and self-organization in musical composition, influenced by the organization of complex systems in Stafford Beer’s The Brain of the Firm.

Stafford Beer was a member of the cybernetics thread of systems thought (if that sounds baffling, read George Richardson’s excellent book on the evolution of thinking about systems).

Interactive diagrams – obesity dynamics

Food-nutrition-health-exercise-energy interactions are an amazing nest of positive feedbacks, with many win-win opportunities, but more on that another time.

Instead, I’m hoisting an interesting influence diagram about obesity from the comments. At first glance, it’s just another plate of spaghetti.

ForesightObesity

But when you follow the link (do it now), there’s an interesting innovation: the diagram is interactive. You can zoom, scroll, and highlight particular sectors and dynamics. There’s some narrative here and here. (Update: the interactive link seems to be down, but the diagram is still here: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/295153/07-1177-obesity-system-atlas.pdf)

It took me a while to decide whether I’d call this a causal loop diagram or not. I think the primary distinction between a CLD and other kinds of mindmaps or process diagrams is the use of variables. On a CLD, each label represents a quantity that can vary, with a definite direction – TV Watching, Stress, Use of Medicines. Items on other kinds of diagrams might represent events or fuzzier constellations of concepts. This diagram doesn’t have link polarities (too bad) or loop polarities (which would be pretty incomprehensible anyway), but many other CLDs also avoid such labels for simplicity.

I think there’s a lot of potential for further exploration of this idea. There’s a lot you could do to relate structure to behavior, or at least to explain the rationale for structure (both shortcomings of the diagram). Each link, for example, could have its tale revealed when clicked, and key loops could be animated individually, with stories told. Drill-down could be extended to provide links between top-level subsystem relationships and more microscopic views.

I think huge diagrams like the one above are always going to be overwhelming to a layperson. Also, it’s hard to make even a small CLD good, so making a big one really accurate is tough. Therefore, I’d rather see advanced CLD presentations used to improve the communication of simpler stories, with a few loops. However, big or small, there might be many common technological benefits from dedicated diagramming software.

Gallatin County's Zoning Enforcement Trap

I’m playing a big role in a local effort to get the regulations of our zoning district enforced in the case of an egregious violation. Our planning and zoning commission’s habit, and apparent preference in this case, is not to enforce. Instead, it is proposed to enable the violation through a PUD amendment, and issue a trivial fine ($200, or 0.2% of the stated value of the structure).

Unfortunately, this proposal is illegal, because it contradicts existing covenants and a variety of goals and specific provisions of our General Plan and Zoning Regulation. This action might make sense if it were a naked political ploy to undermine the zoning through administrative rather than legislative means, which I hope is not the case. I think it is more likely an effort to “play nice” with violators and to avoid costly enforcement action.

If so, the resulting weak enforcement posture is a short-sighted avoidance of conflict, that encourages far more problems in the long run. As the diagram below illustrates, backing down on the case at hand solves the immediate problem, but has terrible consequences.

Enforcement Dynamics

  • The precedent for non-enforcement and amendments to legalize violations erodes the legal basis for future enforcement actions.
  • Accommodation creates an expectation of forgiveness, encouraging owners and builders to violate in the future.
  • Exceptions created to accommodate violations make planning documents and title histories more complex, creating more opportunities for errors.

These side effects of lax enforcement accumulate. As violations mount, time that could be spent on productive activity (ensuring a thorough permitting process, or revising zoning regulations to clarify standards and streamline processes) gets squeezed out by time wasted on enforcement.

These reinforcing feedbacks create a deadly trap, into which the unsuspecting can easily step. Once triggered, the vicious cycle creates more pressure to relax enforcement standards, capturing the county in an undesirable equilibrium with many violations and no meaningful enforcement. Ultimately, the citizens (who initiated the zoning district) suffer from the side effects of density granted to violators, that is unavailable to those who comply with the law.

Fortunately, with a little fortitude, the process can be reversed. A single forceful enforcement action has a salutary effect on expectations, stemming the tide of violations and freeing up time for the improvement of regulations. There’s still the hangover of side effects of past accommodation to contend with, but surely the withdrawal is better than the addiction to accommodation.

Cap & trade is dead. Long live cap & trade?

Democrats have pulled the plug on a sweeping energy bill this year. There is no heir apparent. This is not cause for panic. In climate, as in education, there are no emergencies.

However, the underlying reasons may be cause for panic. It seems that voters are unwilling to accept any policy that will significantly raise the price of emissions. Given that price is a predominant information carrier in our economy, other polices are unlikely to work efficiently, absent a price signal. That leaves us in a bit of a pickle. What to do?

If you don’t want to buck public opinion, advise the people to invest in (then pray for) a technological miracle. Ask yourself, “Do I feel lucky?” It might even work.

Alternatively, you might conclude that the public hasn’t quite grasped the nature of the problem – that wait and see is not a good policy in systems with long delays. But then you’d be accused of scientism, for the equivalent of challenging the efficient market hypothesis or the notion that the customer is always right. That’s rather puzzling, given that there’s direct evidence that people don’t intuitively appreciate the dynamics of accumulation, and that snowstorms in the East cause half of Americans to question the reality of climate change.

The anti-scientism, pro-technology crowd takes opposition to meaningful mitigation policy as a sure sign that the public is on to something. The wisdom of crowds is powerful when there’s diverse information and rapid feedback, as in price discovery through a market. But it has a pretty disastrous history in the runup to bubbles and other catastrophes, as we’ve recently seen. Surely there are some legitimate worries about current climate proposals (I’ve expressed a number here), but it doesn’t follow that pricing emissions is a bad decision.

So, what’s a modeler to do? Opening up political debates is a good idea, though not quite in the way that I think proponents intend. We already have plenty of political debates. The problem is that they tend to lack ready access to scientific or other information that can be agreed upon or at least presented in a way that permits testing of hypotheses against data or evaluation of decisions against contingencies. That means that questions of values and distribution of benefits (which politics is rightfully about) get mixed up with muddled thinking about science, economics, and social system dynamics.

The solution typically proposed is to open up science and models to more public scrutiny. That’s a good idea for a variety of reasons, but by itself it’s a losing proposition for scientists- they get all the criticism, and the public process doesn’t assimilate much of their insights. What’s needed is a fair exchange, where everyone shows their hands. Scientists make their stuff accessible, and in return participants in policy debates actually use it, and additionally submit to formalization of their arguments to facilitate shared understanding and testing.

Coming back to cap & trade, I don’t see that the major political players are willing to do that. Following a successful round of multi-stakeholder workshops that brought a systems perspective to conversations about climate policy, funded by the petro industry in California, we spent a fair amount of time marketing the idea of a model-assisted deliberation process targeted at shared design of federal climate policy. Lobbyists at some of the big stakeholders told us very forthrightly that they were unwilling to engage in any process with an outcome that they couldn’t predict and control.

In an environment where everyone’s happy with their own entrenched position, their isn’t much hope for a good solution to emerge. The only solution I see is to make an end run around the big players, and go straight to the public with better information, in order to expand the set of things they’ll accept. I hope there’s time for that to work.

Policy Resistance – Immigration & Prohibition

Complex systems find many ways of resisting or evading pressures, resulting in policy failure, backlashes, whack-a-mole games and other unintended consequences. Some great examples just wandered by my desk:

Via Economist’s View:

Immigration reform has a long history of unintended consequences: More than two decades of increased enforcement since the passage of the Immigration Reform and Control Act of 1986 has done little to reduce the number of illegal immigrants. In fact, it seems to have increased their numbers. …

Princeton University sociologist Douglas Massey pointed out … that measures to secure the border seemed to produce almost the opposite of what was intended. … With increasing border enforcement, workers who used to shuttle between jobs in California or Texas and home in Zacatecas or Michoacán simply began to stay put and sent for their families, becoming permanent, if sometimes reluctant, residents. According to Massey, post-IRCA border enforcement may have increased the size of the permanent Mexican population in the United States by a factor of nearly four.

From a great article on Wayne Wheeler, The Man Who Turned Off the Taps, in Smithsonian:

But for all his political might, Wheeler could not do what he and all the other Prohibitionists had set out to do: they could not purge alcoholic beverages from American life. Drinking did decline at first, but a combination of legal loopholes, personal tastes and political expediency conspired against a dry regime.

As declarative as the 18th Amendment was—forbidding “the manufacture, sale, or transportation of intoxicating liquors”—the Volstead Act allowed exceptions. You were allowed to keep (and drink) liquor you had in your possession as of January 16, 1920; this enabled the Yale Club in New York, for instance, to stockpile a supply large enough to last the full 14 years that Prohibition was in force. Farmers and others were allowed to “preserve” their fruit through fermentation, which placed hard cider in cupboards across the countryside and homemade wine in urban basements. “Medicinal liquor” was still allowed, enriching physicians (who generally charged by the prescription) and pharmacists (who sold such “medicinal” brands as Old Grand-Dad and Johnnie Walker). A religious exception created a boom in sacramental wines, leading one California vintner to sell communion wine—legally—in 14 different varieties, including port, sherry, tokay and cabernet sauvignon.

By the mid-’20s, those with a taste for alcohol had no trouble finding it, especially in the cities of the East and West coasts and along the Canadian border. At one point the New York police commissioner estimated there were 32,000 illegal establishments selling liquor in his city. In Detroit, a newsman said, “It was absolutely impossible to get a drink…unless you walked at least ten feet and told the busy bartender what you wanted in a voice loud enough for him to hear you above the uproar.” Washington’s best-known bootlegger, George L. Cassiday (known to most people as “the man in the green hat”), insisted that “a majority of both houses” of Congress bought from him, and few thought he was bragging.

Worst of all, the nation’s vast thirst gave rise to a new phenomenon—organized crime, in the form of transnational syndicates that controlled everything from manufacture to pricing to distribution. A corrupt and underfunded Prohibition Bureau couldn’t begin to stop the spread of the syndicates, which considered the politicians who kept Prohibition in place their greatest allies. Not only did Prohibition create their market, it enhanced their profit margins: from all the billions of gallons of liquor that changed hands illegally during Prohibition, the bootleggers did not pay, nor did the government collect, a single penny of tax.

The prohibition article also poses an interesting puzzle. If prohibition was more or less quickly and broadly unpopular, how did it get passed by such landslide margins in the first place? I can’t believe that ignorance of the possible outcome was universal, so there must have been some powerful positive feedback behind the initial passage of the policy. Perhaps it was a tipping point effect: once a vote becomes sufficiently lopsided, fewer and fewer politicians want to be on the losing side of a landslide vote, so they join the herd. A modern analogy might be the post-9/11 authorization of the Iraq war.