Tom – Page 65

Writing a good system dynamics paper II

It’s SD conference paper review time again. Last year I took notes while reviewing, in an attempt to capture the attributes of a good paper. A few additional thoughts:

No model is perfect, but it pays to ask yourself, will your model stand up to critique?
Model-data comparison is extremely valuable and too seldom done, but trivial tests are not interesting. Fit to data is a weak test of model validity; it’s often necessary, but never sufficient as a measure of quality. I’d much rather see the response of a model to a step input or an extreme conditions test than a model-data comparison. It’s too easy to match the model to the data with exogenous inputs, so unless I see a discussion of a multi-faceted approach to validation, I get suspicious. You might consider how your model meets the following criteria:
- Do decision rules use information actually available to real agents in the system?
- Would real decision makers agree with the decision rules attributed to them?
- Does the model conserve energy, mass, people, money, and other physical quantities?
- What happens to the behavior in extreme conditions?
- Do physical quantities always have nonnegative values?
- Do units balance?
If you have time series output, show it with graphs – it takes a lot of work to “see” the behavior in tables. On the other hand, tables can be great for other comparisons of outcomes.
If all of your graphs show constant values, linear increases (ramps), or exponentials, my eyes glaze over, unless you can make a compelling case that your model world is really that simple, or that people fail to appreciate the implications of those behaviors.
Relate behavior to structure. I don’t care what happens in scenarios unless I know why it happens. One effective way to do this is to run tests with and without certain feedback loops or sectors of the model active.
Discuss what lies beyond the boundary of your model. What did you leave out and why? How does this limit the applicability of the results?
If you explore a variety of scenarios with your model (as you should), introduce the discussion with some motivation, i.e. why are the particular scenarios tested important, realistic, etc.?
Take some time to clean up your model diagrams. Eliminate arrows that cross unnecessarily. Hide unimportant parameters. Use clear variable names.
It’s easiest to understand behavior in deterministic experiments, so I like to see those. But the real world is noisy and uncertain, so it’s also nice to see experiments with stochastic variation or Monte Carlo exploration of the parameter space. For example, there are typically many papers on water policy in the ENV thread. Water availability is contingent on precipitation, which is variable on many time scales. A system’s response to variation or extremes of precipitation is at least as important as its mean behavior.
Modeling aids understanding, which is intrinsically valuable, but usually the real endpoint of a modeling exercise is a decision or policy change. Sometimes, it’s enough to use the model to characterize a problem, after which the solution is obvious. More often, though, the model should be used to develop and test decision rules that solve the problem you set out to conquer. Show me some alternative strategies, discuss their limitations and advantages, and describe how they might be implemented in the real world.
If you say that an SD model can’t predict or forecast, be very careful. SD practitioners recognized early on that forecasting was often a fool’s errand, and that insight into behavior modes for design of robust policies was a worthier goal. However, SD is generally about building good dynamic models with appropriate representations of behavior and so forth, and good models are a prerequisite to good predictions. An SD model that’s well calibrated can forecast as well as any other method, and will likely perform better out of sample than pure statistical approaches. More importantly, experimentation with the model will reveal the limits of prediction.
It never hurts to look at your paper the way a reviewer will look at it.

Another look at inadequate Copenhagen pledges

Joeri Rogelj and others argue that Copenhagen Accord pledges are paltry in a Nature Opinion,

Current national emissions targets can’t limit global warming to 2 °C, calculate Joeri Rogelj, Malte Meinshausen and colleagues — they might even lock the world into exceeding 3 °C warming.

Nations will probably meet only the lower ends of their emissions pledges in the absence of a binding international agreement

Nations can bank an estimated 12 gigatonnes of Co2 equivalents surplus allowances for use after 2012

Land-use rules are likely to result in further allowance increases of 0.5 GtCO2-eq per year

Global emissions in 2020 could thus be up to 20% higher than today

Current pledges mean a greater than 50% chance that warming will exceed 3°C by 2100

If nations agree to halve emissions by 2050, there is still a 50% chance that warming will exceed 2°C and will almost certainly exceed 1.5°C

Via Nature’s Climate Feedback, Copenhagen Accord – missing the mark.

Computer models running the EU? Eruptions, models, and clueless reporting

The EU airspace shutdown provides yet another example of ignorance of the role of models in policy:

Computer Models Ruining EU?

Flawed computer models may have exaggerated the effects of an Icelandic volcano eruption that has grounded tens of thousands of flights, stranded hundreds of thousands of passengers and cost businesses hundreds of millions of euros. The computer models that guided decisions to impose a no-fly zone across most of Europe in recent days are based on incomplete science and limited data, according to European officials. As a result, they may have over-stated the risks to the public, needlessly grounding flights and damaging businesses. “It is a black box in certain areas,” Matthias Ruete, the EU’s director-general for mobility and transport, said on Monday, noting that many of the assumptions in the computer models were not backed by scientific evidence. European authorities were not sure about scientific questions, such as what concentration of ash was hazardous for jet engines, or at what rate ash fell from the sky, Mr. Ruete said. “It’s one of the elements where, as far as I know, we’re not quite clear about it,” he admitted. He also noted that early results of the 40-odd test flights conducted over the weekend by European airlines, such as KLM and Air France, suggested that the risk was less than the computer models had indicated. – Financial Times

Counting emissions – pledges, airplanes, volcanoes

Pew Climate has a nice summary of attempts to add up country emissions, including Climate Interactive‘s.

Somewhere in the blogosphere I ran across this nice infographic contrasting European aviation and Icelandic volcano emissions:

Cascading failures in interconnected networks

Wired covers a new article in Nature, investigating massive failures in linked networks.

The interesting thing is that feedback between the connected networks destabilizes the whole:

“When networks are interdependent, you might think they’re more stable. It might seem like we’re building in redundancy. But it can do the opposite,” said Eugene Stanley, a Boston University physicist and co-author of the study, published April 14 in Nature.

…

The interconnections fueled a cascading effect, with the failures coursing back and forth. A damaged node in the first network would pull down nodes in the second, which crashed nodes in the first, which brought down more in the second, and so on. And when they looked at data from a 2003 Italian power blackout, in which the electrical grid was linked to the computer network that controlled it, the patterns matched their models’ math.

Wired

Interestingly, the interconnection alters the relationship between network structure (degree distribution) and robustness:

Surprisingly, a broader degree distribution increases the vulnerability of interdependent networks to random failure, which is opposite to how a single network behaves.

Nature

Chalk one up for counter-intuitive behavior of complex systems.

What looks like last year’s version of the paper is on arXiv.

Calculators of the global climate deal, unite!

Read all about it at Climate Interactive.

NUMMI – an innovation killed by its host's immune system?

This American Life had a great show on the NUMMI car plant, a remarkable joint venture between Toyota and GM. It sheds light on many of the reasons for the decline of GM and the American labor movement. More generally, it’s a story of a successful innovation that failed to spread, due to policy resistance, inability to confront worse-before-better behavior and other dynamics.

I noticed elements of a lot of system dynamics work in manufacturing. Here’s a brief reading list:

Workers fear improving themselves out of a job
TQM improvements can undercut themselves when they interact with existing organizational systems
Firefighting is a trap that keeps lines running in a low-productivity state
Engineering and production don’t get along (I think there’s more on this in Daniel Kim’s thesis)
Systems have trouble making far-sighted decisions when confronted with worse-before-better behavior (but there might be good reasons)

Montana's climate future

A selection of data and projections on past and future climate in Montana:

Pederson et al. (2010) A century of climate and ecosystem change in Western Montana: what do temperature trends portend? Climatic Change 98:133-154. It’s hard to read precisely off the graph, but there have been significant increases in maximum and minimum temperatures, with the greatest increases in the minimums and in winter – exactly what you’d expect from a change in radiative properties. As a result the daily temperature range has shrunk slightly and there are fewer below freezing and below zero days. That last metric is critical, because it’s the severe cold that controls many forest pests. There’s much more on this in a poster.

Not every station shows a trend – the figure above contrasts Bozeman (purple, strong trend) with West Yellowstone (orange, flat). The Bozeman trend is probably not an urban heat island effect – surfacestations.org thinks it’s a good site, and White Sulphur (a nice sleepy town up the road a piece) is about the same. The red line is an ensemble of simulations (GISS, CCSM & ECHAM5) from climexp.knmi.nl, projected into the future with A1B forcings (i.e., a fairly high emissions trajectory). I interpolated the data to latitude 47.6, longitude -110.9 (roughly my house, near Bozeman). Simulated temperature rises about 4C, while precipitation (green) is almost unmoved. If that came true, Montana’s future climate might be a lot like current central Utah.

The figure above – from John W. Williams, Stephen T. Jackson, and John E. Kutzbach. Projected distributions of novel and disappearing climates by 2100 AD. PNAS, vol. 104 no. 14 – shows global grid points that have no neighbors within 500km that now have a climate like what the future might bring. In panel C (disappearing climates with the high emissions A2 scenario), there’s a hotspot right over Montana. Presumably that’s loss of today’s high altitude ecosystems. As it warms up, climate zones move uphill, but at the top of mountains there’s nowhere to go. That’s why pikas may be in trouble.

MT Field Guide

Sea Level Roundup

Realclimate has Martin Vermeer’s reflections on the making of his recent sea level paper with Stefan Rahmstorf. At some point I hope to post a replication of that study, in a model with the Grinsted and Rahmstorf 2007 structures, but I haven’t managed to replicate it yet. The problem may be that I haven’t yet tackled the reservoir storage issue.

At Nature Reports, Olive Heffernan introduces several sea level articles. Rahmstorf contrasts the recent set of semi-empirical models, predicting sea level of a meter or more this century, with the AR4 finding. Lowe and Gregory wonder if the semi-empirical models are really seeing enough of the dynamic ice signal to have predictive power, and worry about overadaptation to high scenarios. Mark Schrope reports on underadaptation – vulnerable developments in Florida. Mason Inman reports on ecological engineering, a softer approach to coastal defense.

Who eats the risk?

From the Asilomar geoengineering conference, via WorldChanging:

Lesson two: Nobody has any clear idea how to resolve the inequalities inherent in geoengineering. One of the most quoted remarks at the conference came from Pablo Suarez, the associate director of programs with the Red Cross/Red Crescent Climate Centre, who asked during one plenary session, “Who eats the risk?” In Suarez’s view, geoengineering is all about shifting the risk of global warming from rich nations — i.e., those who can afford the technologies to manipulate the climate — to poor nations. Suarez admitted that one way to resolve this might be for rich nations to pay poor nations for the damage caused by, say, shifting precipitation patterns. But that conjured up visions of Bangladeshi farmers suing Chinese geoengineers for ruining their rice crop — a legalistic can of worms that nobody was willing to openly explore.

If geoengineering is a for-profit operation, it presumably also involves the public bearing the risk of private acts, because investors aren’t likely to have an appetite for the essentially unlimited liability.