Don't just do something, stand there! Reflections on the counterintuitive behavior of complex systems, seen through the eyes of System Dynamics, Systems Thinking and simulation.
Grist covers a detailed report on the rebound effect, which recently appeared at ElectricityPolicy.com (pdf from NRDC). The report discusses a wide range of rebound arguments, basically concluding that rebounds are not a big deal.
Some of the reasons derive from the microeconomic effects of efficiency improvements. For example, improving the efficiency of light bulbs makes light services cheaper. But user’s don’t immediately increase lighting in proportion to the cost reduction, because their demand for lighting is saturated: there are only so many fixtures in a house, hours in the day requiring light, etc. Similarly, the elasticity of dirty dish production with respect to the energy cost of running a dishwasher is pretty darn low. This is reminiscent of the dynamics of process improvement at Analog Devices, where TQM improved productivity, but the company had a hard time translating that to expansion of its market niche in the short term.
I think the report underweights the long term effects of efficiency though. Efficiency increases contribute to aggregate productivity growth in the economy (more than you’d expect, if you believe that agency problems and other market failures create a bias toward overuse of energy). With wealth comes an expansion of energy use, hence the boom in such energy hogs as undercounter freezers and wine chillers, countering Energy Star improvement in refrigeration. However, this is not really an efficiency problem; it’s a progress problem, and it brings welfare benefits along with the added energy (at least until you get to the absurd margin).
The report cites an Energy Policy survey of empirical estimates:
Improvements in energy efficiency make energy services cheaper, and therefore encourage increased consumption of those services. This so-called direct rebound effect offsets the energy savings that may otherwise be achieved. This paper provides an overview of the theoretical and methodological issues relevant to estimating the direct rebound effect and summarises the empirical estimates that are currently available. The paper focuses entirely on household energy services, since this is where most of the evidence lies and points to a number of potential sources of bias that may lead the effect to be overestimated. For household energy services in the OECD, the paper concludes that the direct rebound effect should generally be less than 30%. doi:10.1016/j.enpol.2008.11.026
Sadly, a press release for related studies from the same research group spins this as a catastrophe:
‘Rebound Effects’ Threaten Success of UK Climate Policy
This is really only a catastrophe for a politician foolish enough to try to set and hit a hard emissions target, with efficiency mandates as the only measure for achieving it. As soon as you have any course correction (i.e. negative feedback) built into your policies, like an adaptive carbon tax or cap & trade system (the latter being the less stable option), the catastrophe goes away. The real catastrophe is failing to price GHG emissions and other externalities due to misperceptions about efficiency.
The real bottom line for rebound effects should be, “who cares?” If rebound effects are large, efficiency programs have small energy effects, but potentially large welfare improvements (if you accept that there are energy market failures tending towards overconsumption), and emissions pricing has large energy effects, because high rebound implies high price elasticity. If rebound effects are small, efficiency programs work and emissions pricing is a good way to collect taxes. Neither condition is a reason to avoid efficiency or emissions pricing, though emissions pricing is the preferable way to proceed.
Eli Pariser argues that “filter bubbles” are bad for us and bad for democracy:
As web companies strive to tailor their services (including news and search results) to our personal tastes, there’s a dangerous unintended consequence: We get trapped in a “filter bubble” and don’t get exposed to information that could challenge or broaden our worldview.
Filter bubbles are close cousins of confirmation bias, groupthink, polarization and other cognitive and social pathologies.
As confidence in an idea grows, the delay in recognition (or frequency of outright rejection) of anomalous information grows larger. As a result, confidence in the idea – flat earth, 100mpg carburetor – can grow far beyond the level that would be considered reasonable, if contradictory information were recognized.
The dynamics resulting from this and other positive feedbacks play out in many spheres. Wittenberg & Sterman give an example:
The dynamics generated by the model resemble the life cycle of intellectual fads. Often a promising new idea rapidly becomes fashionable through excessive optimism, aggressive marketing, media hype, and popularization by gurus. Many times the rapid influx of poorly trained practitioners, or the lack of established protocols and methods, causes expectations to outrun achievements, leading to a backlash and disaffection. Such fads are commonplace, especially in (quack) medicine and most particularly in the world of business, where “new paradigms” are routinely touted in the pages of popular journals of management, only to be displaced in the next issue by what many business people have cynically come to call the next “flavor of the month.”
Typically, a guru proposes a new theory, tool, or process promising to address persistent problems facing businesses (that is, a new paradigm claiming to solve the anomalies that have undermined the old paradigm.) The early adopters of the guru’s method spread the word and initiate some projects. Even in cases where the ideas of the guru have little merit, the energy and enthusiasm a team can bring to bear on a problem, coupled with Hawthorne and placebo effects and the existence of “low hanging fruit” will often lead to some successes, both real and apparent. Proponents rapidly attribute these successes to the use of the guru’s ideas. Positive word of mouth then leads to additional adoption of the guru’s ideas. (Of course, failures are covered up and explained away; as in science there is the occasional fraud as well.) Media attention further spreads the word about the apparent successes, further boosting the credibility and prestige of the guru and stimulating additional adoption.
As people become increasingly convinced that the guru’s ideas work, they are less and less likely to seek or attend to disconfirming evidence. Management gurus and their followers, like many scientists, develop strong personal, professional, and financial stakes in the success of their theories, and are tempted to selectively present favorable and suppress unfavorable data, just as scientists grow increasingly unable to recognize anomalies as their familiarity with and confidence in their paradigm grows. Positive feedback processes dominate the dynamics, leading to rapid adoption of those new ideas lucky enough to gain a sufficient initial following. …
The wide range of positive feedbacks identified above can lead to the swift and broad diffusion of an idea with little intrinsic merit because the negative feedbacks that might reveal that the tools don’t work operate with very long delays compared to the positive loops generating the growth. …
For filter bubbles, I think the key positive loops are as follows:
Loops R1 are the user’s well-worn path. We preferentially visit sites presenting information (theory x or y) in which we have confidence. In doing so, we consider only a subset of all information, building our confidence in the visited theory. This is a built-in part of our psychology, and to some extent a necessary part of the process of winnowing the world’s information fire hose down to a usable stream.
Loops R2involve the information providers. When we visit a site, advertisers and other observers (Nielsen) notice, and this provides the resources (ad revenue) and motivation to create more content supporting theory x or y. This has also been a part of the information marketplace for a long time.
R1 and R2 are stabilized by some balancing loops (not shown). Users get bored with an all-theory-y diet, and seek variety. Providers seek out controversy (real or imagined) and sensationalize x-vs-y battles. As Pariser points out, there’s less scope for the positive loops to play out in an environment with a few broad media outlets, like city newspapers. The front page of the Bozeman Daily Chronicle has to work for a wide variety of readers. If the paper let the positive loops run rampant, it would quickly lose half its readership. In the online world, with information customized at the individual level, there’s no such constraint.
Individual filtering introduces R3. As the filter observes site visit patterns, and preferentially serves up information consistent with past preferences. This introduces a third set of reinforcing feedback processes, as users begin to see what they prefer, they also learn to prefer what they see. In addition, on Facebook and other social networking sites every person is essentially a site, and people include one another in networks preferentially. This is another mechanism implementing loop R1 – birds of a feather flock together and share information consistent with their mutual preferences, and potentially following one another down conceptual rabbit holes.
The result of the social web and algorithmic filtering is to upset the existing balance of positive and negative feedback. The question is, were things better before, or are they better now?
I’m not exactly sure how to tell. Presumably one could observe trends in political polarization and duration of fads for an indication of the direction of change, but that still leaves open the question of whether we have more or less than the “optimal” quantity of pet rocks, anti-vaccine campaigns and climate skepticism.
My suspicion is that we now have too much positive feedback. This is consistent with Wittenberg & Sterman’s insight from the modeling exercise, that the positive loops are fast, while the negative loops are weak or delayed. They offer a prescription for that,
The results of our model suggest that the long-term success of new theories can be enhanced by slowing the positive feedback processes, such as word of mouth, marketing, media hype, and extravagant claims of efficacy by which new theories can grow, and strengthening the processes of theory articulation and testing, which can enhance learning and puzzle-solving capability.
In the video, Pariser implores the content aggregators to carefully ponder the consequences of filtering. I think that also implies more negative feedback in algorithms. It’s not clear that providers have an incentive to do that though. The positive loops tend to reward individuals for successful filtering, while the risks (e.g., catastrophic groupthink) accrue partly to society. At the same time, it’s hard to imagine a regulatory that does not flirt with censorship.
Absent a global fix, I think it’s incumbent on individuals to practice good mental hygiene, by seeking diverse information that stands some chance of refuting their preconceptions once in a while. If enough individuals demand transparency in filtering, as Pariser suggests, it may even be possible to gain some local control over the positive loops we participate in.
I’m not sure that goes far enough though. We need tools that serve the social equivalent of “strengthening the processes of theory articulation and testing” to improve our ability to think and talk about complex systems. One such attempt is the “collective intelligence” behind Climate Colab. It’s not quite Facebook-scale yet, but it’s a start. Semantic web initiatives are starting to help by organizing detailed data, but we’re a long way from having a “behavioral dynamic web” that translates structure into predictions of behavior in a shareable way.
It might be subtitled, “Better Lies,” a reference to modeling as the pursuit of better stories about the world, which remain never quite true (a variation on the famous Box quote, “All models are wrong but some are useful.”). A few nice points that I picked out along the way,
All thinking, even about the future, is retrospective.
Big Data is Big Dumb, because we’re collecting more and more detail about a limited subset of reality, and thus suffer from sampling and “if your only tool is a hammer …” bias.
A crucial component of a modeling approach is a “bullshit detector” – reality checks that identify problems at various levels on the ladder of inference.
Model design is more than software engineering.
Often the modeling process is a source of key insights, and you don’t even need to run the model.
Modeling is a social process.
Coming back to the comment,
I think one of the greatest values of a model is that it can bring you to the point where you say “There isn’t any way to build a model within this methodology that is not self-contradicting. Therefore everyone in this room is contradicting themselves before they even open their mouths.”
I think that’s close to what Dana Meadows was talking about when she placed paradigms and transcendence of paradigms on the list of places to intervene in systems.
It reminds me of Gödel’s incompleteness theorems. With that as a model, I’d argue that one can construct fairly trivial models that aren’t self-contradictory. They might contradict a lot of things we think we know about the world, but by virtue of their limited expressiveness remain at least true to themselves.
Going back to the elasticity example, if I assert that oilConsumption = oilPrice^epsilon, there’s no internal contradiction as long as I use the same value of epsilon for each proposition I consider. I’m not even sure what an internal contradiction would look like in such a simple framework. However, I could come up with a long list of external consistency problems with the model: dimensional inconsistency, lack of dynamics, omission of unobserved structure, failure to conform to data ….
In the same way, I would tend to argue that general equilibrium is an internally consistent modeling paradigm that just happens to have relatively little to do with reality, yet is sometimes useful. I suppose that Frank Ackerman might disagree with me, on the grounds that equilibria are not necessarily unique or stable, which could raise an internal contradiction by violating the premise of the modeling exercise (welfare maximization).
Once you step beyond models with algorithmically simple decision making (like CGE), the plot thickens. There’s Condorcet’s paradox and Arrow’s impossibility theorem, the indeterminacy of Arthur’s El Farol bar problem, and paradoxes of zero discount rates on welfare.
Maybe the challenge is that the universe is self-referential and full of paradoxes and irreconcilable paradigms. Therefore as soon as we attempt to formalize our understanding of such a mess, either with nontrivial models, or trivial models assisting complex arguments, we are dragged into the quagmire of self-contradiction.
Personally, I’m not looking for the cellular automaton that runs the universe. I’m just hoping for a little feedback control on things that might make life on earth a little better. Maybe that’s a paradoxical quest in itself.
If a global oil shock reduces supply 10%, the price of crude will rise to $20,000/barrel, with fuel expenditures consuming more than the entire GDP of importing nations.
At least that’s what you’d predict if you think the price elasticity of oil demand is about -0.02. I saw that number in a Breakthrough post, citing Kevin Drum, citing Early Warning, citing IMF. It’s puzzling that Breakthrough is plugging small price elasticities here, when their other arguments about the rebound effect require elasticities to have large magnitudes. Continue reading “Elasticity contradictions”
A future where everything goes right for nuclear power, with advancing technology driving down costs, making reactors a safe and ubiquitous energy source, and providing a magic bullet for climate change, might bring other surprises.
For example, technology might also make supersonic cruise missiles cheap and ubiquitous.
The Fukushima operators appear to be hanging in there. But imagine how they’d be coping if someone fired a missile at them once in a while.
Fortunately, reactors today are mostly in places where peace and rule of law prevail.
But peace and good governance aren’t exactly the norm in places where emissions are rising rapidly, or the poor need energy.
Building lots of nuclear power plants is ultimately a commitment to peace, or at least acceptance of rather dreadful consequences of war (not necessarily war with nuclear weapons, but war with conventional weapons turning nuclear reactors into big dirty bombs).
One would hope that abundant, clean energy would reduce the motivation to blow things up, but how much are we willing to gamble on that?
Last week, on April 13, 2011, President Obama gave all Democrats and all progressives a remarkable gift. Most of them barely noticed. They looked at the President’s speech as if it were only about budgetary details. But the speech went well beyond the budget. It went to the heart of progressive thought and the nature of American democracy, and it gave all progressives a model of how to think and talk about every issue.
I’m definitely in the “barely noticed” category. The interesting thing, George argues, is that the speech is really about systems. Part concerns a system of values:
The policy topic happened to be the budget, but he called it “The Country We Believe In” for a reason. The real topic was how the progressive moral system defines the democratic ideals America was founded on, and how those ideals apply to specific issues.
More interesting to me, another key theme is systems in the “systems thinking” sense:
Systems Thinking
President Obama, in the same speech, laid the groundwork for another crucial national discussion: systems thinking, which has shown up in public discourse mainly in the form of “systemic risk” of the sort that led to the global economic meltdown. The president brought up systems thinking implicitly, at the center of his budget proposal. He observed repeatedly that budget deficits and “spending” do not occur in isolation. The choice of what to cut and what to keep is a matter of factors external to the budget per se. Long-term prosperity, economic recovery, and job creation, he argued, depend up maintaining “investments” — investments in infrastructure (roads, bridges, long-distance rail), education, scientific research, renewable energy, and so on. The maintenance of American values, he argued, is outside of the budget in itself, but is at the heart of the argument about what to cut. The fact is that the rich have gotten rich because of the government — direct corporate subsidies, access to publicly-owned resources, access to government research, favorable trade agreements, roads and other means of transportation, education that provides educated workers, tax loopholes, and innumerable government resources are taken advantage of by the rich, but paid for by all of us. What is called a ”tax break” for the rich is actually a redistribution of wealth from the poor and middle class—whose incomes have gone down—to those who have considerably more money than they need, money they have made because of tax investments by the rest of America.
The President provided a beautiful example of systems thinking. Under the Republican budget plan, the President would get a $200,000 a year tax break, which would be paid for by cutting programs for seniors, with the result that 33 seniors would be paying $6,000 more a year for health care to pay for his tax break. To see this, you have to look outside of the federal budget to the economic system at large, in which you can see what budget cuts will be balanced by increased in costs to others. A cut here in the budget is balanced by an increase outside the federal budget for real human beings.
…
When a system has causal effects, as in the above cases, we speak of “systemic causation.” “Systemic risks” are the risks created when there is systemic causation. Systemic causation contrasts with direct causation, as when, say, someone lifts something, or throws something, or shoots someone.
Linguists have discovered that every language studied has direct causation in its grammar, but no language has systemic causation in its grammar. Systemic causation is a harder concept and has to be learned either through socialization or education.
This got me interested in the original speech (transcript, video).
From our first days as a nation, we have put our faith in free markets and free enterprise as the engine of America’s wealth and prosperity. More than citizens of any other country, we are rugged individualists, a self-reliant people with a healthy skepticism of too much government.
But there has always been another thread running throughout our history – a belief that we are all connected; and that there are some things we can only do together, as a nation. We believe, in the words of our first Republican president, Abraham Lincoln, that through government, we should do together what we cannot do as well for ourselves.
There’s some feedback:
Ultimately, all this rising debt will cost us jobs and damage our economy. It will prevent us from making the investments we need to win the future. We won’t be able to afford good schools, new research, or the repair of roads and bridges – all the things that will create new jobs and businesses here in America. Businesses will be less likely to invest and open up shop in a country that seems unwilling or unable to balance its books. And if our creditors start worrying that we may be unable to pay back our debts, it could drive up interest rates for everyone who borrows money – making it harder for businesses to expand and hire, or families to take out a mortgage.
And recognition of systemic pressures for deficits:
But that starts by being honest about what’s causing our deficit. You see, most Americans tend to dislike government spending in the abstract, but they like the stuff it buys. Most of us, regardless of party affiliation, believe that we should have a strong military and a strong defense. Most Americans believe we should invest in education and medical research. Most Americans think we should protect commitments like Social Security and Medicare. And without even looking at a poll, my finely honed political skills tell me that almost no one believes they should be paying higher taxes.
Because all this spending is popular with both Republicans and Democrats alike, and because nobody wants to pay higher taxes, politicians are often eager to feed the impression that solving the problem is just a matter of eliminating waste and abuse –that tackling the deficit issue won’t require tough choices. Or they suggest that we can somehow close our entire deficit by eliminating things like foreign aid, even though foreign aid makes up about 1% of our entire budget.
There’s a bit of dynamics implicit in the discussion (e.g., the role of debt accumulation), but I think one thing is missing: straightforward grappling with worse-before-better behavior. The president proposes to go after waste (a favorite of all politicians) and tax breaks for the rich (far more sensible than the Ryan proposal), but doesn’t quite come to grips with the underlying question of how we can continue to feel prosperous and secure, when fundamentally we can’t (or at least shouldn’t) return to a previous pattern of unsustainable consumption in excess of our income funded by budget, trade and environmental deficits. What we really need, per yesterday’s post, is a reframing of what is now perceived as austerity as an opportunity to live with better health, relationships and security.
I part ways with Lakoff a bit on one topic:
Progressives tend to think more readily in terms of systems than conservatives. We see this in the answers to a question like, “What causes crime?” Progressives tend to give answers like economic hardship, or lack of education, or crime-ridden neighborhoods. Conservatives tend more to give an answer like “bad people — lock ‘em up, punish ‘em.” This is a consequence of a lifetime of thinking in terms of social connection (for progressives) and individual responsibility (for conservatives). Thus conservatives did not see the President’s plan, which relied on systemic causation, as a plan at all for directly addressing the deficit.
Differences in systemic thinking between progressives and conservatives can be seen in issues like global warming and financial reform. Conservatives have not recognized human causes of global warming, partly because they are systemic, not direct. When a huge snowstorm occurred in Washington DC recently, many conservatives saw it as disproving the existence of global warming — “How could warming cause snow?” Similarly, conservatives, thinking in terms of individual responsibility and direct causation, blamed homeowners for foreclosures on their homes, while progressives looked to systemic explanations, seeking reform in the financial system.
Certainly it is true that self-interested denial of feedback (or externalities, as an economist might describe some feedbacks) has found its home in the conservative and libertarian movements. But that doesn’t mean all conservative thought is devoid of systems thinking, and one can easily look back at history and find progressive or liberal policies that have also ignored systemic effects. Indeed, the conservative critique of progressive policies addressing crime and poverty issues has often been evolutionary arguments about the effects of incentives – a very systemic view. The problem is, words don’t provide enough formalism or connection to data to determine whose favorite feedback loops might dominate, so philosophical arguments about the merits of turn-the-other-cheek or an-eye-for-an-eye can go on forever. Models can assist with resolving these philosophical debates. However, at present public discourse is almost devoid of thinking, and often anti-intellectual, which makes it tough to contemplate sophisticated solutions to our problems.
I just ran across a nice talk by Tim Jackson, author ofProsperity Without Growth, on BigIdeas. It’s hard to summarize such a wide-ranging talk, but I’d call it a synthesis of the physical (planetary boundaries and exponential growth) and the behavioral (what is the economy for, how does it influence our choices, and how can we change it?). The horns of the dilemma are that growth can’t go on forever, yet we don’t know how to run an economy that doesn’t grow. (This of course begs the question, “growth of what?” – where the what is a mix of material and non-material things – a distinction that lies at the heart of many communication failures around the Limits to Growth debate.)
Hat tip to John Sterman & Travis Franck for passing along this cool example of positive feedback, discovered on Amazon by evolutionary biologist Michael Eisen. Two sellers apparently used algorithmic pricing that led to exponential growth of the price of a book:
This reminds me of a phenomenon that’s puzzled me for some time: “new economy” firms have at least as many opportunities for systemic problems as any others, yet modeling remains somewhat “old economy” focused on physical products and supply chains and more traditional services like health care. Perhaps this is just my own observational sampling bias; I’d be curious to know whether others see things the same way.
SEED asked eleven researchers to share the single most vital lesson from their life’s work. Every answer is about systems. Two samples:
“You can make sense of anything that changes smoothly in space or time, no matter how wild and complicated it may appear, by reimagining it as an infinite series of infinitesimal changes, each proceeding at a constant (and hence much simpler) rate, and then adding all those simple little changes back together to reconstitute the original whole.” —Steven Strogatz is a mathematician at Cornell University.
“Many social and natural phenomena—societies, economies, ecosystems, climate systems—are complex evolving webs of interdependent parts whose collective behavior cannot be reduced to a sum of parts; small, gradual changes in any component can trigger catastrophic and potentially irreversible changes in the entire system that can propagate, in domino fashion, even across traditional disciplinary boundaries.” —George Sugihara is a theoretical biologist at the Scripps Institution of Oceanography.
You really should read it, but here are some of the juicier excerpts:
Peek-a-boo
There were actually two sets of results. The first showed real GDP immediately rising by $33.7 billion in 2012 (or 0.2%) relative to the baseline, with total employment rising 831 thousand (or 0.6%) and the civilian unemployment rate falling a stunning 2 percentage points, a decline that persisted for a decade. (This path for the unemployment rate is labeled “First Result” in the table.) The decline in the unemployment rate was greeted — quite correctly, in our view — with widespread incredulity. Shortly thereafter, the initial results were withdrawn and replaced with a second set of results that made no mention of the unemployment rate, but not before we printed a hardcopy! (This is labeled “Second Result” in the table.)
…
Multiplier Mischief
The simulation shows real federal non-defense purchases down by $37.4 billion in 2012, but real GDP up by $33.7 billion, so the short-run “fiscal multiplier” is negative.[11] As noted above, that analysis was prepared using the GI model of the US economy. We are not intimately familiar with this model but have the impression it is a structural macro model in which near-term movements in GDP are governed by aggregate demand while long-term trends in output are determined by the labor force, the capital stock, and total factor productivity. Obviously we can’t object to this paradigm, since we rely on it, too.
However, precisely because we are so familiar with the characteristics of such systems, we doubt that the GI model, used as intended, shows a negative short-run fiscal multiplier. Indeed, GI’s own discussion of its model makes clear the system does, in fact, have a positive short-run fiscal multiplier.[12] This made us wonder how and on what grounds analysts at Heritage manipulated the system to produce the results reported.
Crowding Out Credibility
So, as we parsed the simulation results, we couldn’t see what was stimulating aggregate demand at unchanged interest rates and in the face of large cuts in government consumption and transfer payments…until we read this:
“Economic studies repeatedly find that government debt crowds out private investment, although the degree to which it does so can be debated. The structure of the model does not allow for this direct feedback between government spending and private investment variables. Therefore, the add factors on private investment variableswere alsoadjusted to reflect percentage changes in publicly held debt (MA italics).”
…
In sum, we have never seen an investment equation specified this way and, in our judgment, adjusting up investment demand in this manner is tantamount to assuming the answer. If Heritage wanted to show more crowding in, it should have argued for a bigger drop in interest rates or more interest-sensitive investment, responses over which there is legitimate empirical debate. These kinds of adjustments would not have reversed the sign of the short-run fiscal multiplier in the manner that simply adjusting up investment spending did.
Hilarious Housing?
In the simulation, the component of GDP that initially increases most, both in absolute and in percentage terms, is residential investment. This is really hard to fathom. There’s no change in pre-tax interest rates to speak of, hence the after-tax mortgage rate presumably rises with the decline in marginal tax rates even as the proposed tax reform curtails some or all of the mortgage interest deduction. …
The list of problems goes on and on, and there are others. MacroAdviser’s bottom line:
In our opinion, however, the macroeconomic analysis released in conjunction with the House Budget Resolution is not relevant to the coming discussion.We believe that the main result — that aggressive deficit reduction immediately raises GDP at unchanged interest rates — was generated by manipulating a model that would not otherwise produce this result, and that the basis for this manipulation is not supported either theoretically or empirically.Other features of the results — while perhaps unintended — seem highly problematic to us and seriously undermine the credibility of the overall conclusions.
This is really unfortunate, both for the policy debate and the modeling profession. Using models as arguments from authority, while manipulating them to produce propagandistic output, poisons the well for all rational inputs to policy debates. Unfortunately, there’s a long history of such practice, particularly in economic forecasting:
Not surprisingly, the forecasts produced by econometric models often don’t square with the modeler’s intuition. When they feel the model output is wrong, many modelers, including those at the “big three” econometric forecasting firms—Chase Econometrics, Wharton Econometric Forecasting Associates, and Data Resources – simply adjust their forecasts. This fudging, or add factoring as they call it, is routine and extensive. The late Otto Eckstein of Data Resources admitted that their forecasts were 60 percent model and 40 percent judgment (“Forecasters Overhaul Models of Economy in Wake of 1982 Errors,” Wall Street Journal, 17 February 1983). Business Week (“Where Big Econometric Models Go Wrong,” 30 March 1981) quotes an economist who points out that there is no way of knowing where the Wharton model ends and the model’s developer, Larry Klein, takes over. Of course, the adjustments made by add factoring are strongly colored by the personalities and political philosophies of the modelers. In the article cited above, the Wall Street Journal quotes Otto Eckstein as conceding that his forecasts sometimes reflect an optimistic view: “Data Resources is the most influential forecasting firm in the country… If it were in the hands of a doom-and- gloomer, it would be bad for the country.” John Sterman, A Skeptic’s Guide to Computer Models
As a historical note, GI – Global Insight, maker of the model used by Heritage CDA for the Ryan analysis – is the product of a Wharton/DRI merger, though it appears that the use of the GI model may have been outside their purview in this case.
What’s the cure? I’m not sure there is one as long as people are cherry-picking plausible sounding arguments to back up their preconceived notions or narrow self-interest. But assuming that some people do want intelligent discourse, it’s fairly easy to get it by having high standards for model transparency and quality. This means more than peer review, which often entails only weak checks of face validity of output. It means actual interaction with models, supported by software that makes it easy to identify causal relationships and perform tests in extreme conditions. It also means archiving of models and results for long-term replication and quality improvement. It requires that modelers invest more in testing the limits of their own insights, communicating their learnings and tools, and fostering understanding of principles that help raise the average level of debate.
doi:10.1016/j.enpol.2008.11.026
