Detecting the inconsistency of BS

DARPA put out a request for a BS detector for science. I responded with a strategy for combining the results of multiple models (using Rahmandad, Jalali & Paynabar’s generalized meta-analysis with some supporting infrastructure like data archiving) to establish whether new findings are consistent with an existing body of knowledge.

DARPA didn’t bite. I have no idea why, but could speculate from the RFC that they had in mind something more like a big data approach that would use text analysis to evaluate claims. Hopefully not, because a text-only approach will have limited power. Here’s why.

First, here’s why DARPA’s idea has merit in general. In math and logic, as soon as you admit one false proposition (1=2), you can leverage that to prove lots of false stuff. As soon as you start detecting conflicting conclusions, you know you’re hot on the trail of some BS. Physics is a little trickier, because things are happening on many different scales, with a multitude of physical quantities to map to math, and sometimes ambiguous measurements, but it still works. In social sciences its still harder, because there isn’t even agreement about the underlying constructs being measured.

For this to work as it does in math, you have to have a reliable way of determining whether claims are compatible with one another. This is seldom binary (“raising the tax rate increases revenue, or not”). You have to control for a variety of influences (What kind of taxes? Was there a war?) and nonmonotonic or state-dependent effects (like the Laffer curve). Text descriptions of models and findings just don’t contain enough information to support that. I recently showed how causal loop diagrams and system archetypes are hopelessly underspecified models. Words are usually even less structured and more ambiguous. So, while there’s lots of potential for using algorithms to discover laws, the fodder for such discovery is big data, not big talk.

Consider the example of back radiation. There’s a cultish corner of the climate denial world that claims there is no greenhouse effect because the atmosphere can’t radiate longwave energy back to the surface. The idea has even made it into the swampy bottom tier of supposedly peer-reviewed journals. Many web pages, like http://nov79.com/gbwm/bacr.html (which I won’t dignify with a link) are dedicated to this idea. There are lots of absurdities on this site that would turn up red flags, even for a pretty dumb robot:

At the claimed global average surface temperature of the earth (15°C), the error in the Stefan-Boltzmann constant appears to be at least a factor of 30, and perhaps 50, as energy flows cannot be balanced properly with such high levels of radiation.

Not the least reason for the error is that Planck’s constant is used to derive the SBC, while there is no Planck’s constant, because the whole concept of photons is absurd and admittedly in conflict with the wave nature of light.

http://nov79.com/gbwm/sbc.html

Any my favorite:

Disequilibrium is an impossibility.

http://nov79.com/gbwm/rte.html

But then, many statements are locally correct. The following is (roughly) true, if you’re thinking of mixing stocks of hot and cold gases. It’s just out of context, because the real world is not about that; it’s about energy stocks influenced by flows from radiation:

Total carbon dioxide is 400 parts per million in the atmosphere. That means 2,500 air molecules surrounding each CO2 molecule. To heat the air 1°C, each CO2 molecule would have to be 2,500°C

http://nov79.com/gbwm/heatg.html

Debunking such tripe often takes a lot more work and words than the original. To an algorithm that doesn’t have any underlying understanding of the problem space, real and fake science may look equally plausible, just as they do to a human with no subject matter expertise. That’s why you need the actual models and data, not just descriptions of the output of them, in order to perform quality checks, make, test and track predictions, and on the whole exercise the system to actively determine whether components are behaving coherently.

If there’s one thing algorithms might detect easily, its whether a paper uses models and data to make testable predictions at all. That’s an Achilles’ heel of climate skepticism in general, and certainly there is no coherent framework for the alternative reality on the nov79 site.

Even with a system of interlocking models, I think it may be tough to automatically discover anomalies. Consider this “proof” that a pulled ladder falls infinitely fast. Its math is correct. The logical steps to the conclusion are OK. So where’s the fault? It’s rather subtle. (I’ll leave it to you to work out.) Because this is physics, it’s easy to catch a whiff of BS in the notion that a physical object can move infinitely fast. But what if the subject matter was social dynamics? I think algorithms are a long way from having the kind of general intelligence needed to sniff out the problem.

If we had a BS detector, would it be influential? Certainly it would work for many users, but more broadly, I’m not sure. It’s increasingly evident that there’s no selection pressure against certain kinds of wrong beliefs. And if you can’t reject the specifics of a correct idea that you find distasteful, you can always posit a grand conspiracy or reject science and rational empiricism altogether.