Here is something I learned about from Shermer's Skeptic Society. Sometimes called the Baloney Detection Kit, here it is the Bullshit detection kit. Read, learn and apply to liberate yourself from Jeff Prathers' Warrior Cult.
How does science deal with such subjective biases? How do we
know when a claim is bogus or real? We want to be open-minded
enough to accept radical new ideas when they occasionally come
along, but we don’t want to be so open-minded that our brains
fall out. This problem led us at the Skeptics Society to create an
educational tool called the BULLSHIT Detection Kit, inspired by
Carl Sagan’s discussion of how to detect “BULLSHIT” in his marvelous
book The Demon-Haunted World. In this BULLSHIT Detection
Kit, we suggest ten questions to ask when encountering any
claim that can help us decide if we are being too open-minded in
accepting it or too closed-minded in rejecting it.
1. How reliable is the source of the claim? As Daniel Kevles
showed so effectively in his 1999 book The Baltimore Affair
investigating possible scientific fraud there is a boundary problem
in detecting a fraudulent signal within the background noise
of mistakes and sloppiness that is a normal part of the scientific
process. The investigation of research notes in a laboratory affiliated
with Nobel laureate David Baltimore by an independent
committee established by Congress to investigate potential fraud
revealed a surprising number of mistakes. But science is messier
than most people realize. Baltimore was exonerated when it
became clear that there was no purposeful data manipulation.
2. Does this source often make similar claims? Pseudoscientists
have a habit of going well beyond the facts, so when individuals
make numerous extraordinary claims, they may be more than
just iconoclasts. This is a matter of quantitative scaling, since
some great thinkers often go beyond the data in their creative
speculations. Cornell’s Thomas Gold is notorious for his radical
ideas, but he has been right often enough that other scientists
listen to what he has to say. Gold proposes, for example, that oil
is not a fossil fuel at all, but the by-product of a deep hot biosphere.
Hardly any earth scientists I have spoken with take this
thesis seriously, yet they do not consider Gold a crank. What we
are looking for here is a pattern of fringe thinking that consistently
ignores or distorts data.
3. Have the claims been verified by another source? Typically
pseudoscientists will make statements that are unverified, or
verified by a source within their own belief circle. We must ask
who is checking the claims, and even who is checking the checkers.
The biggest problem with the cold fusion debacle, for example,
was not that scientists Stanley Pons and Martin Fleischman
were wrong; it was that they announced their spectacular discovery
before it was verified by other laboratories (at a press
conference no less), and, worse, when cold fusion was not replicated,
they continued to cling to their claim.
4. How does the claim fit with what we know about how the
world works? An extraordinary claim must be placed into a
larger context to see how it fits. When people claim that the
pyramids and the Sphinx were built more than ten thousand
years ago by an advanced race of humans, they are not presenting
any context for that earlier civilization. Where are the rest of
the artifacts of those people? Where are their works of art, their
weapons, their clothing, their tools, their trash? This is simply
not how archaeology works.
5. Has anyone gone out of the way to disprove the claim, or has
only confirmatory evidence been sought? This is the confirmation
bias, or the tendency to seek confirmatory evidence and
reject or ignore disconfirmatory evidence. The confirmation bias
is powerful and pervasive and is almost impossible for any of us
to avoid. It is why the methods of science that emphasize checking
and rechecking, verification and replication, and especially
attempts to falsify a claim are so critical.
6. Does the preponderance of evidence converge to the
claimant’s conclusion, or a different one? The theory of evolution,
for example, is proven through a convergence of evidence
from a number of independent lines of inquiry. No one fossil, no
one piece of biological or paleontological evidence has “evolution”
written on it; instead there is a convergence of evidence
from tens of thousands of evidentiary bits that adds up to a
story of the evolution of life. Creationists conveniently ignore
this convergence, focusing instead on trivial anomalies or currently
unexplained phenomena in the history of life.
7. Is the claimant employing the accepted rules of reason and
tools of research, or have these been abandoned in favor of
others that lead to the desired conclusion? UFOlogists suffer
this fallacy in their continued focus on a handful of unexplained
atmospheric anomalies and visual misperceptions by eyewitnesses,
while conveniently ignoring the fact that the vast majority
(90 to 95 percent) of UFO sightings are fully explicable with
prosaic answers.
8. Has the claimant provided a different explanation for the
observed phenomena, or is it strictly a process of denying the
existing explanation? This is a classic debate strategy—criticize
your opponent and never affirm what you believe in order to
avoid criticism. But this stratagem is unacceptable in science. Big
Bang skeptics, for example, ignore the convergence of evidence of
this cosmological model, focus on the few flaws in the accepted
model, and have yet to offer a viable cosmological alternative that
carries a preponderance of evidence in favor of it.
9. If the claimant has proffered a new explanation, does it
account for as many phenomena as the old explanation? The
HIV-AIDS skeptics argue that lifestyle, not HIV, causes AIDS.
Yet, to make this argument they must ignore the convergence of
evidence in support of HIV as the causal vector in AIDS, and
simultaneously ignore such blatant evidence as the significant
correlation between the rise in AIDS among hemophiliacs
shortly after HIV was inadvertently introduced into the blood
supply. On top of this, their alternative theory does not explain
nearly as much of the data as the HIV theory.
10. Do the claimants’ personal beliefs and biases drive the conclusions,
or vice versa? All scientists hold social, political, and
ideological beliefs that could potentially slant their interpretations
of the data, but how do those biases and beliefs affect their
research? At some point, usually during the peer-review system,
such biases and beliefs are rooted out, or the paper or book is
rejected for publication. This is why one should not work in an
intellectual vacuum. If you don’t catch the biases in your
research, someone else will.
There is no definitive set of criteria we can apply in determining
how open-minded we should be when encountering new
claims and ideas, but with mathematical calculations on the
odds of weird things happening and with an analysis of the sorts
of questions we should ask when we encounter weird things, we
have made a start toward coming to grips with our weird and wonderful world.
No comments:
Post a Comment