Can You Believe It?

4. How did they get the evidence?

Where data collection is concerned, the devil is in the details. Exactly how measurements are made, with what equipment, and under what conditions, can have make-or-break significance.

Methodologies are important not just in polling situations, but in every science—even in the “hard” sciences, where measurements may be made using billion-dollar machines. No matter the field, data collected one way may support one conclusion; data collected another way may support a completely different conclusion.

Huh? This is science, isn’t it? Actually, the process of collecting data is fraught with error. First of all, there is no such thing as an exact measurement—all results contain a certain unavoidable fudge factor called error, the result of living in an imperfect, imprecise world. Then there’s the possibility of a systematic error, a flaw in a measuring device or method that skews the data one way or another. Uncontrolled variables can play evil tricks on data, too; these are factors that influence results but haven’t been taken into account, possibly because no one even knows about them.

So ask yourself: What methods were used to collect the evidence for this claim? Are the methods even explained? Be warned that even methods that seem reasonable may rest on false assumptions. One hundred years ago, scientists perfected methods of estimating human intelligence by measuring the volume of a person's brain cavity. The method of measurement was fine, but the underlying assumption—that brain size predicts intelligence—was bogus.