1. What’s the claim?

2. Who says?

3. What’s the evidence?

4. How did they get the evidence?

5. Is there anything (or anyone) to back up this claim?

6. Could there be another explanation?

7. Who cares?

Screaming headlines abound in our media-saturated world.  “Killer Moths Invade Homes.” “New Drug Promises Alzheimer’s Cure.”  “Experts Confirm: Sky Is Falling.”

Some are obvious click bait, but others seem legit—especially when seemingly backed by science. Inevitably, we’re all regularly confronted with scientific facts and factoids. Claims about household products, technology, medicine, and even politics often come steeped in the presumed authority of scientific research.

The truth is slippery, and it’s not always graspable even by experts. So how can we non-experts decide what to believe?

The seven questions here can help you weigh the validity of scientific information, wherever it might appear. You may not be able to get answers to them all—and this in itself might be telling—but if you can, you’ll be well on your way to separating science fact from science fiction.

1. What's the claim?

Simple as it might sound, the first step toward weighing a scientific claim is to establish what it is, as well as what it isn’t.

Read or listen carefully. What exactly is the claim? Where does it sit on the spectrum from likely to outlandish? Do the findings confirm or challenge existing beliefs?

Try to read between the lines when you’re assessing the validity of a claim. Ask yourself: What aren’t they saying? A reputable source will acknowledge missing pieces of the puzzle, or areas where more research is needed.

Finally, don’t be tricked into confusing correlation with causation. Correlation is when two things change together: the relationship could be chance, or there could be a third variable causing both changes. Causation, on the other hand, is a direct cause-and-effect relationship between two things.

Suppose you read that high rates of violent crime are associated with increased sales of ice cream. Ice cream sales may be correlated with violent crimes—both might increase with warmer

2. Who says?

They say you’re only as good as your reputation . . . but who is “they?”

Any decent claim requires that someone stand behind it—preferably a well-respected source from an equally well-respected institution. You may not know the reputation of the scientist or institution involved, but chances are you can find out.

Once you figure out who did the research and where, you can go further by finding where the research was originally published. Most respected scientific journals are peer-reviewed, which means that other scientists read the articles vying for publication and screen out any shoddy science.

Beware science stories that go directly to mainstream media, also known as “science by press release.” This can be a ploy to circumvent the peer-review process. Notorious examples include a 1989 press conference announcing successful cold fusion and a 2002 press conference announcing successful human cloning. Both stories were later debunked.

Last but not least, it never hurts to find out who paid for the research. Research funded by sources with vested interests (drug companies and advocacy groups, for example) should be given extra scrutiny. Some manufacturers publicize the positive aspects of their product while burying any research that doesn’t support their desired outcome.

3. What's the evidence?

Evidence is the bread and butter of science. Reaching a scientific conclusion of any kind requires observation and measurement—ideally, the careful, repeated observation and measurement known as empirical evidence.

Evidence can take many forms, because research itself can take many forms. Sometimes, evidence may appear pictorially as a chart or graph. Pay careful attention to the labels and scales on graphs and charts, because just like words, visuals can mislead, as well as tell hidden stories.

Whatever form evidence takes, it's likely to be at least partly numeric. Alas, it's at precisely the moment when numbers appear that most people begin to tune out. That's unfortunate because numbers can't (usually) lie, which is why looking at the actual evidence can be most illuminating in evaluating a claim. For starters, how much data was collected? You don't need a degree in statistics to know that the more people there are involved in a study, the less likely it is that the results are just chance.

Sometimes, a claim may be made with no empirical evidence at all. File these claims under “S” for speculation. In other cases, a claim may rest on evidence that is limited or downright scanty. In paleontology, for example, where preserved specimens of ancient life are few, entire theories may rest precariously on the discovery of a single bone. In physics, string theory redefines the universe without any evidence at all. String theory holds that everything in our universe results from vibrations of miniscule strings, but no one has figured out how to test if the theory is true.

4. How did they get the evidence?

Where data collection is concerned, the devil is in the details. Exactly how measurements are made, with what equipment, and under what conditions, can have make-or-break significance.

Methodologies are important not just in polling situations, but in every science—even in the “hard” sciences, where measurements may be made using billion-dollar machines. No matter the field, data collected one way may support one conclusion; data collected another way may support a completely different conclusion.

Huh? This is science, isn’t it? Actually, the process of collecting data is fraught with error. First of all, there is no such thing as an exact measurement—all results contain a certain unavoidable fudge factor called error, the result of living in an imperfect, imprecise world. Then there’s the possibility of a systematic error, a flaw in a measuring device or method that skews the data one way or another. Uncontrolled variables can play evil tricks on data, too; these are factors that influence results but haven’t been taken into account, possibly because no one even knows about them.

So ask yourself: What methods were used to collect the evidence for this claim? Are the methods even explained? Be warned that even methods that seem reasonable may rest on false assumptions. One hundred years ago, scientists perfected methods of estimating human intelligence by measuring the volume of a person’s brain cavity. The method of measurement was fine, but the underlying assumption—that brain size predicts intelligence—was bogus.

5. Is there anything (or anyone) to back up this claim?

No one—not even an astrophysicist—works in a void. All research takes place in the context of what we currently believe to be true, and this context can either lend credibility to a claim or erode it. The newer and stranger the result, the greater the burden of proof.

How does this claim compare to other studies on the same subject? Is there consensus in the field? Who disagrees, and why?

Scientists form a community, and as in all communities, not everyone is in perfect agreement. Even so, if there’s one thing all scientists agree on it’s reproducibility. For one person’s research to be believable, other people using the same tools or techniques must be able to produce the same result.

Has anyone else in the field verified the result? If a researcher is using a new tool or technique, are there other tools or techniques that can verify the result? Searching online for other articles on the same topic is an easy way to find a second opinion—and often a third and fourth, to boot.

6. Could there be another explanation?

Sometimes, it’s not the research methods or the data that are flawed, but the interpretation of the data.

It’s human nature to see what we’re looking for—whether it’s really there or not—and not see what we’re not looking for. Scientific truth sometimes falls prey to this tendency of ours, when scientists inadvertently leap to conclusions their research doesn’t really support.

For a classic (and literal) example of just such a logical “leap,” consider the story of Italian anatomist Luigi Galvani who, in 1871, poked a brass hook into one of the frog legs he was preparing for dissection. When he saw the leg jump, he wrongly attributed the phenomenon to “animal electricity,” a then-popular concept that animal tissues contained a reservoir of electricity that gave them life.

Actually, it wasn’t the frog leg that had produced the electricity, but contact between the brass hook and the iron railing from which it hung—a misunderstanding not corrected until years later. Galvani didn’t realize it, but he hadn’t discovered proof of “animal electricity” at all. He’d discovered the battery.

Sometimes researchers will admit to other possible interpretations of their results, but mistakes are often lodged hopelessly where no one can see them (yet), within the dominant paradigm. All science is necessarily provisional; today’s facts become tomorrow’s fiction as new measuring tools, new discoveries, and new paradigms continually expand our knowledge and understanding.

7. Who cares?

There are always people interested in the outcome of scientific research—other researchers within and outside the field, funders, special-interest groups, manufacturers, and anyone else who needs the information to make personal or policy decisions.

All research happens in a social context, and that context can be at least as important as the claim itself. Bias and predispositions can affect whether research happens at all, whether and how the results are made public and, most importantly, how results are “spun” and interpreted both inside and outside the scientific community.

Who supports the claim, and who doesn’t? What are their biases? Who funded the research? Why? Be extra wary of research that was either funded or conducted by someone with something to gain.

Finally, keep in mind the hype factor. Big news sells, and science stories are easy prey for people looking to make mountains out of molehills. Never take headlines or sound bites at face value. There’s almost always more to the story.

....................

This content is from How Do We Know What We Know? Resources for the Public Understanding of Scientific Evidence, which was made possible by a grant from the National Science Foundation with the additional generosity of the Gordon and Betty Moore Foundation, The Jim Clark Endowment for Internet Education, and the McBean Family Foundation.

Download Booklet
This folded booklet contains all seven questions in a handy to-go format.