Reading Science, Redux

I once had a series of posts I put on here that I offered as corrective to the multitude of “How to Be Scientifically Literate” posts, articles, and infographics that flood my feed every so often. I ended up pulling them because I found them to be clunky and overlong–the exact opposite thing they needed to be to achieve what I had hoped for them.

So, if I am to offer a primer on how to be scientifically literate, it would start here:

  1. All individual scientific experiments have flaws that limit the size, scope, or generalizability of their finding. This is true regardless of the statistical or probabalistic model used to demonstrate the relationship or its strength, magnitude, and direction. This is even more true in our current scientific climate that rewards findings that reject the null hypothesis and other issues in the “publication bias” world. If you cannot accurately describe the limitations of the study you are citing, then you should not be confident in its findings.
  2. All findings are subject to statistical noise. That is, even if we know and understand the limitations (and maybe flaws) of the study in question, there is always the chance that this finding is an anomaly. And if the study you are using is the only such one with the finding you need to support your argument (or in the general minority), you probably should be cautious in deploying it.

Knowing these limitations, this is how you should approach a scientific article.

  1. Ask yourself, “What do I already know about this subject?” and “What do I think is the answer to the question the authors are attempting to answer?” And ask yourself, how confident you are in this conclusion. Don’t allow yourself to give yourself a 100% confident score.
  2. Ask yourself next, “What is the most direct way to answer that question; what data would I need?” Go ahead and think big, act like you have magic powers that can compel individuals to join your study for free and forever. Act like you have all the money in the world and can buy whatever equipment or hire whatever staff you need to answer the question. (Note: Your powers are limited, you cannot just wish the answer into existence.)
  3. Then compare your dream study to the real one. How do they differ?
  4. Finally, are these differences so vast, or the experiment so indirect that they do not even answer the question in any real way? (e.g., are they only really measuring proxy variables?)

If, at this point, you agree (with yourself) that the study in question, though limited, does point a finger toward the answer they seek, compare their findings to what you already thought the answer might be. Does it agree with what you thought they’d find? Disagree?

The study should move your prior opinion one way or the other–make you more confident it’s true (but not entirely) or less confident that it is. How much and in which direction depends on, among other things, your assessment of the relative strengths of this study (compared to the ones that helped  you form your original opinion) and its flaws relative to the scientific gold standard.

Note: If the findings agree with what you already believe, you should re-assess the article with the specific intent of trying to see its limitations through the eyes of someone who disagrees with you. You are more likely to reduce the importance of limitations of an argument whose conclusions reinforce previously held beliefs.