The Secular Humanists of the Lowcountry

Join / Donate

All Forums > Science and Skepticism >

Science and Skepticism

Jan 31 2011
Truth Wears Off: Something wrong with science?

A friend recently sent me a copy of Jonah Lehrer's article The Truth Wears Off: Is there something wrong with the scientific method? that appeared in the New Yorker in December 2010 and asked my opinion.

Here, for anyone who is curious to hear a rebuttal, are my thoughts on the matter:

This article asks "Is there something wrong with the scientific method?"

My answer is "No, at least not anything we did not already know about from the beginning. The biggest problems are with misunderstandings about the scientific method and the distortions created by the media."

Allow me to explain.

Interestingly, this article does make several of the points I wish to make, but it does so in a way which misrepresents the overall situation.

For example, it mentions towards the beginning that reproducibility is viewed by those who believe in the scientific method as a key feature that ensures it works properly. Count me as one of those who believes this. In fact, the author picks a few examples in which the initial results were not reproducible and suggests that this indicates there is a problem with science. In contrast, I would say that it is an indication that there was a problem with those results, and the fact that they were later found to be irreproducible showed the scientific method working correctly and eliminating those mistakes.

Look at it this way: Imagine a machine designed to make mouse traps. The first step is that it produces lots of mouse traps, but with the understanding that some of those might not work. The next step is to test the traps and eliminate those that do not work. The third step is to test again the ones that worked the first time and eliminate those that failed to work a second time. It would be a mistake to claim that there is a problem with this process because some of the mouse traps that come out of the machine initially fail to work...that's part of the process and ignores the rest.

There are really three reasons for the "irreproducibility of initial results" such as those the author mentions, each of which gets discussed in the article to some extent.

* IT'S A FLUKE: Nearly every non-scientist and even an unfortunately large number of scientists fails to understand the basic role of probability in experimental design. Indeed, as the author explains, there is an arbitrary threshold (often 5 in 100 or sometimes 1 in 100) that is considered an acceptable level of "statistical significance" for publication of an article. What this means is that, even if everything in the experiment is right, 5 out of every 100 times an experiment is tried it will seem to support some hypothesis which is actually false.

This is a complicated one, so let me explain with an example. Suppose a lot of people believe that when you paint a quarter blue, it will land on "heads" five times in a row. Suppose we know, in fact, that even if it is painted blue, a quarter still only has a 50% chance of landing on heads with each flip and so has a probability of only about 3/100 of landing on heads five times in a row. If you imagine one person running this experiment of painting a coin blue and flipping it five times, it then seems unlikely that it will land on heads each time. But, if 1000 people ran this experiment, you would expect about 30 of them to find that their blue coin DID land on heads five times in a row.

Is this a problem for the scientific method? No, not at all. As in the "mousetrap example" from above, this is the expected outcome. Anyone who understands the process knows that this is a possibility and hence will be looking for more experiments to provide further evidence before accepting the "blue quarter" hypothesis as having been proved.

* FLAWED EXPERIMENTAL DESIGN: Many experiments are flawed in such an obvious way that I really can blame the researcher. In other cases, the flaw is so subtle, I can see why the scientist thought they were doing everything correctly. In either case, however, it is important that the scientific method is not JUST that first experiment (the machine that makes the mousetraps) but ALSO the additional process which involves replication of the results by others (testing them and getting rid of the flawed ones).

* IT REPRESENTS AN ACTUAL CHANGE IN THE ENVIRONMENT: If an experiment regarding lion behavior from the 20th century no longer works today, it COULD be because the population of lions today actually is different than lions were at the time.

I think some combination of these three ideas fully explain the supposed "decline effect" that the author of the article is trying to create a fuss about in this article.

The "precognition" study, for example, is almost certainly just a consequence of the first two: fluke and flaw. I know this is not as much fun as believing in psychic powers and Thor and the Easter Bunny, but realistically, it is so much more likely that there WAS no such psychic effect and that is why it is irreproducible than any conspiracy theory about a problem with the scientific method!

The idea of a change in the environment is a leading contender to explain the first example mentioned in the article: the change in the effectiveness of anti-psychotic medications. In particular, as the author hints, the real explanation of this may be that many people being diagnosed today as "psychotics" are people who would never have received that diagnosis decades ago. Consequently, the pool of experimental subjects is actually different than it was decades ago. If so, then this is another success of the scientific method, revealing a change in the environment that was not even really the purpose of the investigation.

As for the decline in the symmetry effect, I'm not sure WHAT Lehrer is talking about. Laura just looked online and found lots of articles on this subject published in the last year, and they all continue to find evidence for this phenomenon in nature.

One of the biggest problems with the scientific method is not anything about the scientific method but merely the fact that the popular media, in their desperate search for anything that will hold viewer/reader's attention through the advertising, comb through all of the results of scientific studies and make a big deal about the studies with seemingly interesting outcomes, even when they are most likely just flukes or flaws.

And, as if that was not bad enough, now in THIS article we have the media making things even worse by suggesting that somehow this indicates a fundamental flaw in the entire system!

Imagine if you ran the mousetrap factory and were proud of the fact that 99.999% of the mousetraps that you actually sell are fully functional, but the newspaper insists on looking at the ones that come out of the machine (before your quality control tests eliminate the duds) and publishing articles about how bad those are!

Finally, regarding the author's final words: certainly, it is true that not EVERY truth is provable through science, and moreover true that you cannot automatically assume something is true just because a study or some scientists claim that it is. However, to leap from this recognition that science is not omnipotent nor infallible to the implied conclusion that science is a crapshoot as likely to be wrong as right is completely unjustified. If we consider the whole process (and not just treat any initial study as if it was the final word), science has a spectacular track record for discovering truths and eliminating popular falsehoods. If those who wish to dwell on its flaws would like to propose a method for "building a better mousetrap," I'd be happy to listen, but think it is unlikely that they can propose an alternative that would be even half as effective!


Feb 1 2011
Re: Truth Wears Off: Something wrong with science?

Of course, another possible explanation for a result not being reproducible, which I failed to mention above, is actual fraud on the part of the researcher.

This does happen, but once again the requirement of reproducibility is a way to eliminate the falsehoods and often to catch the perpetrators of the fraud.

For example, it was obviously a big deal in the late 1990s when a study claimed to find a link between autism and vaccination. When the first study came out, this should have been viewed as a reason for further investigation, not "scientific proof". In particular, the study could have found an actual effect, it could have been just a consequence of the "flaw or fluke" explanations I describe above, or it could have been a complete lie.

Over the years, as many studies (including many much more statistically powerful than the original one) failed to find this supposed effect, it has started to seem overwhelmingly unlikely that it was a real effect.

Now, there appears to be sufficient evidence to say that the author of the original study was not just mistaken, but actually lied. Consider this quote from this article:

Dr. Godlee said the BMJ and investigator Brian Deer uncovered "clear evidence of falsification" of Wakefield’s data. Godlee's editorial states that "Wakefield altered numerous facts about the patients' medical histories in order to support his claim to have identified a new syndrome; how his institution, the Royal Free Hospital and Medical School in London, supported him as he sought to exploit the ensuing MMR scare for financial gain; and how key players failed to investigate thoroughly in the public interest when Deer first raised his concerns."

Is this a problem with science? Well, it would be nice if there were no frauds, but as that does not seem likely in any scenario involving humans, the requirement of reproducibility seems to me to be a fantastic way of weeding out the good results from the bad. The fact that many naive parents and greedy media outlets went crazy for Wakefield's claim of a link between autism and vaccines without waiting to see whether the results were reproduced by independent researchers, IMHO, is their problem and not a problem with the scientific method. (Does Lehrer think that this is yet another example of "decline"?)


Return to Science and Skepticism Forum
Return to Discussion Home

Webmaster: Alex Kasman 2016