Why scientists seem to change their minds

August 5, 2012 by kirstyevidence 3 Comments

(This blog was originally published as a series of 10 articles here http://blog.inasp.info/why-scientists-seem-to-change-their-minds-1/)

Approximately once a year, I get into an argument with my father about the reliability of scientific evidence. My dad likes to tell me that scientists are always getting it wrong and, therefore, scientific knowledge should not be put on a pedestal above other forms of knowledge. It can certainly seem that scientists are constantly backtracking but I would argue that this is more to do with the imperfect humans (whose values and beliefs influence how they do research and how they interpret scientific findings) rather flaw in the scientific method per se.

The reason that I believe we need to use scientific methods to evaluate things is that human beings are extremely susceptible to prejudice, group think, placebo effects, confirmation bias and a whole host of other factors. This means that some of the things which we strongly believe to be true, are in fact not! Scientific investigation attempts to overcome some of these effects to get a more objective view of an issue. A key tenet of the scientific method is that results must be reproducible given the same conditions. If a finding cannot be reproduced, it is not scientifically proven.

So if this is the case, why is it that scientists always seem to be changing their minds?! Well, there a number of reasons which I will outline below. The important thing to bear in mind is that the scientific method attempts to give objective answers to specific questions. Scientists are not perfect and therefore at times human subjectivity creeps in. But rather than using this as a reason to reject science, I suggest we concentrate on ways to improve the scientific method but also to consider how it can be used to compliment other forms of evidence.

Reason 1: The conditions have changed

The scientific method allows us to test whether an intervention works, in given conditions, better than a control intervention. So for example, some research may demonstrate that a new painkiller reduces the severity of headaches better than a similarly administered placebo in a group of women between the age of 18 and 30. Providing that this finding is reproducible, it is scientifically proven that this pill achieves the outcome of interest in these conditions. However, this does not prove that this painkiller has the same effect in other conditions. For example, the research does not tell you whether the painkiller performs better than a placebo in reducing back pain in elderly men or reducing toothache in children. You may hypothesise that it is likely to do so, but you would need to carry out more research to demonstrate if this is true. Similarly, when researchers ‘model’ a situation they define the exact conditions of the model. The results of such a model are true only if the assumptions (conditions) that they have defined are also true. This can be clearly seen in the economic models which failed to predict the recent banking crisis. In fact the results of these models may well have been correct for the conditions they used, the problem is that these conditions did not reflect the real world adequately. Certain key assumptions (for example that bonds based on sub-prime mortgages were relatively safe investments) were fundamentally wrong. For this reason, the results of the models, while correct for the conditions assumed, were, in fact, not useful for predicting the future.

Reason 2: They didn’t ask the right questions

A scientific experiment gives you the answer to a specific question (e.g. does this intervention achieve the specified outcome better than a specified control in specified conditions). As mentioned in the previous post, this does not tell you if the intervention works in other conditions. However, there are also many other important questions that this research does not answer — for example, is the intervention acceptable to the community?; Is the outcome desirable?; Is the intervention safe?; Is it cost effective?; etc. Sometimes subsequent research asks a different question and the answer means that the policy decision based on the initial research is reversed. For example, research might demonstrate that treating a certain crop with a certain fertiliser increases yield. However, some years later further research may indicate that the fertiliser is damaging to the local environment and thus the fertiliser is withdrawn. This does not mean that the original research was wrong — the fact that the fertiliser increased yield is still true. But the additional information gained by asking a different question has changed the policy decision.

Reason 3: They lied (or at least stretched the truth!)

Let’s be completely honest. Scientists are human beings and human beings lie. There are numerous cases where the results that scientists report are in fact fabricated. Some of these cases are high profile — such as the fraud committed by Korean scientist Hwang Woo-Suk. In addition to blatant fraud, there are also plenty of cases where scientists unconsciously inflate effects because they believe that they exist. There is a detailed (and sometimes quite amusing) record of scientific results which have been retracted for various reasons online.

Various strategies exist to counteract human error (or fraud) in scientific findings. For example, the concept of double blinding in clinical trials, where neither the patient nor the scientist knows which patients have received the ‘true’ drug, aims to prevent scientists unconsciously over-reporting effects in the treatment group. The principal of reproducibility also counteracts false findings. If a scientist has faked his data, it is likely that other scientists will not be able to reproduce the findings and therefore the idea will not be considered scientifically proven.

Reason 4: The scientist misinterpreted the results

Sometimes the scientific findings are correct but the way they are interpreted is not. I can give an example from my own scientific career as an immunologist. I used to study the response of a certain type of human blood cell (an NK cell) to malaria parasites. In one study, using microscopy, we were able to show that some NK cells attached to some malaria parasites. In the paper we wrote to report this, we speculated that perhaps the NK cells were in fact directly communicating with, and becoming activated by, the malaria parasite via a structure known as an immunological synapse. The reason we speculated that this might be the case was that if that was true it would have been extremely exciting (at least to nerdy immunologists like us!). However, subsequent experiments suggested that while our observation was correct (that some NK cells sometimes stick to some malaria parasites) our interpretation of the observation (that an immune synapse was being formed) was probably not. Our observations were correct, our interpretation of them was just wishful thinking!

Reason 5: Someone else misinterpreted the results

Yesterday I mentioned that scientists sometimes misinterpret what they observe but of course it is not only scientists who interpret scientific findings. Journalists, who wish to make scientific findings into compelling news stories are sometimes guilty of misinterpreting (and sometimes downright misreporting) scientific findings. So for example, a study which demonstrates that individuals who consume a small amount of chocolate every day have a lower risk of developing a certain type of cancer, may be reported as ‘Scientists prove that chocolate is good for you!’. Of course this headline does not accurately convey what the study proved. Unfortunately, in a few months’ time, if another study shows that eating chocolate is positively correlated with a different disease, you might get another headline announcing ‘Scientists say chocolate is bad for you!’. In this case the scientific findings of both studies may well have been correct but to a non-scientist reading newspaper headlines it may seem as if scientists just can’t get it right! The balance between writing compelling scientific news stories and providing accurate information on the limitations of scientific findings is an area of on-going tension between scientists and journalists.

Reason 6: The entire dataset was not considered

The scientific method is a good way to get more objective answers to questions. However, the way that we publish scientific findings is rather less objective. Scientific journals are not all equal. Some are seen as more ‘sexy’ than others. The measure of a journal’s ‘sexiness’ is its impact factor; so journals with high impact factors are seen as the best to publish in. Scientists are judged (by promotion committees, funding agencies etc.) according to the impact factors of the journals they have published in. Unfortunately, this sets up a bias against publishing negative results. Positive results (where you prove that something works) are intrinsically more ‘sexy’ than negative results. Therefore scientists are much more likely to get results published (particularly in high impact journals) if they have found a positive result. There is a much lower incentive to publish your negative or inconclusive results since these will only be accepted by low impact journals.

This phenomenon is called publication bias. For example, imagine that ten studies were carried out to test the impact of a new educational reform on children’s learning and five demonstrate a positive impact and five demonstrate a negative impact. It is conceivable that some of the researchers who found negative impacts feel that the incentive to write up their results and get them published is not high enough so they never get the negative results published (for example perhaps they have been involved in some other research which has given more ‘interesting’ results and they prioritise that instead). For somebody who examines the research literature, it may seem as if the balance of evidence shows that the intervention works when in fact the true picture is that the results are balanced for and against.

Again I have experienced this phenomenon in my own work. Early in my career, I carried out some research looking at HIV infection in a certain type of brain cell known as astrocytes. Despite my best efforts I was never able to find HIV in the astrocytes obtained from the post-mortem brains of HIV-positive individuals. I never got round to publishing this finding, mainly because it didn’t seem very interesting, but many others did publish when they found HIV in astrocytes. In this small way I contributed to the publication bias in this field.

Reason 7: Part of the dataset was suppressed

Above I talked about research which does not get published simply because the incentive to publish it is not high enough. This happens frequently but unfortunately, sometimes more sinister factors are at play. Some pharmaceutical companies have been accused of deliberately suppressing findings which suggest that their drugs do not work. This phenomena is well known and there has been a lot of action in recent years to ensure that the results of clinical trials are more accessible. In some countries, anyone conducting a clinical trial is compelled to register it on a clinical trial registry. This means that anyone wanting to review the evidence will be able to find the results — including those which gave results which contradict drug company claims.

Reason 8: Their belief was not based on scientific data

Just because a scientist believes something does not mean that it is scientifically proven. There are many examples of ‘flat-earth’ beliefs — things which many scientists hold to be true but which have actually never been proven. A good example comes from the medical profession. For decades, doctors working in emergency settings have treated critically ill children by giving a large initial infusion of saline (salt water). This practice was so well established that no one thought to test it. However, recently, to the shock of the medical community, a trial comparing different types of infusion found that the children in the control group, who received no infusion, actually did best. It is therefore vital that we don’t confuse what scientists believe with what has been proven.

Reason 9: The scientists haven’t changed their minds, but many people believe they have

There are a number of high profile issues which many members of the public believe are not resolved by scientists, where in fact there is broad scientific agreement. A classic example is the theory of evolution. Many members of the public believe that there is controversy amongst scientists about evolution; however this is simply not true. Evolution is the cornerstone of modern biological research. I have met hundreds of biological researchers but I have never met one who thought that evolution does not happen. To be honest in all my years working as a researcher I never even heard the matter discussed and it was only later that I discovered, to my surprise, that many people think that it is a matter of scientific controversy. If you are a biologist you see evolution take place in every experiment you do. For example, as mentioned before I used to work on HIV and if you change the conditions in which you are growing HIV (or in the real world, if it is transmitted to a new person) within days you can see that it evolves to become genetically distinct from its ancestors.

Of course if you look hard enough you will be able to find one or two individuals with scientific qualifications who will question evolution but it is certainly not a mainstream scientific viewpoint. So why do people think it is a matter of controversy? The reason is that there are many people who do not believe in evolution for religious reasons and some individuals have tried to support their religious viewpoint by arguing that even the scientists have not made up their mind on this point. A similar situation is occurring with climate change. In this case, many individuals do not believe that humans contribute to climate change for political reasons. They are also stating (often very loudly) that there is scientific controversy on this issue when the fact is that there is broad scientific consensus that humans do contribute to climate change.

Although scientists seem to change their minds… let’s not throw the baby out with the bath water

Above I have outlined some of the reasons why scientist may appear to be constantly changing their minds. I hope they have been informative but I just want to conclude with a plea. The scientific method is not perfect and the individuals who implement it and interpret scientific findings are human beings who get things wrong. However, please let’s not throw the baby out with the bath water! The scientific method is a really valuable approach to finding more objective answers to some important questions. There are many questions that we really need objective answers to!

I feel that critiquing scientific findings is great – indeed it is a vital part of the scientific process. But just because sometimes it seems that scientists have changed their minds (for the reasons outlined) and just because science cannot provide all the answers to all the important questions, let’s not throw the baby out with the bath water!