What does significant mean to you?

What does the word significant mean to you? In general usage it's another word for important or substantial, but when it comes to scientific results there's a very different spin to the word. In science results are usually classed as being 'statistically significant', and by science we include medicine and medical research. Very often new results are announced and we are told that these are 'significant', this is particularly the case when these results are announced in the press, especially the popular press rather than the scientific press. The magic phrase 'statistically significant' often gets turned into 'significant', and for the reader not aware of the difference between the two it's normally taken to mean that a result is important or substantial.

Unfortunately 'statistically significant' is just a way of saying that there's only a certain chance that the results could have happened by chance. Normally scientists will talk about a result being statistically significant at a p-level, often p=0.05, which is to say that there's around a 5% (1 in 20) chance that the result could have happened by accident. It doesn't tell us that the result is important, or substantial or even particularly interesting, all it tells us if that you repeated the experiment (or drug trial) you would expect to have to run it 20 times before you got this result by chance.

There are a couple of obvious things to say at this point. The first is to say that p=0.05 sets a pretty low bar. Another way of looking at this is to say that 5 out of every 100 results are just due to chance. Those odds might be fine for the casino or the occasional horse race, but they're way too high for drugs that can kill (or save) people. Surely for medical research we need to be looking at setting the bar higher - we should be looking at results at the p=0.01 or p=0.001 level to make sure that we're not getting spurious results. Even then, a result that is significant at the p=0.001 level means that we're ten times more sure it's not an accident compared to the p=0.01 level, but that's all it means.


Secondly, statistical significance doesn't tell us what we really want to know, which is whether a result is important or not. This is really important when scientists say that a new treatment offers a significant survival advantage to patients. Even if the researcher is absolutely clear to the press and says that the new treatment offers a 'statistically significant' survival advantage, most people will interpret that as meaning that the patient will gain an important survival advantage - in other words that they will survive for longer. But this isn't what that statement means, what it means is that the new treatment offers a survival advantage that only has a 5% chance of being accidental and unrelated to the treatment. What it doesn't say at all is how long this additional advantage might be. And there the facts in many cases is that the additional advantage is marginal, sometimes only in terms of a few weeks. This isn't to say that those weeks aren't worth having, but in making announcements about significant improvements in outcomes there's a real danger of lying to patients.

Aside from urging scientists to be more cautious about the language they use in announcing results, this is also a cautionary tale for patients and people looking for treatment options for friends and relatives. Don't be caught out by this sometimes sloppy use of language, particularly in the mass media who love to trumpet new cures for cancer every other day. Not to mention the endless stream of articles that promise instant weight loss or protection from cancer or diabetes if you eat/don't eat food x, y or z because a study has found some 'significant' result.