What does the word significant mean to you? In general usage it's another word for important or substantial, but when it comes to scientific results there's a very different spin to the word. In science results are usually classed as being 'statistically significant', and by science we include medicine and medical research. Very often new results are announced and we are told that these are 'significant', this is particularly the case when these results are announced in the press, especially the popular press rather than the scientific press. The magic phrase 'statistically significant' often gets turned into 'significant', and for the reader not aware of the difference between the two it's normally taken to mean that a result is important or substantial.
Unfortunately 'statistically significant'
is just a way of saying that there's only a certain chance that the
results could have happened by chance. Normally scientists will talk
about a result being statistically significant at a p-level, often
p=0.05, which is to say that there's around a 5% (1 in 20) chance that
the result could have happened by accident. It doesn't tell us that the
result is important, or substantial or even particularly interesting,
all it tells us if that you repeated the experiment (or drug trial) you
would expect to have to run it 20 times before you got this result by
There are a couple of obvious things to say at
this point. The first is to say that p=0.05 sets a pretty low bar.
Another way of looking at this is to say that 5 out of every 100 results
are just due to chance. Those odds might be fine for the casino or the
occasional horse race, but they're way too high for drugs that can kill
(or save) people. Surely for medical research we need to be looking at
setting the bar higher - we should be looking at results at the p=0.01
or p=0.001 level to make sure that we're not getting spurious results.
Even then, a result that is significant at the p=0.001 level means that
we're ten times more sure it's not an accident compared to the p=0.01
level, but that's all it means.
significance doesn't tell us what we really want to know, which is
whether a result is important or not. This is really important when
scientists say that a new treatment offers a significant survival
advantage to patients. Even if the researcher is absolutely clear to the
press and says that the new treatment offers a 'statistically
significant' survival advantage, most people will interpret that as
meaning that the patient will gain an important survival advantage - in
other words that they will survive for longer. But this isn't what that
statement means, what it means is that the new treatment offers a
survival advantage that only has a 5% chance of being accidental and
unrelated to the treatment. What it doesn't say at all is how long this
additional advantage might be. And there the facts in many cases is that
the additional advantage is marginal, sometimes only in terms of a few
weeks. This isn't to say that those weeks aren't worth having, but in
making announcements about significant improvements in outcomes there's a
real danger of lying to patients.
Aside from urging
scientists to be more cautious about the language they use in announcing
results, this is also a cautionary tale for patients and people looking
for treatment options for friends and relatives. Don't be caught out by
this sometimes sloppy use of language, particularly in the mass media
who love to trumpet new cures for cancer every other day. Not to mention
the endless stream of articles that promise instant weight loss or
protection from cancer or diabetes if you eat/don't eat food x, y or z
because a study has found some 'significant' result.