Good science vs bad science: weeding out fact from fiction

There’s a lot of ‘research’ and ‘evidence’ out there, but how can you tell the difference between real research, which is the result of a proper scientific study, and poor research, which sometimes produces big claims but is actually meaningless?

Here at NCPIC science is our business. We’re based at a university and everything we write and publish needs to be evidence-based. But what exactly does that mean? What is evidence? Are all studies equal when it comes to weighing up the evidence? You may hear people make outrageous claims (e.g. that cannabis cures cancer) and argue that there is evidence and studies to prove this is true. And maybe those studies exist, but are they really proper science and are they relevant to the question? Do they follow the accepted academic rigour to be published in a highly respected academic journal like The Lancet?

What is good science?

Not all science is created equal. There are certain rules and procedures you need to follow in order to produce a meaningful piece of scientific work.

Good science typically starts with a hypothesis. The hypothesis must be falsifiable, which means it must be able to be shown to be untrue or it is not a suitable question to be researched. Hence scientific studies start with a null hypothesis, such as: smoking tobacco does not causes cancer, and then scientists set off to prove whether or not this this hypothesis is true or false. A hypothesis is derived from related research findings or arising from clinical/scientific observations. The scientist then needs to devise the best type of study to prove or disprove their hypothesis. For human research studies, the gold standard is a randomised control trial (this is a type of study where the scientist randomly assigns participants to either receive the treatment/exposure or not, and is ideally unaware which participants are receiving the treatment). Some of the other types of studies include case-control, longitudinal and cohort studies. In order to produce robust findings, the scientist then needs to make sure they recruit enough participants (in order to minimise any result just being chance), and make sure they re recruited in an unbiased manner.

Good science involves getting the word out

Peer review is a very important part of the scientific process. You may have heard of the saying you haven t done science unless you ve published it . The current accepted scientific method is that scientists need to write up their findings in a journal article and then submit it to various academic journals for publishing. Part of the acceptance process to an academic journal is peer review. This means 2-5 other experienced scientists who work in the same field review the submitted article to evaluate it for errors and appropriateness. The types of things that may cause them to have a low evaluation of the work might be: a low sample size, poor statistical analysis, flaws in the methods used to collect the data or that the claims made in the paper are not supported by the study findings.

Another important aspect of good science is whether the experiment can be replicated by other scientists. Replicating an experiment and comparing results will highlight any incidences where results were achieved by chance or where any bias had occurred.

What is bad science, and why is it bad?

There are a variety of ways that the scientific method can be wrongly applied, which can lead to the results not really reflecting the truth. Selectively using data to prove the hypothesis that the scientist desired, bias in the selection process, not correctly measuring the variables of interest (i.e. smoking exposure, and cancer diagnoses), failing to account for other known or likely causes of the outcome, not a large enough sample size to account for chance findings, and incorrect statistical methods are just some of the ways the scientific process can go wrong. A recent well known example of bad science was by Andrew Wakefield who claimed the Measles, Mumps and Rubella (MMR) vaccine caused autism in toddlers, which was later found to be completely false (largely due to the researcher s conflicts of interest).The legacy of this has been reduced vaccination rates in toddlers over the past 10 years in the Western world due to the misplaced fear that this example of bad science promoted.

There are a lot of strong opinions on the topic of cannabis, and a lot of people claiming there is research and evidence behind claims like cannabis cures cancer , but if you delve a little deeper with a critical eye, you ll find none of these studies stand up to closer scrutiny regarding such claims. There are no human clinical trials that show cannabis cures cancer. The screeds of study titles pumped out by proponents invariably turn out to be laboratory test tube studies, animal studies where synthetic THC is injected into brains and other organs, or very small experimental studies. Of course you don t need to be a scientist to work out that if cannabis really did cure cancer everyone would know about it and we would have found in the many hundreds of thousands of cancer trials that cannabis was protective let alone curative!

At NCPIC, we use research in two ways (if it s not our own). Firstly, to underpin materials we are developing, such as our bulletins or factsheets. Secondly, we may repost articles from the media on our Facebook or Twitter page. When selecting and using research, we take care to either select pieces that meet these standards listed above, or identify less quality research that still meets some of the criteria. In the case of the latter, we will often comment on the piece that more research needs to be done, or that it isn’t yet fully researched but is an area or interest.

If you are looking for facts and evidence-based research, it’s always important to know your good science from your bad science, so you don’t make important decisions, or raise your hopes without the correct information guiding you.