Complementary Medicine Research: 7 Smart Ways to Spot a Study Worth Trusting – by Jayney Goddard MSc, PG Dip Ed, FCMA, FRSM. Founder and President, The Complementary Medical Association. This article is valued at one hour of CMA CPD. See instructions at the end. 

Reading a complementary medicine study well comes down to a handful of habits, not a statistics degree. The most discerning practitioners ask the same questions every time: what was the treatment compared against, how big was the effect, how many people were studied, and has anyone managed to reproduce the result? Master those, and you can tell a robust finding from research noise within minutes – which protects your clients, your reputation and your professional confidence.

At the CMA we are wholehearted about evidence – not because it makes us cautious, but because it makes us credible. A practitioner, trainer or student who can read the literature honestly is far more persuasive in front of a sceptical audience (GP or medical team for example), than one who simply repeats a headline. This guide walks through the seven checks that matter most, with real examples from the research itself.

1. Start with the question the study actually asked

Before anything else, identify the precise question a study set out to answer – because it is rarely the broad claim made in the press release. “Acupuncture eases back pain” is a headline. “Did needling at classical points relieve chronic low back pain more than simulated needling over eight weeks?” is a research question. The two can have opposite answers.

A study can be beautifully conducted and still tell you very little about your own practice if its population, dose or timescale bears no resemblance to the people you treat. Check who was studied (age, condition, severity), what exactly was delivered, for how long, and what outcome was measured. A six-week trial in young, healthy volunteers says almost nothing about a frail eighty-year-old with three comorbidities.

2. Find the control group – and ask what it really controlled for

The single most revealing question about any therapy study is: compared with what? An intervention can look impressive against nothing at all and unremarkable against a fair comparison. This is where complementary medicine research becomes genuinely interesting, because the “fair comparison” is so hard to build.

Two ideas matter here. The first is the placebo – an inert stand-in that lets us separate the effect of a treatment from the effect of simply being treated. The second is the sham – a convincing imitation of the real procedure (e.g. blunt non-penetrating needles, switched-off devices, identical-looking creams) used when a sugar pill will not do.

The placebo response is real but more modest than folklore suggests. In a landmark review of 130 trials that compared placebo with no treatment at all, Hróbjartsson and Gøtzsche found little evidence that placebos produce powerful clinical effects, particularly for objective measures – the benefit showed up mainly in subjective, self-reported outcomes such as pain, and faded as studies grew larger and better designed (Hróbjartsson & Gøtzsche, 2001). The lesson is not that placebo is meaningless, but that “better than nothing” is a low bar. A serious study compares a therapy against a credible sham, not against an empty chair.

Sham controls also explain some of the field’s most honest findings. In a well-run trial of nearly 640 people with chronic low back pain, real acupuncture, simulated acupuncture and usual care were compared head to head. Both real and simulated needling beat usual care – yet the two needling groups did about equally well as each other, suggesting that the ritual and attention mattered more than the specific points used (Cherkin et al., 2009). That is not a reason to abandon a therapy clients value; it is a reason to be precise and truthful about why it may help.

3. “Statistically significant” is not the same as “clinically meaningful”

A “significant” result tells you a difference is unlikely to be pure chance – it tells you nothing about whether the difference is large enough to matter. This distinction is the most common place where good readers are misled, and it is worth internalising for life.

The p-value is the usual culprit. The American Statistical Association issued a rare formal statement on the matter, warning that a p-value does not measure the size of an effect or the importance of a result, and that scientific conclusions should never rest on whether a number crosses 0.05 (Wasserstein & Lazar, 2016). The Association later went further, urging researchers to stop treating “statistical significance” as a pass-or-fail gate altogether (Wasserstein, Schirm & Lazar, 2019).

So what should you look at instead? Two things:

• The effect size – how much better did people actually get? A treatment that lowers a pain score by half a point on a ten-point scale may be “significant” in a large trial and still be useless to a real person.
• The confidence interval – the plausible range for the true effect. A wide interval that includes “no benefit” should give you pause, however small the p-value attached to it.

A good habit: whenever you see the word “significant”, mentally replace it with “detectable” and then go hunting for the actual numbers.

4. Mind the size – and beware getting better on your own

Small studies produce unstable, often exaggerated results, so treat dramatic findings from tiny samples with healthy suspicion. A handful of participants can swing a result through luck alone, which is why early, eye-catching trials so often shrink or vanish when larger teams try to repeat them.

Two natural phenomena flatter almost any therapy if you are not watching for them. The first is natural history: many complaints – colds, back twinges, low moods, flare-ups – improve on their own. If someone seeks help at their worst, they are likely to feel better afterwards, no matter what was done. The second is regression to the mean: extreme measurements tend to drift back toward the average on retesting, so a “high” reading at intake will often look improved at follow-up for purely statistical reasons. A study without a comparison group cannot separate genuine treatment effect from either of these. Before-and-after testimonials, however heartfelt, never can. Nevertheless, they are highly valuable, particularly if testimonials speak to your expertise as a practitioner and the way that the client found the work that you do.

5. One study is never the whole story

No single study settles a question – the weight of evidence comes from whether independent teams can reproduce a finding. A result that has been replicated across different groups, countries and designs is worth far more than a lone spectacular paper.

There are structural reasons to be cautious about any one result. In one of the most cited papers in modern medicine, Ioannidis argued that a large proportion of published findings may be false, driven by small studies, small true effects, flexible analysis and the simple fact that many teams chase the same fashionable questions (Ioannidis, 2005). Add publication bias – positive results get published, “nothing happened” results quietly languish in drawers – and the literature you can easily find is tilted toward good news. When you read a glowing single study, the right question is not “is this true?” but “has anyone else seen this too?”

6. Follow the funding and the motivation

Who paid for the study, and who stood to gain, should shape how much weight you give its conclusions. This is not cynicism – it is routine due diligence, and reputable journals now require authors to declare it.

A trial funded by the maker of the supplement under test is not automatically wrong, but it earns extra scrutiny: was the comparison fair, were unfavourable outcomes reported, was the analysis decided in advance? Look, too, for studies registered before they began. Pre-registration makes it far harder to quietly change the goalposts once the data disappoint – sorry, once the data are in. Independent replication by people with no stake in the result remains the gold standard.

My top tip here is to also consider the researcher’s bias; historically, in complementary medicine, there have been researchers who clearly have an anti-complementary medical bias, and it brings their research papers into question. so if you are looking at quoting a paper, do a little bit of research on the authors to see whether they have an established “position” and possibly an axe to grind!

7. A quick checklist you can use in five minutes

When a new study lands in your inbox, run it through these questions before you share or act on it:

• The question: Does the population and treatment actually resemble my practice?
• The control: Was the therapy compared against a credible placebo or sham – or against nothing?
• The effect size: How much did people improve, in real terms, not just “significantly”?
• The numbers: How many participants, and how wide is the confidence interval?
• The replication: Has this been seen by other, independent teams?
• The interests: Who funded it, and was it registered in advance?
• The honesty test: Does the conclusion match the data, or does it reach further than the results allow?

Why this matters for your practice

Reading evidence carefully is not about becoming hard-nosed or dismissive – it is about becoming trustworthy. The therapies many of us use can offer real comfort, real relief and real value, and we honour them best by describing their benefits accurately rather than overselling them. A practitioner who can say “the strongest evidence suggests this helps with X, the picture is mixed on Y, and we genuinely do not yet know about Z” is the practitioner a doctor will refer to, a journalist will quote and a client will return to.

That blend – open-hearted about what people experience, clear-eyed about what the data show – is exactly the standard the CMA exists to uphold. The more fluently you read the research, the more confidently you can hold both at once.

Frequently asked questions

What is the difference between a placebo and a sham control?

A placebo is an inert stand-in, such as a sugar pill, used to mimic a treatment that comes in pill form. A sham control is a convincing imitation of a physical procedure – for example, blunt non-penetrating needles or a switched-off device – used when a pill cannot stand in for the therapy being tested. Both serve the same purpose: separating the effect of the treatment itself from the effect of being treated.

Does “statistically significant” mean a treatment works?

No. Statistical significance only indicates that a result is unlikely to be due to chance; it says nothing about how large or clinically useful the effect is. The American Statistical Association has stated plainly that a p-value does not measure the size or importance of an effect. Always look for the effect size and confidence interval alongside any claim of significance.

Why should I distrust a single impressive study?

Single studies, especially small ones, are easily skewed by chance, bias and the way results are selected for publication. Confidence comes from replication – independent teams reaching the same conclusion in different settings. A finding that has not yet been reproduced is a promising lead, not an established fact.

Is the placebo effect powerful enough to explain most complementary therapy results?

The placebo response is real but more modest than commonly believed, and shows up mainly in subjective outcomes such as pain. The honest position is that attention, ritual and expectation contribute to many therapy effects, which is a reason to describe benefits precisely rather than to dismiss them.

What is the single most useful question to ask about any therapy study?

“Compared with what?” The comparison group reveals more than almost anything else. A therapy can look remarkable against no treatment and ordinary against a credible sham, so the quality of the comparison tells you how much to trust the result.

Activity: Reading and reflection – “How to Read a Complementary Medicine Study Without Being Fooled”
Type of CPD: Self-directed learning (reading + structured reflection)
Suggested time: 30–45 minutes

Use this box to turn the article into a recorded piece of continuing professional development. Read it through, then work the prompts below in your own words. Honest, specific reflection counts for far more than length – a few considered sentences against each question is plenty.

Intended learning outcomes

By the end of this activity you should be able to:

• Identify the comparison group in a therapy study and judge whether it was a fair test (placebo, sham, or nothing at all).
• Explain why “statistically significant” does not mean “clinically meaningful”, and locate the effect size and confidence interval instead.
• Recognise how small samples, natural history, regression to the mean and publication bias can flatter a result.
• Apply a quick appraisal checklist before sharing or acting on a new piece of research.

Reflective prompts

1. Before reading: How did I usually decide whether a study was worth trusting? Be honest about what I relied on – the headline, the journal name, my own instinct?
2. What challenged me: Which point in the article most unsettled a belief I held about evidence in my field? Why?
3. In my own practice: Can I think of a claim I have made to a client, a colleague or on my website that I would now phrase more carefully? What would I say instead?
4. Communication: How might reading the evidence more accurately change the way I explain a therapy’s benefits to a sceptical client or a referring GP?
5. Limits of my knowledge: Where do I need to say “the evidence is mixed” or “we do not yet know” – and am I comfortable doing so?

Action plan

Commit to one to three specific, achievable actions. For example:

• Run the five-minute checklist over the next study I am tempted to share.
• Review one claim on my website or marketing materials for accuracy.
• Practise one honest “we do not yet know” sentence I can use with confidence.

Remember to log in to your CMA Membership portal to upload your work on this article. 

References

1. Hróbjartsson A, Gøtzsche PC. Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. New England Journal of Medicine. 2001;344(21):1594–1602. doi:10.1056/NEJM200105243442106
2. Cherkin DC, Sherman KJ, Avins AL, et al. A randomized trial comparing acupuncture, simulated acupuncture, and usual care for chronic low back pain. Archives of Internal Medicine. 2009;169(9):858–866. doi:10.1001/archinternmed.2009.65
3. Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician. 2016;70(2):129–133. doi:10.1080/00031305.2016.1154108
4. Wasserstein RL, Schirm AL, Lazar NA. Moving to a world beyond “p < 0.05”. The American Statistician. 2019;73(sup1):1–19. doi:10.1080/00031305.2019.1583913
5. Ioannidis JPA. Why most published research findings are false. PLoS Medicine. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124

About the Author

Jayney Goddard MSc, PG Dip Ed, FCMA, FRSM is the Founder and President of the Complementary Medical Association (CMA), which she has built over more than three decades into a leading international standards-setting body for complementary and integrative medicine. A respected writer, educator and speaker on health, wellness and longevity, she is known for an approach that is warm and humane yet firmly grounded in evidence – championing the therapies people value while remaining clear-eyed about what the research does, and does not, show. Her work helps practitioners hold both at once: open-hearted about the people they care for, and honest about the science behind their practice.

Read more from Jayney at The CMA, or sign up for her newsletter.