How to read evidence: five concepts disqualify most supplement ads automatically

1. RCT (randomized controlled trial)

A trial that assigns participants by lottery to the "supplement group" and the "placebo group". This is the trustworthy baseline in supplement research. Observational studies ("people who eat a lot of X have less Y") show correlation, not causation. Most supplement ads use observational studies or test-tube experiments as if they were clinical effects.

2. Meta-analysis

A statistical pooling of multiple RCTs. A meta-analysis of 10+ trials is stronger evidence than one "groundbreaking study". What to look at:

• Count and participants: "35 RCTs, 3,165 people" and "1 RCT, 20 people" carry completely different weight.
• Consistency (I²): if trials disagree with each other (I² above 80%), treat the conclusion softly.
• 95% CI: if the confidence interval crosses the null value (e.g. OR 0.61, CI 0.27–1.40), the result is "not significant".

3. Effect size (SMD): how big "significant" is

A p-value only says "not by chance". How much it helps is the effect size. Rule of thumb: SMD 0.2 = small, 0.5 = moderate, 0.8 = large. The 0.3–0.5 range common in supplements means "it works, mildly". If an ad says "dramatic" or "overwhelming" next to an SMD of 0.3, the ad is the lie.

4. GRADE certainty (the quality rating of the evidence)

The same "significant" result can rest on evidence rated high / moderate / low / very low. This site's A–E grades combine certainty and trial count. A claim based on a single low-certainty trial should be read as provisional.

5. Publication bias and industry funding

Trials that found nothing are less likely to get published, and supplement research has a high share of industry funding. Countermeasures: prefer independently funded meta-analyses, and check whether multiple independent groups reached the same conclusion.

Field checklist (run it on every ad)

1. Is the source a human RCT, or animals / test tubes?
2. Is it a single trial, or a meta-analysis?
3. Does the dose match the dose used in trials?
4. How large is the "significant difference" (SMD, %)?
5. Does the claim exceed what the product's regulatory category legally allows (in Japan: the registered scope of a Foods for Specified Health Uses claim)?

These five questions mark most commercial supplement ads as "claim larger than evidence". That is less "fraud" than the industry's standard writing style. Having a way to deduct points is the reader's only defense.