Decoding the Hype

Identifying real evidence in health & nutrition studies

John (The John & Calvin Podcast)

Health and nutrition research is messy.

No study is flawless. Every design has limits.

Headlines & influencers turn nuance into clickbait.

The Fix:
study design • basics stats • common pitfalls

Overview

The Research Question
Study Population
Study Design (Power)
Study Structure
How Results Are Reported
Interpreting Results
Controlling for Other Factors
Misleading Studies & Influencers

The Research Question

Main goal of the study

Drives study design, statistics, sample size
Includes measured metric (e.g., BP change, risk)

States the primary outcome (vs. exploratory or secondary)

Best practice (especially in RCTs): pre-register outcomes and analysis plan

Example (published RCT)
“We tested the hypothesis that long-term supplementation with omega-3 fatty acids would reduce cardiovascular events in this population.”
— NEJM, 2012

The Research Question

Questions to Ask

What single question is the study trying to answer?
Is the primary outcome clearly stated—and the right one for that question?
Are secondary outcomes labelled as exploratory, or being oversold?
Does this question matter to you?

Study Population

Population matters for relevance
Not all studies are done on humans, and not all humans are like you.

Cells (in vitro)
Animals (in vivo)
Humans (in vivo)

Key idea

The subjects of the study affect how applicable the results are.

The subjects tell you how far the findings can travel—cells → animals → humans is a “mechanistic → translational → applied” ladder.

Cells (in vitro)
• Isolated cells in dishes or test tubes.
• Great for uncovering biological mechanisms, drug targets, or toxicity pathways.
• Cannot predict whole-body effects on their own.

Animals (in vivo)
• Usually mice or rats; sometimes larger mammals.
• Provide whole-organism context and dosing clues.
• Differences in metabolism and lifespan still limit direct translation to people.

Humans (in vivo)
• Ultimately most relevant, but generalisability varies by:
– Age, sex, health status
– Baseline diet, lifestyle, genetics
• Always ask: “How close is this cohort to me or my patients?”

Illustrative mismatch
A high-protein diet trial on elderly Japanese women with osteoporosis undergoing survival training probably says little about a 35-year-old office worker in Madrid.

Take-home line
Before judging a result, check who or what was studied and whether that population resembles the one you care about.

Study Population

Questions to Ask

Who or what was actually studied?
Why did the researchers pick that population or model?
How similar are these subjects to you or the group you care about?

Study Design

What a study can claim depends on its design.

Observational

• Cross-sectional
• Prospective cohort
• Case-control

Interventional

• RCT
• Pre–post (single-arm)

Evidence Synthesis

• Systematic review
• Meta-analysis

Key message
- Observational → explores associations.
- Interventional → can test causality.
- Synthesis → pools or summarises evidence for stronger estimates.

Observational
- Cross-sectional – snapshot at one point in time.
- Prospective cohort – follow exposure over time to see who develops the outcome.
- Case-control – compare people with vs. without the outcome, look backward for exposures.

→ Good for prevalence, trends, hypothesis generation; cannot by themselves prove cause.

Interventional
- Pre–post / single-arm – measure before vs. after in the same group.
- Randomized Controlled Trial – participants randomly assigned; balances confounders.

→ Designed to answer causal questions and estimate treatment effects.

Evidence Synthesis
- Systematic review – structured search + critical appraisal of all relevant studies.
- Meta-analysis – statistical pooling step inside a systematic review (if data allow).

→ Top of the evidence hierarchy when built on well-done RCTs; still limited by the quality and consistency of those underlying studies.

Other Study Designs (reference)

Observational

• Case-control – with vs without outcome
• Case report / series – detailed look at few patients
• Ecological – group-level data only
• Retrospective cohort – past records follow exposure → outcome

Mixed / Natural

• Longitudinal – same subjects over time
• Natural experiment – exposure assigned by external factors

Interventional

• N-of-1 trial – single participant, alternating treatments
• Cross-over trial – each subject receives all treatments
• Before–after study – compare pre- vs post-intervention

Synthesis

• Systematic review – structured summary, no pooling
• Umbrella review – review of systematic reviews

Study Design

Sample Size & Power

Power (1 − β): the probability of finding an effect if it really exists
Usual target: 80%
Grows with bigger N, larger effect, lower noise

Calculated before the trial to size it properly

Set from the minimum clinically important difference (MCID) (ideally)

1 What power really is

β = Probability of not rejecting the null if the alternative hypothesis true.
80 % is convention (β = 0.20) because it balances feasibility and false-negative risk. 90 % is used in large cardiovascular mega-trials. 80%? If there really is an effect, there’s an 80% chance your study will detect it (i.e. produce a statistically significant result) β is the chance you miss the effect (a false negative). So 1 − β is the chance you catch it (a true positive).

2 How it’s calculated

Depends on four inputs:
1) expected effect size 2) outcome variability 3) α (usually 0.05) 4) N.
➜ change any one and power shifts.

3 Pitfalls of being under- or over-powered

Under-powered → high type-II error (false negative); and anything found that is significant often over-estimates the true effect (Ioannidis 2005).
Over-powered → detect 2 mm Hg or 0.15 kg and call it “significant.” Clinical relevance?

4 Which designs quote power?

RCTs — CONSORT requires a sample-size/power statement in Methods.
Large cohort studies often justify N for rare-event precision.
Cross-sectional & case-control papers rarely show a power calc; they discuss post-hoc detectable effect instead. Cohorts usually enroll as many people as practical and then analyse whatever number of events occur; they report detectable effect sizes afterward rather than prospectively ‘powering’ the study like an RCT.

Study Design

Sample Size & Power

Too small → wide CIs, missed effects
Too large → trivial but ‘significant’ findings

Rule-of-thumb study sizes
• < 50 per arm = pilot / feasibility
• 50 – 300 per arm = typical nutrition RCT
• > 300 per arm = large, clinically robust

Power targets the primary outcome; secondaries often under-powered

5 Primary vs secondary outcomes

Power calc is done for the primary endpoint only.
Secondary outcomes are usually under-powered; a significant result there is hypothesis-generating unless pre-specified with multiplicity correction. The secondary outcomes could be adequately powered A null secondary is only convincing when its CI is tight enough to rule out any benefit we’d care about; otherwise it’s just under-powered—and we should label it inconclusive, not negative.

6 Case-study hook (twin vegan RCT)

Only 22 twin pairs → designed as a pilot.
Authors admit they lacked power for insulin resistance, yet headlines still hyped a –30 % insulin claim.

“When you hear ‘the study was under-powered,’ translate that to ‘big uncertainty and effect sizes that might shrink or vanish at scale.’”

RCTs should at least justify the number of subjects in the trial.

Rule-of-thumb footnote
✦ Chow & Liu, Design & Analysis of Clinical Trials, Ch 3 – rough ranges, always adjust for variance.

Study Design

Questions to Ask

What type of study is this: observational, interventional, or a synthesis?
Can this design support a causal claim, or only association?
Is the control / comparison group appropriate?
For meta-analyses, are the pooled studies similar enough?
Was the sample size justified, and did they report a power calculation for the primary endpoint?

Study Structure

Abstract
Introduction
Methods
Results
Discussion
Conclusion
Supplementary

Abstract – One-paragraph snapshot: question, design, key findings, and take-home.
Introduction – Background, gap in knowledge, explicit research question, and why it matters.
Methods – Who/what was studied, how data were gathered, study design, statistics used; lets you judge rigour and reproducibility.
Results – Objective numbers: tables, figures, raw effect sizes. No interpretation here.
Discussion – Authors’ interpretation, context with past work, strengths, limitations, and speculation.
Conclusion – Concise answer to the research question; sometimes folded into Discussion.
Supplementary – Extra data, detailed protocols, code, or extended figures that wouldn’t fit in the main text.

Talking cue: Emphasize that Methods and Results are where the “meat” lives; if those sections are thin, proceed with caution.

Study Structure

Questions to Ask

Does the Abstract match what’s really in the paper?
Are the Methods detailed enough to judge rigor?
Are Results shown with raw numbers, effect sizes and confidence intervals?
Are any Results missing or highlighted more than reasonably?
Do the Discussion/Conclusion stay within the data’s limits?

How Results Are Reported

Population Metrics

Counts – raw tally

Percent / Proportion – share of the population

Rates – cases per person / time

Standardised rates – adjusted for age / other factors

Incidence vs prevalence – new vs total cases

How Results Are Reported

Group difference

Percent change
Mean difference

Risk comparison

Absolute risk (AR)
Relative risk (RR)
Odds ratio (OR)
Number Needed to Treat (NNT)

How Results Are Reported

Time-to-Event Metrics

Incidence rate
Hazard ratio (HR)
Survival curves

Metrics in Meta-analyses

Forest plot
Standardized Mean Difference

How Results Are Reported

Questions to Ask

Which metric is used: count, rate, risk ratio, mean difference, etc.?
Is that metric appropriate, or could it mislead (e.g., relative vs. absolute)?
Does it answer the main research question?
Could the metric exaggerate or downplay the finding?

Interpreting Results

P-values & Confidence Intervals

Is this effect real? How exact is it?

Concept	What it tells us	Quick example
Statistical significance (p-value)	Chance vs. real effect?	Supplement ↓ BP 5 mm Hg, p = 0.03
Precision (95 % CI)	How exact is the estimate?	5 mm Hg (CI −8 to −2)

Why we need these measures

Studies usually measure a sample, not the entire population. Random variation in the sample can produce differences, even if there’s no real effect. If we had data from the full population, we wouldn’t need p-values or confidence intervals.

P-values test whether those differences are likely due to chance; confidence intervals show how precisely we’ve pinned down the effect size.

Key definitions

P-value – probability of obtaining data this extreme assuming the null-effect model is true.
95 % CI – range that would capture the true effect in ~95 out of 100 identical studies.

ASA informal definition
“A p-value is the probability, under a specified statistical model, that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.” — American Statistical Association, 2016

Notes to hit verbally:
* Threshold test: conventionally p < 0.05 marks ‘statistically significant,’ but the cutoff is arbitrary.
* p-value ≠ probability the effect is real.
* Narrow CI ⇒ precise estimate; wide CI ⇒ more uncertainty.
* A huge sample can make trivial effects “significant”; always couple p with effect size & CI.
* Once p is well below the cutoff (e.g., 0.0003), extra decimal places add no meaning.

Interpreting Results

The p-value

Threshold test → typically p < 0.05
Smaller p → stronger evidence against chance

A p-value means little unless you also know the effect size

p ≠ probability the result is true

“5 mm Hg decrease, p = 0.03” → significant

Interpreting Results

Statistical vs. Practical Significance

Finding (effect)	p-value	Interpretation
−0.15 kg (12 wk)	0.001	Statistically significant; clinically trivial
−8.5 mm Hg systolic BP	0.09	Not statistically significant; could matter if real

Interpreting Results

Questions to Ask

Is the result statistically significant and by how much?
What does the confidence interval say about size and precision?
Is the effect big enough to matter in real life?
If it’s borderline, could high variation, small N, selective data processing or multiple testing explain it?

Controlling for Other Factors

Goal: isolate the effect of one variable while accounting for others.

Coffee & Heart Disease
Control for smoking so coffee isn’t blamed for smokers’ risk.

How?

Include the other factors in the statistical model.

Controlling for Other Factors

Example Cohort Study

Question: Does coffee raise heart-disease risk?

We record coffee cups/day, smoking, age, and heart-disease outcome.

Model with controls
Heart disease = Coffee + Smoking + Age + error
→ Estimates the coffee effect independent of smoking & age

Model without smoking
Heart disease = Coffee + Age + error
→ Smoking still influences both coffee & disease → confounding

Controlling for Other Factors

Under the Hood

\[ Y = \alpha \;+\; \beta_1 X_{coffee} \;+\; \beta_2 X_{smoke} \;+\; \beta_3 X_{age} \;+\; \varepsilon \]

Symbol	Plain English
$Y$	Outcome (heart disease)
$\alpha$	Baseline when all X = 0
$\beta_{1,2,3}$	Effect of each variable holding the others constant
$\varepsilon$	Random noise / unexplained variation

More math? See the linked PSU resource

Controlling for Other Factors

Works only for variables we measured and included.
Unmeasured confounding can still bias results.
Randomized trials help by balancing both known and unknown confounders.

Controlling for Other Factors

Questions to Ask

Did the study adjust for the key known confounders?
What important factors might be missing or unmeasured?
Is the adjustment method explained and sensible?

Core Questions to Ask

Is the main question clear, and is it the one you care about?

Study type: does it address causation, or only association?

Who / what (and how many!) was studied, and how relevant is that to you?

Metric chosen: what does it reveal and what does it hide?

Result strength: significant? How large, precise, and practically useful?

Red flags: stats ‘tricks’, sensational claims, or conflicts?

Studies Can Mislead

Small samples inflate effects

Placebo isn’t inert

Publication bias

Funding or pet theories

Multiple comparisons (p-hacking), cherry-picking subgroups

Self-reported or recall data

Lack of blinding

Surrogate ≠ clinical outcome

Small samples inflate effects – Big swings occur by chance; replication shrinks them.
Placebo isn’t inert – “Control” oils, shakes, or pills often have biological activity.
Publication bias – Null results stay in file drawers; positive ones get published.
Perhaps you’d take fewer supplements if all the no-effect studies were published.
Funding / pet theories – Sponsor or author bias can frame questions or spin conclusions; doesn’t invalidate, but requires scrutiny.
Multiple comparisons / p-hacking – Testing dozens of outcomes raises the odds of a “significant” result by luck; preregistration helps curb this.
Cherry-picked subgroups – Headline highlights “80 % risk drop in women between 30 and 40” but ignores that 10 other subgroups showed nothing; classic media spin.
Self-reported or recall data – Diet, exercise, or symptoms often logged from memory; measurement error can blur real associations.
Lack of blinding – Open-label nutrition trials invite behavior or reporting bias (“I’m on the ‘healthy diet,’ so I drop the afternoon soda”).
Surrogate ≠ clinical outcome – LDL, insulin, DNA-methylation are useful markers, but they aren’t heart-attack counts or lifespan. A surrogate can move while the hard outcome stays flat (ACCELERATE trial, 2016).

Influencers Do Mislead

One study ≠ truth

Petri-dish (mechanism) hype

Diet ≠ morality

Relative risk without absolute numbers

Association ≠ causation

Anecdotes ≠ strong evidence

Claim comes from headline, not published study

Three Things to Remember

Start with the study’s main question
Know the real question, the headline / influencer might not.

Know what kind of study you’re looking at
Association ≠ causation. RCTs are strongest, but every type has limits.

Don’t skip the statistics
A result is only as strong as the methods and statistics behind it.

Point 1 – Start with the study’s main question
If you don’t understand what the study was actually trying to answer, you can’t judge whether the results are meaningful.
Influencers and headlines often jump to bold conclusions that don’t match the research question or primary outcome — especially when the study has multiple endpoints or exploratory results.
This is why the real question — not just what was reported — matters most.

Point 2 – Know what kind of study you’re looking at
Different study types can answer different kinds of questions.
An RCT can test cause-and-effect, while an observational study can only suggest associations — no matter how dramatic the result sounds.
And even RCTs have limitations: they can be underpowered, biased, or irrelevant to your population.
Understanding the design sets the boundaries for what you can believe.

Point 3 – Don’t skip the statistics
The numbers behind the result — sample size, effect size, p-values, confidence intervals — tell you whether a claim is solid or just noise.
Headlines rarely mention the uncertainty, the adjustments, or whether the result was actually powered to detect a difference.
If the stats are shaky, the finding probably is too — even if it sounds exciting.

Follow-up on Vegan Twin Study

Follow-up work the Stanford team did after the 8-week twin RCT

• Nutrient-cost analysis – Using the same delivered-meal menus, they priced all ingredients at U.S. retail averages.
– Vegan meals cost ≈ $1.50 less per person per day (about 14 % cheaper) than the omnivore menus.
– Published as a short “Research Letter” in JAMA Network Open (Feb 2024).

• Epigenetic-age sub-study – They thawed the twins’ stored blood and ran three DNA-methylation clocks (Horvath, PhenoAge, GrimAge).
– Vegan arm showed a –1.9-year change in PhenoAge vs –0.3 years in omnivore (Δ ≈ –1.6 y, p≈0.04).
– Reported in BMC Medicine (2024); authors called it “exploratory.”

How the media mashed it up

• New York Post 29 Jul 2024 headline:
“2 months of eating this diet knocks years off your biological age.”
– Copy lifts the –1.9-year figure from the BMC Medicine paper.
– Mentions the identical-twin setup but ignores the small sample, short duration, and that the meals were chef-prepared.
– No reference to LDL-C, weight loss, or the true cost difference.

• New York Post 22 Nov 2024 headline:
“Vegan diet can make you wealthy, in addition to healthy, study finds.”
– Pulls the grocery-cost analysis ($1.50/day cheaper) and frames it as a wealth booster, again skipping sample size and duration caveats.

“Secondary analyses can be interesting, but headlines often elevate them above the primary outcomes—and leave out all the limitations.”

Symbol	Plain English
\(Y\)	Outcome (heart disease)
\(\alpha\)	Baseline when all X = 0
\(\beta_{1,2,3}\)	Effect of each variable holding the others constant
\(\varepsilon\)	Random noise / unexplained variation