Hypothesis testing explained: p-values, errors and tails

What are the null hypothesis, the alternative hypothesis and alpha?

A test starts by assuming the null hypothesis (H₀): no difference between groups, no relationship between variables. The alternative hypothesis (H₁) is what you expect if H₀ is wrong. The test then asks: if H₀ were true, how surprising would data like mine be?

Before you collect data, you fix alpha (α), the significance level. It is the false-positive rate you are willing to accept: with α = .05, if H₀ is true you will wrongly reject it in about 5% of studies. The decision rule is simple. If p ≤ α, reject H₀. If p > α, do not reject it.

A courtroom is a fair comparison. The accused is presumed innocent (H₀) and convicted only if the evidence is strong enough. A “not guilty” verdict means the evidence was insufficient, not that innocence was proved. A non-significant result means the same thing: not enough evidence against H₀.

The .05 convention is a habit, not a law of nature. Some fields use .01; particle physics uses far stricter thresholds. Whatever you choose, state it in your methodology chapter before the results.

What does a p-value tell you, and what doesn’t it?

Suppose a t-test comparing two teaching methods gives p = .03. That means: if the two methods truly produced the same average score, and the test’s assumptions held, a difference as large as yours or larger would turn up in about 3% of samples of this size. That is all it means.

In 2016 the American Statistical Association issued a statement with six principles on p-values, the first time the ASA board had issued a statement on the issue. Two of them are worth copying into your notebook: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone,” and “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” The statement also says conclusions should not rest only on whether a p-value passes a threshold, and that proper inference needs full reporting.

In practice, this means a thesis should never say “the result is 97% certain” because p = .03, and should treat p = .049 and p = .051 as nearly the same evidence, whatever the decision rule says.

What are Type I and Type II errors, and what is power?

Every test decision can be wrong in one of two ways.

In a thesis, a Type I error might be concluding that a training programme improved scores when it didn’t. A Type II error might be missing a real improvement because only 40 people took part. The two trade off: as the NIST/SEMATECH e-Handbook of Statistical Methods puts it, the risk β increases as the risk α decreases. Making α stricter without enlarging the sample makes real effects harder to find.

Power

Power is the probability of rejecting H₀ when a real effect of a given size exists, so power = 1 − β. It depends on four things: the true effect size, the sample size, α, and whether the test is one- or two-tailed. The usual target is 0.80.

The numbers are sobering. For an independent-samples t-test at α = .05 (two-tailed) and a medium effect (d = 0.5), you need about 64 people per group for 80% power. With 42 per group, power is about 62%: more than a one-in-three chance of missing an effect that is really there. Our power analysis calculator does these calculations for t-tests, ANOVA and correlations, and the sample size guide explains how to justify the effect size you plan for.

Plan power before you collect data. “Observed power” calculated afterwards from your own result adds nothing, because it is a direct function of the p-value you already have.

Should you use a one-tailed or a two-tailed test?

A two-tailed test looks for a difference in either direction. At α = .05 it puts .025 in each tail; for a z-test, the critical value is ±1.96. A one-tailed test puts all .05 in one tail, so the critical value drops to 1.645 and an effect in the predicted direction is easier to detect. The price is that an effect in the other direction, however large, cannot be declared significant.

A one-tailed test is defensible only when all of these hold:

• the direction was stated in your synopsis or proposal, before data collection;
• theory or earlier studies give a clear reason for it;
• an effect in the opposite direction would lead to the same conclusion as no effect.

What is not defensible is running a two-tailed test, getting p = .08, and then switching to one-tailed to halve it. Examiners know this move. Most software reports two-tailed p-values by default, so check the label on the output before you interpret it. When in doubt, use two-tailed; nobody will question it.

Why report effect sizes and confidence intervals with the p-value?

The p-value answers “is there evidence of an effect?” It does not answer “how big is it?” or “how precisely do we know?” Two studies show why that matters. With 5,000 respondents, a trivial difference can give p < .001. With 30, a meaningful difference can give p = .20. The p-values alone would mislead you both times.

An effect size puts the size of the result on a standard scale: Cohen’s d for a difference in means, r or R² for relationships, eta squared for ANOVA, odds ratios for logistic regression. The statistical test guide lists which goes with which test. Cohen’s rough benchmarks for d (0.2 small, 0.5 medium, 0.8 large) are a last resort; comparing your effect with those in similar studies in your field says far more.

A 95% confidence interval gives the range of values for the true effect that are reasonably compatible with your data. If the interval for a difference in means runs from 0.96 to 8.04 points, the data point to a real reduction, somewhere between barely noticeable and substantial. If a 95% interval for a difference includes zero, the two-tailed test at α = .05 will not be significant. Wide intervals are an honest sign of a small sample.

How do you report a hypothesis test in APA style?

Here is a complete example. The study and every number in it are invented for illustration.

Invented scenario: 84 B.Ed. trainees were randomly assigned to a four-week stress-management workshop or to a waiting list (42 each). H₁ (non-directional): anxiety scores after four weeks differ between the groups. α = .05, two-tailed. An independent-samples t-test was used.

Invented result, written in APA style:

Trainees who attended the workshop reported lower anxiety (M = 38.6, SD = 7.9) than those on the waiting list (M = 43.1, SD = 8.4), t(82) = −2.53, p = .013, d = 0.55, 95% CI of the difference [−8.04, −0.96]. H₀ was rejected.

Each piece does a job: the group means and SDs show the direction and spread; t(82) gives the statistic with its degrees of freedom (84 − 2); the exact p; the effect size; and the interval. The APA Style numbers and statistics guide sets the conventions: statistical symbols in italics, exact p values rounded to two or three decimal places, values below .001 written as p < .001, and no zero before the decimal point for statistics that cannot exceed 1.

If the result had not been significant, the report keeps the same shape: “The groups did not differ significantly, t(82) = −1.12, p = .266, d = 0.24, 95% CI [−5.53, 1.55]. H₀ was not rejected.” Then discuss power: with 42 per group, a small effect could easily be missed.

What hypothesis-testing mistakes do examiners find in Indian PhD theses?

These come up again and again in examiner reports and pre-submission seminars:

• “H₀ is accepted.” Write “not rejected” or “the data did not support H₁”. If you need to show that two groups are practically the same, that calls for an equivalence test, not a non-significant t-test.
• p = 0.000 copied from SPSS. SPSS rounds to three decimals, so .000 means p < .0005. Report p < .001 in text and tables.
• Dozens of hypotheses with no correction. Eight demographic variables tested against six constructs is 48 tests. At α = .05, about 2 or 3 would come out significant by chance even if nothing were going on.
• Treating significance as importance. With 1,200 respondents, a 0.1-point difference on a 5-point scale can be significant and still mean nothing in practice.
• Choosing the tail, the test or the outliers after seeing the data. Each choice made to reach p < .05 raises the real false-positive rate above the α you report.
• Reporting only the significant results. Every planned hypothesis gets a result, supported or not.

What to do about many tests

Across m independent tests with every null true, the probability that one or more is significant by chance is 1 − 0.95^m: about 23% for 5 tests, 64% for 20 and over 90% for 48. Three responses, in order of preference:

1. Test fewer, planned hypotheses. Each should follow from an objective and your conceptual framework. Label everything else exploratory and don’t build conclusions on it.
2. Correct for the family of tests. The Bonferroni correction tests each at α/m (for 10 tests, .005). Holm’s (1979) step-down method controls the same error rate with more power. Where you are screening many items, the Benjamini–Hochberg (1995) procedure controls the false discovery rate instead.
3. Report every test you ran, so the reader can judge the risk for themselves.