Hypothesis testing explained: p-values, errors and tails

What are the null hypothesis, the alternative hypothesis and alpha?

You start every test by taking the null hypothesis (H₀) as true. As per H₀ there is no difference between your groups, or no relation between your variables. The alternative hypothesis (H₁) is what you expect when H₀ is wrong. Then the test asks only one question. Suppose H₀ is true, how surprising would data like mine be?

You fix alpha (α), the significance level, before you collect any data. Alpha is the false-positive rate you agree to live with. Take α = .05. If H₀ is true, you will reject it wrongly in about 5% of studies. The decision rule after that is easy. If p ≤ α you reject H₀, and if p > α you simply don’t reject it.

The courtroom comparison help here. The accused is presumed innocent, which is H₀, and gets convicted only when the evidence is strong enough. A “not guilty” verdict from the judge just means the evidence wasn’t enough to convict, so innocence is still not proved by anybody. A non-significant result should be read in the same way, as not enough evidence against H₀.

The .05 cut-off is mostly convention, and nothing in nature fixes it there. Some fields work with .01, and particle physics goes for far stricter thresholds. Whichever level you pick, write it in your methodology chapter itself.

What does a p-value tell you, and what doesn’t it?

Say you compare two teaching methods with a t-test and you get p = .03. Now suppose both methods really gave the same average score, and the assumptions of the test held. In that case a difference as large as yours, or larger, would come up in about 3% of samples of this size. The p-value means this much only.

The American Statistical Association put out a statement in 2016 with six principles on p-values. It was the first time the ASA board had issued a statement on this issue, and for a thesis two principles matter most. One says “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” The other one says “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” It also warns you not to draw a conclusion just because p went below some cut-off. Full reporting is asked for too.

So never write in the thesis that “the result is 97% certain” because p = .03. And treat p = .049 and p = .051 as almost the same evidence, even though the decision rule puts them on different sides.

What are Type I and Type II errors, and what is power?

Any decision from a test can go wrong in two ways.

Take a thesis on a training programme. A Type I error would be concluding that the programme improved scores when it actually did not. A Type II error would be missing a real improvement because only 40 people took part. You cannot push both risks down together. The NIST/SEMATECH e-Handbook of Statistical Methods says the risk β increases as the risk α decreases. So if α is made stricter and the sample is not made bigger, real effects become harder to detect.

Power

Power means the probability of rejecting H₀ when a real effect of a given size is present. So power = 1 − β. Four things move it up or down, the true effect size, sample size, α, and whether you go one- or two-tailed. Mostly scholars keep 0.80 as the target.

When you see the actual numbers it is a little worrying. Say you are doing an independent-samples t-test with a medium effect (d = 0.5), two-tailed, α = .05. For 80% power you need about 64 people per group. With 42 per group, power comes down to about 62%. Then the chance of missing an effect that is really there is more than one in three. Our power analysis calculator does these sums for t-tests, ANOVA and correlations. How to justify the effect size you plan for is explained in the sample size guide.

Do the power planning before data collection. An “observed power” calculated afterwards from your own result does not add anything, because it is just a direct function of the p-value you already have.

Should you use a one-tailed or a two-tailed test?

In a two-tailed test you are looking for a difference on either side, higher or lower. At α = .05 it keeps .025 in each tail, and for a z-test the critical value is ±1.96. A one-tailed test keeps the whole .05 in one tail. Because of that the critical value falls to 1.645, and an effect in the predicted direction becomes easier to detect. You pay for this, though, because an effect in the other direction cannot be declared significant, however large it is.

You can defend a one-tailed test only when all of these are true.

• you stated the direction in your synopsis or proposal, before data collection,
• theory or earlier studies give a clear reason for that direction, and
• an effect in the opposite direction would lead you to the same conclusion as no effect.

Suppose you run a two-tailed test, get p = .08, and then change over to one-tailed to cut it in half. This cannot be defended, and examiners know the move well. Most software gives two-tailed p-values by default, so check label on the output before you read the value. When in doubt use two-tailed, which nobody will question.

Why report effect sizes and confidence intervals with the p-value?

The p-value answers “is there evidence of an effect?” It gives no answer to “how big is it?” or to “how precisely do we know?” Think of two studies. With 5,000 respondents, even a trivial difference can give p < .001. With 30 respondents, a meaningful difference can give p = .20. In both cases the p-value alone would mislead you.

An effect size puts the size of your result on a standard scale. Cohen’s d is used when you compare means. For relationships people report r or R². In ANOVA it is eta squared, and logistic regression gives odds ratios. Which one goes with which test is listed in the statistical test guide. Cohen gave rough benchmarks for d (0.2 small, 0.5 medium, 0.8 large), but keep them for a last resort. Comparing your effect with the effects in similar studies of your own field says much more to the reader.

A 95% confidence interval gives the range of values for the true effect that fit reasonably well with your data. For example, the interval for a difference in means may run from 0.96 to 8.04 points. So the data do point to a real reduction, only its size could be anywhere from barely noticeable to quite substantial. When a 95% interval for a difference includes zero, the two-tailed test at α = .05 will not be significant. If your interval is wide, that is an honest sign of a small sample.

How do you report a hypothesis test in APA style?

A full worked example follows. The study is invented, and so is every number in it, only for illustration.

Invented scenario: 84 B.Ed. trainees were randomly assigned either to a four-week stress-management workshop or to a waiting list, with 42 in each group. H₁ (non-directional) says that anxiety scores after four weeks differ between the groups. α = .05, two-tailed. The test used was an independent-samples t-test.

Invented result, written in APA style:

Trainees who attended the workshop reported lower anxiety (M = 38.6, SD = 7.9) than those on the waiting list (M = 43.1, SD = 8.4), t(82) = −2.53, p = .013, d = 0.55, 95% CI of the difference [−8.04, −0.96]. H₀ was rejected.

Every part has its own job. The group means and SDs give the direction and the spread. t(82) shows the statistic along with its degrees of freedom (84 − 2). After that come the exact p, the effect size and the interval. The rules here are from the APA Style numbers and statistics guide. Statistical symbols are put in italics. Exact p values are rounded to two or three decimal places, and anything below .001 is written as p < .001. For statistics that cannot go above 1, no zero is written before the decimal point.

If the result had come out not significant, you write it in the same shape only, for example “The difference between the groups was not significant, t(82) = −1.12, p = .266, d = 0.24, 95% CI [−5.53, 1.55]. H₀ was not rejected.” After that you should discuss about power. With 42 per group, a small effect could easily get missed.

What hypothesis-testing mistakes do examiners find in Indian PhD theses?

Examiner reports and pre-submission seminars keep pointing out the same mistakes.

• “H₀ is accepted.” Write “not rejected”, or “the data did not support H₁”. To show that two groups are practically the same, you need an equivalence test. A non-significant t-test cannot show that.
• p = 0.000 copied from SPSS. SPSS rounds to three decimals, so .000 actually means p < .0005. In the text and in tables, report it as p < .001.
• Dozens of hypotheses with no correction. Test eight demographic variables against six constructs and you already have 48 tests. Even with nothing real there, about 2 or 3 will show significant at α = .05 just by chance.
• Thinking a significant result must be important. When there are 1,200 respondents, a 0.1-point difference on a 5-point scale can be significant and still have no practical meaning.
• Choosing the tail, the test or the outliers after seeing the data. Any such choice made to get p < .05 pushes the real false-positive rate higher than the α you report.
• Reporting only the significant results. The result of every planned hypothesis should go in the thesis, also those your data did not support.

What to do about many tests

Take m independent tests where every null is true. The probability that at least one of them comes significant by chance is 1 − 0.95^m. For 5 tests that is about 23%, for 20 it is 64%, and for 48 it goes over 90%. There are three options, given here in order of preference.

1. Test fewer, planned hypotheses. Each one should come out of an objective and your conceptual framework. Anything outside that, label it as exploratory and keep it away from your main conclusions.
2. Correct for the family of tests. With the Bonferroni correction each test is done at α/m, so for 10 tests it is .005. Holm’s (1979) step-down method controls the same error rate and gives more power. If you are screening many items, use the Benjamini–Hochberg (1995) procedure instead, which controls the false discovery rate.
3. Report every test you ran, so that the reader can judge the risk on their own.