Before you trust the p-value

T-test assumptions

A t-test gives a trustworthy p-value when its assumptions hold well enough. Some matter a lot and cannot be fixed afterwards; others matter less than many guides suggest.

Checked on November 21, 2026. Results computed by the site's engine and tested against R and SciPy.

1. Independent observations

This is the most important assumption and the one no calculation can check. Each observation should carry its own information. Measuring the same person several times and treating the results as separate people, or comparing two classes as if every student were independent, breaks it. Repeated values make the standard error too small and the p-value too optimistic. Fix it in the design: use the paired test for two linked measurements, or a model for clustered data.

2. Roughly normal data, or enough of it

The t-test assumes the sample means are normally distributed. That holds if the data themselves are roughly normal, and it holds approximately for skewed data once samples are moderately large, because of the central limit theorem. A common rule of thumb is about 30 per group, but strongly skewed data need more and nearly symmetric data need fewer.

For the paired test, it is the differences that need to be roughly normal, not each column. For the independent test, it is each group.

Check with a histogram or a normal quantile plot rather than a formal test. Normality tests reject trivial departures in large samples, where they do not matter, and miss serious ones in small samples, where they do. The calculators report skewness for raw data as a quick signal; values beyond about 1 in absolute size are worth a look.

3. No extreme outliers driving the result

The mean and SD are sensitive to extreme values, so a single outlier can change a t-test a lot. In R's example data, adding one value of 200 to a group of 14 moves Welch's p from 1.855 × 10−5 to 0.1245, and the 95% interval widens from −11.1 to −4.9 to −47.2 to 6.4.

The calculators flag values more than 1.5 interquartile ranges beyond the quartiles, but never remove them. Check whether a flagged value is an error. If it is real, keep it, and consider a rank test or report the analysis with and without it.

4. Equal variances (Student's test only)

Student's pooled test assumes both groups share one variance. Welch's test does not, which is why it is our default for independent groups. If you use Student's test because your course requires it, look at the SD ratio in the result; when it is well above 2 and the group sizes differ, the two tests can disagree and Welch's answer is the one to trust. See Welch vs Student.

5. A sensible measurement scale

Means and SDs need numbers where differences make sense: lengths, times, scores on a well-built scale. For ranks or a short ordered scale such as a five-point rating, a rank test often fits better.

What to do when assumptions fail

ProblemOption
Pairs or repeated measurementsPaired t-test, or a mixed model for more than two
Small, skewed samplesMann-Whitney U or Wilcoxon signed rank
Right-skewed positive data (incomes, times)Analyze the logarithms, then report a ratio of geometric means
Unequal variancesWelch's test, already the default here
Outliers from errorsCorrect or remove the error, and say so
Outliers that are realRank test, or report with and without

On the permeability data in R's Mann-Whitney example, a one-sided Welch test gives p = 0.0311 and the rank test gives 0.1272. With such small samples, the rank test's lack of reliance on normality matters. Choose the method before seeing which gives the smaller p.

Assumptions that are often overstated

  • The raw data do not need to be perfectly normal; with moderate samples the test is robust to mild skew.
  • Equal sample sizes are not required, though they make both tests more robust.
  • A significant normality test is not a reason by itself to abandon a t-test with large samples.