A t-test compares an observed difference against the difference sampling noise alone would produce: the effect divided by an estimate of its own standard error. The same raw difference gives a large t when the data are tight and a small t when they are scattered. Three versions are offered — one-sample against a reference mean μ₀, independent for two unrelated groups, and paired for the same units measured twice.
Write x̄ for a sample mean, s for an SD and n for a sample size. One-sample: t = (x̄ − μ₀)/(s/√n), df = n − 1, where the denominator is the standard error of the mean. Paired: reduce to one difference per pair, d = A − B, then apply that same formula to the differences against zero: t = d̄/(s_d/√n), df = n − 1. Independent, Welch (the default): the variances stay separate, se = √(s₁²/n₁ + s₂²/n₂) and t = (x̄₁ − x̄₂)/se, with Welch–Satterthwaite df = se⁴ / [ (s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1) ], which is usually not a whole number. Independent, pooled: clearing the Welch box pools the variances, s_p² = [ (n₁−1)s₁² + (n₂−1)s₂² ]/(n₁+n₂−2) with df = n₁ + n₂ − 2, and this version does assume equal population variances. The p-value comes from the Student t distribution with those df — two-sided by default, or one tail for A > B or A < B — and the interval shown is the 95% confidence interval for the mean difference.
Group A is 23, 25, 28, 24, 27, 22, 26, 25 and group B is 20, 22, 19, 21, 23, 18, 20, 21, both with n = 8. Mean A is 200 ÷ 8 = 25.000 with SD 2.000, mean B is 164 ÷ 8 = 20.500 with SD 1.604, so the difference is 4.500. The Welch standard error is √(4.000/8 + 2.572/8) = 0.9063, giving t = 4.965 with df = 13.4 and a two-sided p of about 0.00024, rendered as < 0.001. The 95% interval is [2.547, 6.453], entirely above zero, and with a pooled SD of 1.813 Cohen's d is 2.483.
The test assumes independent observations and, for the pooled version, roughly equal population variances; the p-value also assumes approximately normal data, which matters most in small samples where one extreme outlier can dominate. The tool checks none of this: no normality test, no Levene or Brown–Forsythe test, no outlier screening, and no correction for multiple comparisons. Twenty unadjusted t-tests at α = 0.05 give roughly a 64% chance (1 − 0.95²⁰) of at least one false positive somewhere. Cohen's d gets no small-sample correction either, so with tiny groups it is biased upward where a package would report Hedges' g, and the interval always uses the two-sided 95% critical value even when you choose a one-sided alternative.
Three things about p are worth keeping straight. It is the probability of data at least this extreme if the null hypothesis were exactly true — not the probability that the null is true, nor the chance your result would replicate. p < 0.05 does not mean the effect is real: with a large enough sample, a difference far too small to matter still produces a small p. And p ≥ 0.05 is not evidence that the effect is zero — it means the data are compatible with no effect, a much weaker claim, especially with eight values per group.
Welch is the safer default: it stays valid when the groups differ in variability and costs little power when they do not.
The mean difference divided by the SD of the differences, usually written dz — generally larger than a d standardised against the pooled SD.
No — it means the test had almost no power. The interval around the difference is enormous, so collect more data instead.
Because it is estimated from the data rather than counted, and the Welch–Satterthwaite formula returns a fractional value.