Calculate p-values for z-tests, t-tests, and chi-square tests with step-by-step solutions
Z-test: z = 2.5, two-tailed
T-test: t = 1.8, df = 15, two-tailed
Chi-square: χ² = 12.5, df = 6
Z-test: z = 3.0, one-tailed
T-test: t = -2.1, df = 25, one-tailed
Chi-square: χ² = 25.2, df = 10
The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the observed value, assuming the null hypothesis is true.
Z-test: For large samples or known population variance
T-test: For small samples with unknown variance
Chi-square: For categorical data and goodness-of-fit tests
Determine if a new medication performs significantly better than placebo by comparing treatment group outcomes using p-values.
Evaluate whether design changes increase conversion rates by testing if differences between versions A and B are statistically significant.
Test if product defect rates exceed acceptable thresholds or if production batches meet quality specifications.
Analyze poll data to determine if observed differences between demographic groups reflect true population differences or sampling variation.
Test if a trading algorithm generates returns significantly different from market benchmarks or if risk factors matter.
Confusing p-value with hypothesis probability
P-value is NOT the probability that your hypothesis is true. It's the probability of seeing your data IF the null hypothesis were true.
Using 0.05 as an absolute threshold
The 0.05 cutoff is arbitrary convention. P = 0.051 is not fundamentally different from p = 0.049. Consider context and effect size.
Ignoring practical significance
With large samples, tiny meaningless effects can have p \u003c 0.001. Always ask: Is the effect size meaningful in practice?
P-hacking and multiple testing
Running many tests and only reporting significant ones inflates Type I error. Use corrections like Bonferroni for multiple comparisons.
Best Practice
Report p-values alongside effect sizes, confidence intervals, and sample sizes. Interpret results in context, not just by arbitrary thresholds.
| P-Value Range | Interpretation | Common Usage | Strength of Evidence |
|---|---|---|---|
| p < 0.001 | Highly significant | Medical trials, safety-critical research | Very strong evidence against H₀ |
| 0.001 ≤ p < 0.01 | Very significant | Experimental research, clinical studies | Strong evidence against H₀ |
| 0.01 ≤ p < 0.05 | Significant (conventional) | Most scientific research, standard threshold | Moderate evidence against H₀ |
| 0.05 ≤ p < 0.10 | Marginally significant | Exploratory research, preliminary findings | Weak evidence, suggestive trend |
| p ≥ 0.10 | Not significant | Fail to reject null hypothesis | Insufficient evidence against H₀ |
Important Note: These thresholds are conventions, not universal laws. Always consider your field's standards, sample size, effect size, and practical importance when interpreting p-values.
Hypothesis testing involves two types of potential errors, each with different consequences and controlled by different parameters.
Definition: Rejecting H₀ when it's actually true
Probability: α (significance level, typically 0.05)
Example: Concluding a drug works when it doesn't
Control: Lowering α (use stricter threshold like 0.01)
Definition: Failing to reject H₀ when it's actually false
Probability: β (depends on sample size and effect size)
Example: Missing a real drug effect in clinical trial
Control: Increase sample size or relax α
Power is the probability of correctly rejecting H₀ when it's false. Higher power (typically 0.80 or 80%) means better ability to detect true effects. Power increases with larger sample sizes and larger effect sizes.
| H₀ is True | H₀ is False | |
|---|---|---|
| Reject H₀ | Type I Error (α) | Correct Decision (Power = 1-β) |
| Fail to Reject H₀ | Correct Decision (1-α) | Type II Error (β) |
Free lessons on significance tests, p-values, and Type I/II errors.
Open-source college textbook covering null hypotheses, p-values, and test procedures.
The American Statistical Association's official guidance on proper use and interpretation of p-values.
I got through an entire statistics course thinking p < 0.05 meant "probably true." It doesn't. Not even close.
A 2016 survey by the American Statistical Association found that the majority of researchers — people who use p-values daily — couldn't correctly define what one means. Some thought it was the probability the result is due to chance. Others thought it measured the probability the hypothesis is true. Both wrong.
Here's what a p-value actually is: the probability of seeing data this extreme (or more extreme) if the null hypothesis were true. Read that again. It's not about your hypothesis being right. It's about how surprising your data would be in a world where nothing interesting is happening.
Forget formulas for a minute. Grab a coin.
Your null hypothesis: this coin is fair (50/50). You flip it 20 times and get 15 heads. Suspicious? Maybe. But how suspicious?
The p-value answers: "If this coin really is fair, what's the probability of getting 15 or more heads in 20 flips?" Run the binomial math and you get p = 0.021. About a 2.1% chance.
That 0.021 doesn't mean there's a 2.1% chance the coin is fair. It means: if the coin is fair, you'd see a result this lopsided only about 2.1% of the time. The p-value assumes the null hypothesis is true and asks how weird your data looks under that assumption.
Since 0.021 < 0.05, most researchers would "reject the null" and conclude the coin is probably loaded. But "probably" is doing a lot of heavy lifting in that sentence.
Ronald Fisher — the statistician who popularized significance testing in the 1920s — once wrote that 0.05 was a "convenient" threshold. Not sacred. Not mathematically derived. Convenient.
It stuck. Journals adopted it. Funding agencies required it. Entire careers now hinge on whether a number lands at 0.049 or 0.051. The difference between "publishable" and "file drawer" is often a rounding error.
Some fields have started pushing back. Particle physics uses p < 0.0000003 (the "5-sigma" standard) before claiming a discovery. Genomics uses even stricter thresholds because they're testing thousands of hypotheses simultaneously. Meanwhile, psychology and social science are still arguing about whether 0.05 is too lenient.
The replication crisis — where landmark studies failed to reproduce — wasn't caused by p-values alone. But the obsession with crossing the 0.05 line created perverse incentives: p-hacking, selective reporting, and "HARKing" (hypothesizing after results are known). When the threshold becomes the goal, the science suffers.
Let's say you run a study testing whether a new drug lowers blood pressure. You get p = 0.03. Here's the scorecard:
| Statement | True or False? |
|---|---|
| "There's a 3% chance the drug doesn't work" | False |
| "There's a 97% chance the drug works" | False |
| "If the drug had no effect, we'd see data this extreme only 3% of the time" | True |
| "The drug has a large effect" | False — p-values say nothing about effect size |
That third row is the only correct interpretation. The p-value lives entirely inside the null hypothesis world. It can't tell you the probability that your alternative hypothesis is true — that requires Bayesian statistics, which is a different conversation entirely.
And here's the part that trips up even experienced researchers: a small p-value with a huge sample size might reflect a real but trivially small effect. A drug that lowers blood pressure by 0.5 mmHg could easily hit p < 0.001 with 50,000 participants. Statistically significant? Yes. Clinically meaningful? Not remotely.
Rejecting the null when it's actually true — that's a Type I error (false positive). The p-value threshold directly controls this: at α = 0.05, you accept a 5% false positive rate.
But there's a mirror image. Failing to reject the null when it's actually false — that's a Type II error (false negative). The probability of avoiding this is called statistical power, and most studies are woefully underpowered.
A study with 80% power (the commonly recommended minimum) still misses real effects 20% of the time. Many published studies have power closer to 50%, which means they're basically coin flips for detecting true effects. Your p-value might be 0.03, but if the study only had 40 participants, the confidence you should place in that result is... limited.
The relationship between these concepts matters more than any single number. A misleading percentage can distort your understanding of data just as badly as a misinterpreted p-value.
P-values aren't useless. They're just one piece of a bigger picture. Here's what a responsible analysis includes:
Effect Size
How big is the difference? Cohen's d, odds ratios, correlation coefficients — these tell you whether the result matters practically, not just statistically.
Confidence Interval
A range of plausible values for the true effect. A 95% CI of [0.1, 15.2] tells you much more than "p = 0.04" alone — the effect could be tiny or huge.
The choice of which average to report can be just as misleading as cherry-picking a p-value. Statistics is full of decisions that shape the story your data tells.
Next time you read "the results were statistically significant," ask two questions: How big was the effect? And how many people were in the study? If the paper doesn't answer both, the p-value alone isn't telling you much.
It means that if the null hypothesis is true (no real effect), you'd see data this extreme about 5% of the time by random chance alone. It does not mean there's a 5% chance the null hypothesis is true, or a 95% chance your result is correct.
Historical convention, not mathematical necessity. Ronald Fisher suggested it as a reasonable cutoff in the 1920s, and it became entrenched in academic publishing. Different fields use different thresholds — particle physics requires p < 0.0000003 for discovery claims.
A p-value gives you a single yes/no decision point. A confidence interval gives you a range of plausible values for the true effect. A 95% CI that doesn't include zero corresponds to p < 0.05, but the interval also shows you how precise your estimate is and how large the effect might be.
Find z, t, chi-square, and F critical values for hypothesis tests and confidence intervals.
Calculate the arithmetic mean (average) of a set of numbers with step-by-step solutions and learn mean calculation methods.
Calculate percentage increase or decrease between two values with step-by-step explanations and real-world examples.