A hypothesis is any subset of the set of all possible probability distributions. In a parametric model, one speaks of a parametric hypothesis.
If the hypothesis contains only one distribution, it is called a simple hypothesis, otherwise it is a composite hypothesis.
In case of parametric hypotheses, one can distinguish one-sided (e.g. or ) and two-sided hypotheses (e.g. ).
Based on a sample, we decide in favour of a so-called null hypothesis or against it. This can be described by a (non-randomized) test which is a function from to . If , we accept , otherwise we reject it.
A randomized test is a function where is the probability that we reject .
An error of the first kind occurs, if we reject although it is true, whereas an error of the second kind occurs, if we accept even though it is wrong. A test is said to have level of significance , if the probability of a first kind error is not greater than , which means
A best test of level is a test of level with the smallest probability of a second kind error.