Testing

A hypothesis is any subset of the set of all possible probability distributions. In a parametric model, one speaks of a parametric hypothesis.

If the hypothesis contains only one distribution, it is called a simple hypothesis, otherwise it is a composite hypothesis.

In case of parametric hypotheses, one can distinguish one-sided (e.g. $theta<theta_{0}$ or $theta>theta_{0}$ ) and two-sided hypotheses (e.g. $theta<>theta_{0}$ ).

Based on a sample, we decide in favour of a so-called null hypothesis or against it. This can be described by a (non-randomized) test which is a function from to $\{0,1\}$ . If $phi(X_1,\,...,\,X_n)=0$ , we accept , otherwise we reject it.
A randomized test is a function where $phi(X_1,\,...,\,X_n)$ is the probability that we reject .

An error of the first kind occurs, if we reject although it is true, whereas an error of the second kind occurs, if we accept even though it is wrong. A test is said to have level of significance , if the probability of a first kind error is not greater than , which means

$E_{theta}(phi) <= alpha forall \, phiin \H$

A best test of level is a test of level with the smallest probability of a second kind error.