Basic Theory of Estimation

Point Estimation

An estimator is a sequence $\widehat{theta}= \{ \widehat{theta}_n| n in N\}$ of statistics, where $\widehat{theta}_n= \widehat{theta}_n(X_1,\,...,\,X_n)$ .

An estimator is called

(weakly) consistent if $\widehat{theta}–> theta$ in probability with respect to $P_{theta}$ .
strongly consistent if $\widehat{theta}–> theta$ with probability one.
unbiased if $E_{theta}(\widehat{theta}_n) = theta$ .
asymptotically unbiased if $E_{theta}(\widehat{theta}_n) –> theta$ .
efficient if it is unbiased and has the smallest variance among all unbiased estimators.

For example, $\overlineX_n$ is an unbiased, strongly consistent estimator of the expectation of the underlying distribution, and is an unbiased, strongly consistent estimator of the variance .

There are two well-known methods for calculating an estimator:

Method of moments:
Let us suppose that the expectation is a function of the parameter that has a continuous inverse $f^{-1}$ . So we have $theta = f^{-1}(E_{theta}(X))$ . If we replace $E_{theta}(X)$ by its estimator $\overlineX_n$ , we get $\widehat{theta}_n= f^{-1}(\overlineX_n)$ .
If there is more than one parameter, use the higher moments $E_{theta}(X^k)$ and replace them by $\overline{X^k_n$ .
Likelihood method:
Simply use the value of that maximizes the likelihood-function as an estimator. This estimator is called maximum likelihood estimator.

The Cramér-Rao theorem provides a lower bound for the variance of an unbiased estimator: Let be a random variable with distribution $P_{theta}$ , where $theta in \Theta$ is a real parameter and $\Theta$ is supposed to be an interval. Moreover, the density $f_{theta}(x)$ should be twice differentiable with respect to , and both and should be bounded above uniformly in by an integrable function, that means

$| \frac{\partial\,{}{\partial\,{\t......n \Theta {with} \; int g(x) \, dµ(x) < infinity$

Furthermore, let $\widehat{theta}$ be an unbiased estimator of . Then

${Var}_{theta}(\widehat{theta}) >= \frac1{I(theta)}$

where

$I(theta) = E( ( \ensure......ial^2\,{\log f_{theta}(X)}{\partial\,{theta}^2)$

is the so-called Fisher information.

If we have a sample of size , we interpret this as one -dimensional random variable, and so we get:

${Var}_{theta}(\widehat{theta}) >= \frac1{n * I(theta)}$

If $\widehat{theta}$ is an unbiased estimator and a sufficient statistic, then $\widetilde{theta} = E_{theta}(\widehat{theta}| T)$ is also an unbiased estimator and has a variance which is not greater then the one of $\widehat{theta}$ . This means, that if we look for an efficient estimator, we only need to consider functions of .

Finally, if is an sufficient statistic and has the property

$E_{theta}(f(T)) = 0 forall \, theta in \Theta ==> }f \equiv 0$

then if $\widehat{theta}= g(T)$ is unbiased, this estimator $\widehat{theta}$ is efficient.