Jackknife

One of the earliest techniques to obtain reliable statistical estimators is the jackknife technique. It requires less computational power than more recent techniques.

Suppose we have a sample $\mathbf{x} = (x_1, x_2, \ldots x_n)$ and an estimator $\hat{\theta} = s(\mathbf{x})$. The jackknife focuses on the samples that leaves out one observation at a time:

$$\mathbf{x}_{(i)} = (x_1, x_2, \ldots x_{i-1}, x_{i+1}, \ldots x_n)$$

for $i = 1, 2, \ldots n$, called jackknife samples. The ith jackknife sample consists of the data set with the ith observation removed. Let $\hat{\theta}_{(i)} = s(\mathbf{x}_{(i)})$ be the ith jackknife replication of $\hat{\theta}$.

The jackknife estimate of s.e. defined by  

$$\widehat{se}_{\text{jack}} = [ \frac{n-1}{n} \sum (\hat{\theta}_{(i)} - \hat{\theta}_{(\cdot)})^2 ]^{1/2}$$ (3)

where $\hat{\theta}_{(\cdot)} = \sum_{i=1}^{n} \hat{\theta}_{(i)} / n$.

The jackknife only works well for linear statistics (e.g., mean). It fails to give accurate estimation for non-smooth (e.g., median) and nonlinear (e.g., correlation coefficient) cases. Thus improvements to this technique were developed.