AIC

Reasons for AIC:

• The maximum (log-)likelihood method can be used to estimate the values of paramters. However, it cannot be used to compare different models without some corrections.
• Using $N^{-1} \ell(\hat{\theta})$ as an estimate of $\text{E}_Y \log f(Y\vert\hat{\theta})$ produces a bias. The reason for such bias to occur is that the same data are used to estimate parameters and to calculate the log-likelihood.
• An unbiased estimate of $\text{E}_Y \log f(Y)$ yields the famous Akaike Information Criterion (AIC).

The expectation value of bias is

$$\text{AIC} = -2 \ell(\hat{\theta}) + 2K$$ (26)

Exercise: AIC of the least squared fit for a regression model. If $y_n = \sum_{i=1}^{m} a_i x_{ni} + \varepsilon_n$ and $\varepsilon_n \sim N(0, \sigma^2)$, find the corresponding AIC number.

Ans. $\text{AIC}_K = N(\log 2\pi \hat{\sigma}_K^2 + 1) + 2(K + 1)$.

Example.Figure 5 shows the normalized asymptotic level changes for the sudden impulses seen at mid-latitude ground magnetometer stations. The trend component (solid line) is estimated based on the hyper-parameter automatically determined in accordance with the AIC minimization procedure [Higuchi, 1991].
  

Figure 5. The normalized aymptotic change in the H component for the sudden impulse events observed by ground stations as a function of local time. The solid line represents the best fit to the data (From Russell and Ginskey, [1995]).

The AIC method also suggests the direction toward selecting the best model by computer calculation.