Appendix C — Estimation
C.1 Probabilistic models
Definition C.1 (Scientific models) Scientific models are attempts to describe physical conditions or changes that occur in the world and universe around us.
Example C.1 (Scientific models in epidemiology) Epidemiologists typically study biological conditions and changes, such as the spread of infectious diseases through populations, or the effects of environmental factors on individuals.
C.1.1 Statistical analysis of scientific models
When we perform statistical analyses, we use data to help us choose between models - specifically, to determine which models best explain that data.
However, physical processes do not produce data on their own. Data is only produced when scientists implement an observation process (i.e., a scientific study), which is distinct from the underlying physical process. In some cases, the observation process and the physical process interact with each other. This phenomenon is called the “observer effect”.
In order to learn about the physical processes we are ultimately interested in, we often need to make special considerations for the observation process that produced the data which we are analyzing. In particular, if some of the planned observations in the study design were not completed, we will likely need to account for the incompleteness of the resulting data set in our analysis. If we are not sure why some observations are incomplete, we may need to model the observation process in addition to the physical process we were originally interested in. For example, if some participants in a study dropped out part-way through the study, we may need investigate why those participants dropped out, as opposed to other participants who completed the study.
These kinds of missing data issues are outside of the scope of this course; see Van Buuren (2018) for more details.
C.2 Estimands, estimates, and estimators
C.2.1 Estimands
Definition C.2 (Estimand) An estimand is an unknown quantity whose value we want to know (Pohl et al. 2021; Lawrance et al. 2020).
Example C.2 (Mean height of students) If we are trying to determine the mean height of students at our school, then the population mean is our estimand.
In statistical contexts, most estimands are parameters of probabilistic models, or functions of model parameters.
Model paramaters and other estimands are often symbolized using lower-case Greek letters: \(\alpha, \beta, \gamma, \delta\), etc.
C.2.2 Estimates
Definition C.3 (Estimate/estimated value) In statistics, an estimate or estimated value is an informed guess of an estimand’s value, based on observed data.
Example C.3 (Mean height of students) Suppose we measure the heights of 50 random students from our school, and the sample mean was 175cm. We might use 175cm as an estimate of the population mean.
C.2.3 Estimators
Definition C.4 (Estimator) An estimator is a function \(\hat\theta(x_1,...x_n)\) that transforms data \(x_1,...x_n\) into an estimate.
When estimators are applied to random variables, the estimators are also random variables.
Estimators are often symbolized by placing a ^ (“hat”) symbol on top of the corresponding estimand; for example, \(\hat\theta\).
Usually, their dependence on the data is implicit:
\[\hat\theta\stackrel{\text{def}}{=}\hat\theta(x_1,...x_n)\]
Example C.4 (Mean height of students) If we want to estimate the mean height of students at our university, which we will represent as \(\mu\), we might measure the heights of \(n= 50\) randomly sampled students as random variables \(X_1,...,X_n\). Then we could use the function
\[\hat\mu(X_1,...,X_n) = \frac{1}{n} \sum_{i=1}^n X_i \stackrel{\text{def}}{=}\bar X\]
as an estimator to produce an estimate \(\hat\mu = \bar x\) of \(\mu\).
Another estimator would be just the height of the first student sampled:
\[\hat\mu^{(2)}(X_1,...,X_n) = X_1\]
A third possible estimator would be the mean of all sampled students’ heights, except for the two most extreme; that is, if we re-order the observations \(X_{(1)} = \min_{i\in 1:n} X_i\), \(X_{(2)} = \min_{i\in \{1:n\} - \arg X_{(1)}} X_i\), …, \(X_{(n)} = \max_{i\in 1:n} X_i\), then we could define the estimator:
\[\hat\mu^{(3)}(X_1,...,X_n) = \frac{1}{n}\sum_{i=2}^{n-1} X_{(i)}\]
Which of these estimators is best? It depends on how we evaluate them (see Section C.3 below).
C.2.4 Contrasting estimands, estimates, and estimators
It’s helpful to keep in mind the mathematical type of each estimation concept:
- estimands are numbers (or vector of numbers)
- estimates are also numbers (or vectors)
- estimators are functions of random variables, so they are also random variables
C.3 Accuracy of estimators
C.3.1 Accuracy
To determine which estimator is best, we need to define best. “Accuracy” is usually most important; easy computation is usually secondary.
Definition C.5 (Accuracy) The accuracy of an estimator for a given estimand does not have a consensus formal definition, but all of the usual candidates are related to the distributions of the errors made by the resulting estimates.
C.3.2 Error
Definition C.6 (Error) The error of an estimate \(\hat\theta\) of a true value \(\theta\), often denoted \(\varepsilon(\hat\theta)\), or more completely \(\varepsilon(\hat\theta, \theta)\), is the difference between the estimate and its estimand \(\theta\); that is:
\[\varepsilon(\hat\theta) \stackrel{\text{def}}{=}\hat\theta- \theta\]
Some frequently-used measures of accuracy include:
C.3.3 Mean squared error
Definition C.7 (Mean squared error) The mean squared error of an estimator \(\hat\theta\), denoted \(\text{MSE}\left(\hat\theta\right)\), is the expectation of the square of the error1:
\[\text{MSE}\left(\hat\theta\right) \stackrel{\text{def}}{=}\mathbb{E}\left[(\varepsilon(\hat\theta))^2\right]\]
C.3.4 Mean absolute error
Definition C.8 (Mean absolute error) The mean absolute error of an estimator is the expectation of the absolute value of the error:
\[ \text{MAE}\left(\hat\theta\right) \stackrel{\text{def}}{=}\mathbb{E}\left[\left|\varepsilon(\hat\theta)\right|\right] \]
C.3.5 Bias
Definition C.9 (Bias) The bias of an estimator \(\hat\theta\) for an estimand \(\theta\) is the expected value of the error:
\[\text{Bias}\left(\hat\theta\right) \stackrel{\text{def}}{=}\mathbb{E}\left[\varepsilon(\hat\theta)\right] \tag{C.1}\]
Theorem C.1 (Bias equals Expectation minus Truth) \[\text{Bias}\left(\hat\theta\right) =\mathbb{E}\left[\hat\theta\right] - \theta\]
Proof. \[ \begin{aligned} \text{Bias}\left(\hat\theta\right) &\stackrel{\text{def}}{=}\mathbb{E}\left[\varepsilon(\hat\theta)\right]\\ &= \mathbb{E}\left[\hat\theta- \theta\right]\\ &=\mathbb{E}\left[\hat\theta\right] - \mathbb{E}\left[\theta\right]\\ &=\mathbb{E}\left[\hat\theta\right] - \theta \end{aligned} \]
The third equality is by the linearity of expectation.
Theorem C.2 (Mean Squared Error equals Bias Squared plus Variance) For any one-dimensional estimator \(\hat\theta\):
\[\text{MSE}\left(\hat\theta\right) = \left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right) \tag{C.2}\]
Proof. Let’s start by expanding each term of the right-hand side:
\[ \begin{aligned} \left(\text{Bias}\left(\hat\theta\right)\right)^2 &=\left(\mathbb{E}\left[\hat\theta\right] - \theta\right)^2\\ &=\left(\mathbb{E}\left[\hat\theta\right]\right)^2 - 2\mathbb{E}\left[\hat\theta\right]\theta+\theta^2\\ \end{aligned} \]
\[\text{Var}\left(\hat\theta\right) = \mathbb{E}\left[\hat\theta^2\right] - \left(\mathbb{E}\left[\hat\theta\right]\right)^2\\\]
Now, add them together and simplify:
\[ \begin{aligned} \left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right) &=\left(\mathbb{E}\left[\hat\theta\right]\right)^2 - 2\mathbb{E}\left[\hat\theta\right]\theta+\theta^2 + \mathbb{E}\left[\hat\theta^2\right] - \left(\mathbb{E}\left[\hat\theta\right]\right)^2\\ &=\mathbb{E}\left[\hat\theta^2\right] - 2\mathbb{E}\left[\hat\theta\right]\theta+\theta^2\\ \end{aligned} \]
Now let’s expand the left-hand side to reach the same expression:
\[ \begin{aligned} \text{MSE}\left(\hat\theta\right) &= \mathbb{E}\left[(\text{e}^{(}\hat\theta))^2\right]\\ &= \mathbb{E}\left[(\hat\theta- \theta)^2\right]\\ &= \mathbb{E}\left[\hat\theta^2 - 2\hat\theta\theta- \theta^2\right]\\ &=\mathbb{E}\left[\hat\theta^2\right] - \mathbb{E}\left[2\hat\theta\theta\right]+\mathbb{E}\left[\theta^2\right]\\ &=\mathbb{E}\left[\hat\theta^2\right] - 2\mathbb{E}\left[\hat\theta\right]\theta+\theta^2\\ \end{aligned} \]
\(\text{MSE}\left(\hat\theta\right)\) and \(\left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right)\) both equal \(\mathbb{E}\left[\hat\theta^2\right] - 2\mathbb{E}\left[\hat\theta\right]\theta+\theta^2\). Equality is transitive, so \(\text{MSE}\left(\hat\theta\right)\) and \(\left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right)\) are equal to each other:
\[\text{MSE}\left(\hat\theta\right) = \left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right)\]
Unbiased estimators
Definition C.10 (unbiased estimator) An estimator \(\hat\theta\) is unbiased if \(\text{Bias}\left(\hat\theta\right) = 0\).
Theorem C.3 (properties of unbiased estimators) If \(\hat\theta\) is unbiased, then:
\[\mathbb{E}\left[\hat\theta\right] = \theta \tag{C.3}\] \[\text{MSE}\left(\hat\theta\right) = \text{Var}\left(\hat\theta\right) \tag{C.4}\]
Proof. If \(\hat\theta\) is unbiased, then:
\[ \begin{aligned} \text{Bias}\left(\hat\theta\right) &= 0\\ \mathbb{E}\left[\hat\theta\right] - \theta &= 0\\ \mathbb{E}\left[\hat\theta\right] &= \theta \end{aligned} \]
\[ \begin{aligned} \text{MSE}\left(\hat\theta\right) &\stackrel{\text{def}}{=}\mathbb{E}\left[\left(\varepsilon(\hat\theta)\right)^2\right]\\ &= \mathbb{E}\left[\left(\hat\theta- \theta\right)^2\right]\\ &= \mathbb{E}\left[\left(\hat\theta- \mathbb{E}\left[\hat\theta\right]\right)^2\right]\\ &\stackrel{\text{def}}{=}\text{Var}\left(\hat\theta\right) \end{aligned} \]
(Alternative proof of Equation C.4) We could have started from Theorem C.2 instead:
\[ \begin{aligned} \text{MSE}\left(\hat\theta\right) &= \left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right)\\ &= \left(0\right)^2 + \text{Var}\left(\hat\theta\right)\\ &= 0 + \text{Var}\left(\hat\theta\right)\\ &= \text{Var}\left(\hat\theta\right)\\ \end{aligned} \]
C.3.6 Standard error
Definition C.11 (Standard error) The standard error of an estimator \(\hat\theta\) is just the standard deviation of \(\hat\theta\); that is:
\[\text{SE}\left(\hat\theta\right) \stackrel{\text{def}}{=}\text{SD}\left(\hat\theta\right)\]
“Standard error” is a confusing concept in a few ways. First of all, it isn’t even defined as a characteristic of the error, \(\varepsilon(\hat\theta)\)! Moreover, it is just a synonym for standard deviation, so it seems like a redundant concept. However, standard errors help us construct p-values and confidence intervals, so they come up a lot - often enough to give them their own name.
We can relate standard error to actual error, by rearranging the result from Theorem C.2:
\[ \begin{aligned} \text{Var}\left(\hat\theta\right) &= \text{Var}\left(\hat\theta- \theta\right)\\ &= \text{Var}\left(\varepsilon(\hat\theta)\right)\\ \end{aligned} \] So the variance of the estimator is equal to the variance of the error, and the standard error is equal to the standard deviation of the error:
\[\text{SE}\left(\hat\theta\right) = \text{SD}\left(\varepsilon(\hat\theta)\right)\]
Corollary C.1 (Standard error squared equals MSE minus squared bias) standard error is what is left over of MSE after bias is removed:
\[\left(\text{SE}\left(\hat\theta\right)\right)^2 = \text{MSE}\left(\hat\theta\right) - \left(\text{Bias}\left(\hat\theta\right)\right)^2\]
Proof. \[ \begin{aligned} \text{MSE}\left(\hat\theta\right) &= \left(\text{Bias}\left(\hat\theta\right)\right)^2 + \text{Var}\left(\hat\theta\right)\\ \therefore\text{Var}\left(\hat\theta\right) &= \text{MSE}\left(\hat\theta\right) - \left(\text{Bias}\left(\hat\theta\right)\right)^2\\ \therefore\left(\text{SE}\left(\hat\theta\right)\right)^2 &= \text{MSE}\left(\hat\theta\right) - \left(\text{Bias}\left(\hat\theta\right)\right)^2\\ \end{aligned} \]
Corollary C.2 (For unbiased estimators, SE = RMSE) If \(\mathbb{E}\left[\varepsilon\left(\hat\theta\right)\right] = 0\), then:
\[\text{SE}\left(\hat\theta\right) = \sqrt{\text{MSE}\left(\hat\theta\right)}\]
(this result is equivalent to Equation C.4)