Appendix B — Probability

---

---

Configuring R

Functions from these packages will be used throughout this document:

library(conflicted) # check for conflicting function definitions
# library(printr) # inserts help-file output into markdown output
library(rmarkdown) # Convert R Markdown documents into a variety of formats.
library(pander) # format tables for markdown
library(ggplot2) # graphics
library(ggeasy) # help with graphics
library(ggfortify) # help with graphics
library(dplyr) # manipulate data
library(tibble) # `tibble`s extend `data.frame`s
library(magrittr) # `%>%` and other additional piping tools
library(haven) # import Stata files
library(knitr) # format R output for markdown
library(tidyr) # Tools to help to create tidy data
library(plotly) # interactive graphics
library(dobson) # datasets from Dobson and Barnett 2018
library(parameters) # format model output tables for markdown
library(haven) # import Stata files
library(latex2exp) # use LaTeX in R code (for figures and tables)
library(fs) # filesystem path manipulations
library(survival) # survival analysis
library(survminer) # survival analysis graphics
library(KMsurv) # datasets from Klein and Moeschberger
library(parameters) # format model output tables for
library(webshot2) # convert interactive content to static for pdf
library(forcats) # functions for categorical variables ("factors")
library(stringr) # functions for dealing with strings
library(lubridate) # functions for dealing with dates and times

Here are some R settings I use in this document:

rm(list = ls()) # delete any data that's already loaded into R

conflicts_prefer(dplyr::filter)
ggplot2::theme_set(
  ggplot2::theme_bw() + 
        # ggplot2::labs(col = "") +
    ggplot2::theme(
      legend.position = "bottom",
      text = ggplot2::element_text(size = 12, family = "serif")))

knitr::opts_chunk$set(message = FALSE)
options('digits' = 4)

panderOptions("big.mark", ",")
pander::panderOptions("table.emphasize.rownames", FALSE)
pander::panderOptions("table.split.table", Inf)
conflicts_prefer(dplyr::filter) # use the `filter()` function from dplyr() by default
legend_text_size = 9

B.1 Random variables

B.1.1 Binary variables

Definition B.1 (binary variable) A binary variable is a random variable which has only two possible values in its range.

Exercise B.1 (Examples of binary variables) What are some examples of binary variables in the health sciences?

---

Solution. Examples of binary outcomes include:

• exposure (exposed vs unexposed)
• disease (diseased vs healthy)
• recovery (recovered vs unrecovered)
• relapse (relapse vs remission)
• return to hospital (returned vs not)
• vital status (dead vs alive)

---

B.1.2 Count variables

Definition B.2 (Count variable) A count variable is a random variable whose possible values are some subset of the non-negative integers; that is, a random variable \(X\) such that:

\[\mathcal{R}{X} \in \mathbb{N}\]

Exercise B.2 What are some examples of count variables?

---

Solution.

• Number of fish in a pond
• Number of cyclones per season
• Seconds of tooth-brushing per session (if rounded)
• Infections per person-year
• Visits to ER per person-month
• Car accidents per 1000 miles driven

---

Exposure magnitude

Definition B.3 (Exposure magnitude)  

For many count outcomes, there is some sense of exposure magnitude, population size, or duration of observation (Table B.1).

Table B.1: Examples of exposure units

outcome	exposure units
infectious disease incidence	number of individuals exposed, time at risk
car accidents	miles driven
worksite accidents	person-hours worked
population size	size of habitat

Exposure units are similar to the number of trials in a binomial distribution, but in non-binomial count outcomes, there can be more than one event per unit of exposure.

We can use \(t\) to represent continuous-valued exposures/observation durations, and \(n\) to represent discrete-valued exposures.

---

Definition B.4 (Event rate)  

When the concept of an exposure magnitude is meaningful, the mean of the outcome \(Y\) is typically modeled as an event rate (denoted \(\lambda\)) times the exposure magnitude (\(t\)). That is:

\[\mathbb{E}[Y|T=t] \stackrel{\text{def}}{=}\mu = \lambda \cdot t\]

\[\lambda \stackrel{\text{def}}{=}\frac{\mu}{t} \tag{B.1}\]

---

Theorem B.1 When the exposure magnitude is 0, there is no opportunity for events to occur:

\[\mathbb{E}[Y|T=0] = 0\]

---

Proof. \[\mathbb{E}[Y|T=0] = \lambda \cdot 0 = 0\]

---

Probability distributions for count outcomes

• Poisson distribution

• Negative binomial distribution

---

B.2 Key probability distributions

B.2.1 The Bernoulli distribution

Definition B.5 (Bernoulli distribution) The Bernoulli distribution family for a random variable \(X\) is defined as:

\[ \begin{aligned} \Pr(X=x) &= \mathbb{1}_{x\in \left\{0,1\right\}}\pi^x(1-\pi)^{1-x}\\ &= \left\{{\pi, x=1}\atop{1-\pi, x=0}\right. \end{aligned} \]

---

B.2.2 The Poisson distribution

Figure B.1: “Les Poissons”

Siméon Denis Poisson

Les Poissons

---

Definition B.6 (Poisson distribution) \[\mathcal{R}(Y) = \left\{0, 1, 2, ...\right\} = \mathbb{N}\]

\[\text{P}(Y = y) = \frac{\mu^{y} e^{-\mu}}{y!}, y \in \mathbb{N} \tag{B.2}\]

(see Figure B.2)

\[\text{P}(Y \le y) = e^{-\mu} \sum_{j=0}^{\left \lfloor{y}\right \rfloor}\frac{\mu^j}{j!} \tag{B.3}\]

(see Figure B.3)

---

library(dplyr)
pois_dists = tibble(
  mu = c(0.5, 1, 2, 5, 10, 20)) |> 
  reframe(
    .by = mu,
    x = 0:30
  ) |> 
  mutate(
    `P(X = x)` = dpois(x, lambda = mu),
    `P(X <= x)` = ppois(x, lambda = mu),
    mu = factor(mu)
  )

library(ggplot2)
library(latex2exp)

plot0 = pois_dists |> 
  ggplot(
    aes(
      x = x,
      y = `P(X = x)`,
      fill = mu,
      col = mu)) +
  theme(legend.position = "bottom") +
  labs(
    fill = latex2exp::TeX("$\\mu$"),
    col = latex2exp::TeX("$\\mu$"),
    y = latex2exp::TeX("$\\Pr_{\\mu}(X = x)$"))

plot1 = plot0 + 
  geom_col(position = "identity", alpha  = .5) +
  facet_wrap(~mu)
  # geom_point(alpha = 0.75) +
  # geom_line(alpha = 0.75)
print(plot1)

Figure B.2: Poisson PMFs, by mean parameter \(\mu\)

---

library(ggplot2)

plot2 = 
  plot0 + 
  geom_step(alpha = 0.75) +
  aes(y = `P(X <= x)`) + 
  labs(y = latex2exp::TeX("$\\Pr_{\\mu}(X \\leq x)$"))

print(plot2)

Figure B.3: Poisson CDFs

---

Exercise B.3 (Poisson distribution functions) Let \(X \sim \text{Pois}(\mu = 3.75)\).

Compute:

• \(\text{P}(X = 4 | \mu = 3.75)\)
• \(\text{P}(X \le 7 | \mu = 3.75)\)
• \(\text{P}(X > 5 | \mu = 3.75)\)

---

Solution.

• \(\text{P}(X=4) = 0.19378025\)
• \(\text{P}(X\le 7) = 0.96237866\)
• \(\text{P}(X > 5) = 0.17711717\)

---

Theorem B.2 (Properties of the Poisson distribution) If \(X \sim \text{Pois}(\mu)\), then:

• \(\mathbb{E}[Y] = \mu\)
• \(\text{Var}(Y) = \mu\)

Exercise B.4 Prove Theorem B.2.

---

Solution. \[ \begin{aligned} \text{E}[X] &= \sum_{x=0}^\infty x \cdot P(X=x)\\ &= 0 \cdot P(X=0) + \sum_{x=1}^\infty x \cdot P(X=x)\\ &= 0 + \sum_{x=1}^\infty x \cdot P(X=x)\\ &= \sum_{x=1}^\infty x \cdot P(X=x)\\ &= \sum_{x=1}^\infty x \cdot \frac{\lambda^x e^{-\lambda}}{x!}\\ &= \sum_{x=1}^\infty x \cdot \frac{\lambda^x e^{-\lambda}}{x \cdot (x-1)!} & [\text{definition of factorial ("!") function}]\\ &= \sum_{x=1}^\infty \frac{\lambda^x e^{-\lambda}}{ (x-1)!}\\ &= \sum_{x=1}^\infty \frac{(\lambda \cdot \lambda^{x-1}) e^{-\lambda}}{ (x-1)!}\\ &= \lambda \cdot \sum_{x=1}^\infty \frac{( \lambda^{x-1}) e^{-\lambda}}{ (x-1)!}\\ &= \lambda \cdot \sum_{y=0}^\infty \frac{( \lambda^{y}) e^{-\lambda}}{ (y)!} &[\text{substituting } y \stackrel{\text{def}}{=}x-1]\\ &= \lambda \cdot 1 &[\text{because PDFs sum to 1}]\\ &= \lambda\\ \end{aligned} \]

---

B.2.3 Accounting for exposure

If the exposures/observation durations, denoted \(T=t\) or \(N=n\), vary between observations, we model:

\[\mu = \lambda\cdot t\]

\(\lambda\) is interpreted as the “expected event rate per unit of exposure”; that is,

\[\lambda = \frac{\mathbb{E}[Y|T=t]}{t}\]

Important

The exposure magnitude, \(T\), is similar to a covariate in linear or logistic regression. However, there is an important difference: in count regression, there is no intercept corresponding to \(\mathbb{E}[Y|T=0]\). In other words, this model assumes that if there is no exposure, there can’t be any events.

Theorem B.3 If \(\mu = \lambda\cdot t\), then:

\[\text{log}\left\{\mu \right\}= \text{log}\left\{\lambda\right\} + \text{log}\left\{t\right\}\]

Definition B.7 (Offset) When the linear component of a model involves a term without an unknown coefficient, that term is called an offset.

---

B.2.4 The Negative-Binomial distribution

{{}}

B.3 Characteristics of probability distributions

Definition B.8 (Density function) The density function \(f(t)\) or \(\text{p}(T=t)\) for a random variable \(T\) at value \(t\) can be defined as the derivative of the cumulative probability function \(P(T\le t)\); that is:

\[f(t) \stackrel{\text{def}}{=}\frac{\partial}{\partial t} \Pr(T\le t)\]

Definition B.9 (Hazard function) The hazard function for a random variable \(T\) at value \(t\) is the conditional density of \(T\) at \(t\), given \(T\ge t\); that is:

\[h(t) \stackrel{\text{def}}{=}p(T=t|T\ge t)\]

If \(T\) represents the time at which an event occurs, then \(h(t)\) is the probability that the event occurs at time \(t\), given that it has not occurred prior to time \(t\).

Definition B.10 (Expectation, expected value, population mean ) The expectation, expected value, or population mean of a continuous random variable \(X\), denoted \(\mathbb{E}\left[X\right]\), \(\mu(X)\), or \(\mu_X\), is the weighted mean of \(X\)’s possible values, weighted by the probability density function of those values:

\[\mathbb{E}\left[X\right] = \int_{x\in \mathcal{R}(X)} x \cdot \text{p}(X=x)dx\]

The expectation, expected value, or population mean of a discrete random variable \(X\), denoted \(\mathbb{E}\left[X\right]\), \(\mu(X)\), or \(\mu_X\), is the mean of \(X\)’s possible values, weighted by the probability mass function of those values:

\[\mathbb{E}\left[X\right] = \sum_{x \in \mathcal{R}(X)} x \cdot \text{P}(X=x)\]

(c.f. https://en.wikipedia.org/wiki/Expected_value)

---

Theorem B.4 (Expectation of the Bernoulli distribution) The expectation of a Bernoulli random variable with parameter \(\pi\) is:

\[\mathbb{E}\left[X\right] = \pi\]

---

Proof. \[ \begin{aligned} \mathbb{E}\left[X\right] &= \sum_{x\in \mathcal{R}(X)} x \cdot\text{P}(X=x) \\&= \sum_{x\in \left\{0,1\right\}} x \cdot\text{P}(X=x) \\&= \left(0 \cdot\text{P}(X=0)\right) + \left(1 \cdot\text{P}(X=1)\right) \\&= \left(0 \cdot(1-\pi)\right) + \left(1 \cdot\pi\right) \\&= 0 + \pi \\&= \pi \end{aligned} \]

---

B.3.1 Variance and related characteristics

Definition B.11 (Variance) The variance of a random variable \(X\) is the expectation of the squared difference between \(X\) and \(\mathbb{E}\left[X\right]\); that is:

\[ \text{Var}\left(X\right) \stackrel{\text{def}}{=}\mathbb{E}\left[(X-\mathbb{E}\left[X\right])^2\right] \]

---

Theorem B.5 (Simplified expression for variance) \[\text{Var}\left(X\right)=\mathbb{E}\left[X^2\right] - \left(\mathbb{E}\left[X\right]\right)^2\]

---

Proof. By linearity of expectation, we have:

\[ \begin{aligned} \text{Var}\left(X\right) &\stackrel{\text{def}}{=}\mathbb{E}\left[(X-\mathbb{E}\left[X\right])^2\right]\\ &=\mathbb{E}\left[X^2 - 2X\mathbb{E}\left[X\right] + \left(\mathbb{E}\left[X\right]\right)^2\right]\\ &=\mathbb{E}\left[X^2\right] - \mathbb{E}\left[2X\mathbb{E}\left[X\right]\right] + \mathbb{E}\left[\left(\mathbb{E}\left[X\right]\right)^2\right]\\ &=\mathbb{E}\left[X^2\right] - 2\mathbb{E}\left[X\right]\mathbb{E}\left[X\right] + \left(\mathbb{E}\left[X\right]\right)^2\\ &=\mathbb{E}\left[X^2\right] - \left(\mathbb{E}\left[X\right]\right)^2\\ \end{aligned} \]

---

Definition B.12 (Precision) The precision of a random variable \(X\), often denoted \(\tau(X)\), \(\tau_X\), or shorthanded as \(\tau\), is the inverse of that random variable’s variance; that is:

\[\tau(X) \stackrel{\text{def}}{=}\left(\text{Var}\left(X\right)\right)^{-1}\]

Definition B.13 (Standard deviation) The standard deviation of a random variable \(X\) is the square-root of the variance of \(X\):

\[\text{SD}\left(X\right) \stackrel{\text{def}}{=}\sqrt{\text{Var}\left(X\right)}\]

---

Definition B.14 (Covariance) For any two one-dimensional random variables, \(X,Y\):

\[\text{Cov}\left(X,Y\right) \stackrel{\text{def}}{=}\mathbb{E}\left[(X - \mathbb{E}\left[X\right])(Y - \mathbb{E}\left[Y\right])\right]\]

---

Theorem B.6 \[\text{Cov}\left(X,Y\right)= \mathbb{E}\left[XY\right] - \mathbb{E}\left[X\right] \mathbb{E}\left[Y\right]\]

---

Proof. Left to the reader.

---

Lemma B.1 (The covariance of a variable with itself is its variance) For any random variable \(X\):

\[\text{Cov}\left(X,X\right) = \text{Var}\left(X\right)\]

Proof. \[ \begin{aligned} \text{Cov}\left(X,X\right) &= E[XX] - E[X]E[X] \\ &= E[X^2]-(E[X])^2 \\ &= \text{Var}\left(X\right) \end{aligned} \]

---

Definition B.15 (Variance/covariance of a \(p \times 1\) random vector) For a \(p \times 1\) dimensional random vector \(X\),

\[ \begin{aligned} \text{Var}(X) &\stackrel{\text{def}}{=}\text{Cov}(X)\\ &\stackrel{\text{def}}{=}E[ \left( X - E\lbrack X\rbrack \right)^{\top}\left( X - E\lbrack X\rbrack \right) ]\\ \end{aligned} \]

---

Theorem B.7 (Alternate expression for variance of a random vector) \[ \begin{aligned} \text{Var}\left(X\right) &= E[ X^{\top}X ] - {E\lbrack X\rbrack}^{\top}E\lbrack X\rbrack \end{aligned} \]

---

Proof. \[ \begin{aligned} \text{Var}\left(X\right) &= E[ \left( X^{\top} - E\lbrack X\rbrack^{\top} \right)\left( X - E\lbrack X\rbrack \right) ]\\ &= E[ X^{\top}X - E\lbrack X\rbrack^{\top}X - X^{\top}E\lbrack X\rbrack + E\lbrack X\rbrack^{\top}E\lbrack X\rbrack ]\\ &= E[ X^{\top}X ] - E\lbrack X\rbrack^{\top}E\lbrack X\rbrack - {E\lbrack X\rbrack}^{\top}E\lbrack X\rbrack + E\lbrack X\rbrack^{\top}E\lbrack X\rbrack\\ &= E[ X^{\top}X ] - 2{E\lbrack X\rbrack}^{\top}E\lbrack X\rbrack + E\lbrack X\rbrack^{\top}E\lbrack X\rbrack\\ &= E[ X^{\top}X ] - {E\lbrack X\rbrack}^{\top}E\lbrack X\rbrack \end{aligned} \]

---

Theorem B.8 (Variance of a linear combination) For any set of random variables \(X_1, \ldots, X_n\) and corresponding constants \(a_1, ... ,a_n\):

\[\text{Var}\left(\sum_{i=1}^na_i X_i\right) = \sum_{i=1}^n\sum_{j=1}^n a_i a_j \text{Cov}\left(X_i,X_j\right)\]

---

Proof. Left to the reader…

---

Lemma B.2 For any two random variables \(X\) and \(Y\) and scalars \(a\) and \(b\):

\[\text{Var}\left(aX + bY\right) = a^2 \text{Var}\left(X\right) + b^2 \text{Var}\left(Y\right) + 2(a \cdot b) \text{Cov}\left(X,Y\right)\]

---

Proof. Apply Theorem B.8 with \(n=2\), \(X_1 = X\), and \(X_2 = Y\).

Or, see https://statproofbook.github.io/P/var-lincomb.html

---

Definition B.16 (homoskedastic, heteroskedastic) A random variable \(Y\) is homoskedastic (with respect to covariates \(X\)) if the variance of \(Y\) does not vary with \(X\):

\[\text{Var}(Y|X=x) = \sigma^2, \forall x\]

Otherwise it is heteroskedastic.

---

Definition B.17 (Statistical independence) A set of random variables \(X_1, \ldots, X_n\) are statistically independent if their joint probability is equal to the product of their marginal probabilities:

\[\Pr(X_1=x_1, \ldots, X_n = x_n) = \prod_{i=1}^n{\Pr(X_i=x_i)}\]

Tip

The symbol for independence, \(⫫\), is essentially just \(\prod\) upside-down. So the symbol can remind you of its definition (Definition B.17).

---

Definition B.18 (Conditional independence) A set of random variables \(Y_1, \ldots, Y_n\) are conditionally statistically independent given a set of covariates \(X_1, \ldots, X_n\) if the joint probability of the \(Y_i\)s given the \(X_i\)s is equal to the product of their marginal probabilities:

\[\Pr(Y_1=y_1, \ldots, Y_n = y_n|X_1=x_1, \ldots, X_n = x_n) = \prod_{i=1}^n{\Pr(Y_i=y_i|X_i=x_i)}\]

---

Definition B.19 (Identically distributed) A set of random variables \(X_1, \ldots, X_n\) are identically distributed if they have the same range \(\mathcal{R}(X)\) and if their marginal distributions \(\text{P}(X_1=x_1), ..., \text{P}(X_n=x_n)\) are all equal to some shared distribution \(\text{P}(X=x)\):

\[ \forall i\in \left\{1:n\right\}, \forall x \in \mathcal{R}(X): \text{P}(X_i=x) = \text{P}(X=x) \]

---

Definition B.20 (Conditionally identically distributed) A set of random variables \(Y_1, \ldots, Y_n\) are conditionally identically distributed given a set of covariates \(X_1, \ldots, X_n\) if \(Y_1, \ldots, Y_n\) have the same range \(\mathcal{R}(X)\) and if the distributions \(\text{P}(Y_i=y_i|X_i =x_i)\) are all equal to the same distribution \(\text{P}(Y=y|X=x)\):

\[ \text{P}(Y_i=y|X_i=x) = \text{P}(Y=y|X=x) \]

---

Definition B.21 (Independent and identically distributed) A set of random variables \(X_1, \ldots, X_n\) are independent and identically distributed (shorthand: “\(X_i\ \text{iid}\)”) if they are statistically independent and identically distributed.

---

Definition B.22 (Conditionally independent and identically distributed) A set of random variables \(Y_1, \ldots, Y_n\) are conditionally independent and identically distributed (shorthand: “\(Y_i | X_i\ \text{ciid}\)” or just “\(Y_i |X_i\ \text{iid}\)”) given a set of covariates \(X_1, \ldots, X_n\) if \(Y_1, \ldots, Y_n\) are conditionally independent given \(X_1, \ldots, X_n\) and \(Y_1, \ldots, Y_n\) are identically distributed given \(X_1, \ldots, X_n\).