Limited Dependent Variables

So far, we have mostly focused on continuous and unconstrained dependent variables, i.e. \(Y \in \mathbb{R}\). However, many interesting variables are limited in some form.

• Probabilities range from 0 to 1
  
• GDP is a positive variable
  
• Political orientation is a categorical variable

Until now, we have treated these variables as approximately continuous, but this may cause severe issues.

A solution is to use specialised limited dependent variable (LDV) models.

Example of LDVs

We can distinguish between classification and regression.

Classification: we speak of a classification model when the outcome is

• Binary: pass vs. fail, good vs. bad, sick vs. not, employed vs. unemployed
  
• Categorical: nationality, dog breeds, means of transport; or ordinal: good – okay – bad

Regression: we speak of a regression model when the outcome is

• Censored, truncated, or positive: wages, wealth, time
  
• Count data: number of votes, days since an accident

Regression is generally used in a broader sense and may encompass classification.

Linear Probability Model

The Linear Probability Model is an OLS regression applied to a binary dependent variable \(\boldsymbol{y} \in \{0,1\}\) as in:

\[ Y = \begin{cases} 1 & \text{with probability } p,\\[6pt] 0 & \text{with probability } 1 - p. \end{cases} \]

The model: \[ \boldsymbol{y} = \boldsymbol{X}\beta + \boldsymbol{u} \]

The expected value of the dependent variable is equal to the probability \(p\) that \(\boldsymbol{y} = 1\).

Conditional on the regressor \(\boldsymbol{X}\), we have: \[ \mathbb{E}[y \mid \boldsymbol{X}] = \mathbb{P}(y = 1 \mid \boldsymbol{X}) = \boldsymbol{X}\beta . \]

Understanding the LPM

The LPM implies that \(\beta_j\) gives us the expected absolute change in probability when \(\boldsymbol{x}_j\) increases by 1. This linearity assumption can be a major limitation, as probabilities are naturally nonlinear.

\[ \mathbb{P}(y \mid \boldsymbol{X}) = \beta_0 + \boldsymbol{x}_1\beta_1 + \dots + \boldsymbol{x}_k\beta_k \]

This linearity assumption can be a major limitation because:

• probabilities are naturally nonlinear
• constant marginal effects are unrealistic
  
• the linear prediction can exceed 0 or 1

Problems with the LPM

Modeling Probabilities

• When dealing with probabilities, the linearity assumption for \(f\) may be too strong. We need another approach.

• We consider a function \(G\) that satisfies \(0 < G(z) < 1\).

• We can use \(G\) to adapt our model to \[ P(\boldsymbol{y} \mid \boldsymbol{X}) = G(\boldsymbol{X}\beta). \]

• In this way, we model a latent variable \(\boldsymbol{z} = \boldsymbol{X}\beta\) using a linear model and link it to the dependent variable \(\boldsymbol{y}\) via the non-linear function \(G\), giving us \(\boldsymbol{y} = G(\boldsymbol{z})\).

• The inverse function \(G^{-1}(z)\) is called the link function.

• We can use different functional forms for \(G\).

Cumulative Distribution Function

The probability function \(G(\boldsymbol{X}\beta)\) comes directly from the CDF of the error term in the latent variable model.

Start from \(\boldsymbol{y}^* = \boldsymbol{X}\beta + u\), where:

• \(\boldsymbol{y}^*\) is an unobserved continuous propensity
  
• \(u\) is a random error term

We only observe: \[ y = \begin{cases} 1 & \text{if } y^* > 0,\\[6pt] 0 & \text{otherwise}. \end{cases} \]

It follows that the expected value of \(y\) depends on the distribution of \(u\):

\[ \mathbb{P}(\boldsymbol{y} = 1 \mid \boldsymbol{X}) = \mathbb{P}(\boldsymbol{y}^* >0 | \boldsymbol{X}) = \mathbb{P}(u > -\boldsymbol{X} \beta)= \mathbb{P}(u < \boldsymbol{X} \beta) = F_u(\boldsymbol{X})\beta) \]

The Logit Model

For the logit model, we use the cumulative distribution function (CDF) of a logistic variable — the logistic function — for \(G\).

The link function is the log-odds: \(\log \frac{p}{1-p}\).

\[ G(\boldsymbol{z}=\frac{e^z}{e^z+1}) \]

The Probit Model

For the probit model, we use the CDF of a standard normal distribution, which gives us the probability that the standard normal variable \(Z\) is smaller than \(z\).

\[ G(z) = \Phi(z) = \mathbb{P}(Z \le z), \quad \text{where } Z \sim \mathcal{N}(0,1). \]

Interpretation

The interpretation of non-linear models such as logit and probit is not as straightforward as in linear models due to their non‐linearity.We can intrepret:

• sign of the coefficients, the direction of the expected change
• significance of coefficients

If \(\beta_j >0\) we expect the probability to increase with \(\boldsymbol{x_j}\).

However, we cannot interpret the magnitude of coefficients as magnitude of the effect of \(\boldsymbol{X}\) on \(\boldsymbol{y}\). Instead it captures the effects of \(\boldsymbol{X}\) on the latent \(\boldsymbol{z}\), which we rarely care about.

Partial effects

The problem with interpreting coefficients is that the partial effects of \(x_j\) are affected by all
other variables. Assume \(x_1\) is a dummy. Then:

\[ \mathbb{P}(y \mid x_1 = 1, x_2, \ldots ) = G(\beta_0 + \beta_1 + x_2\beta_2 + \ldots) \]

\[ \mathbb{P}(y \mid x_1 = 0, x_2, \ldots ) = G(\beta_0 + x_2\beta_2 + \ldots) \]

The change depends on the level of \(x_2\) and other variables.

The same holds for continuous variables, where the partial effect is given by:

\[ \frac{\partial \, \mathbb{P}(y \mid x_j = x_{j,\cdot})}{\partial x_j} = G(z) \]

where \(g(z) = G'(z)\), i.e., the first derivative of the link function.

Partial effects (2)

We can use summary measures to help interpret partial effects in non-linear models.

• Partial effect at the average: the partial effect at the average evaluates the effect when the explanatory variables are at their mean.

\[ g(\,\bar{X}\,\hat{\beta}\,)\,\hat{\beta}_j \]

• Average partial effect: we calculate the partial effect for each observation and then take the average.

\[ \frac{1}{N} \sum_{j=1}^{N} G(X\hat{\beta}) \, \hat{\beta}_j \]

Testing

To test the significance of single coefficients, we can use \(t\) values.

For multiple coefficients we can use the likelihood ratio test

\[ LR = 2(\text{log } \mathcal{L}_u - \text{log } \mathcal{L}_r) \]

We compare the likelihood of the unrestricted (\(\mathcal{L}_u\)) and restricted (\(\mathcal{L}_r\)) models, where the models are required to be nested (the complex model nests the simpler one).

Likelihood: The likelihood function is the joint probability of the observed data, viewed as a function of the parameters.

Comparing Models

We can compare model specifications using

• \(R^2\), the proportion of explained variance, for non-linear models there are various pseudo \(R^2\) measures,
• the likelihood, \(\mathcal{L}\), of a given model,
• information criteria (IC), which rewards model fit but penalizes model complexity

Many measures of model fit always increase with complexity — IC prefer parsimony.

Akaike information criterion \[ \text{AIC} = 2K - 2 \log \hat{\mathcal{L}} \]

Bayesian (or Schwarz) information criterion: \[ \text{BIC} = K \log N - 2 \log \hat{\mathcal{L}} \]

Other Probability Models

Probabilities are not the only limited dependent variables, and there is a range of other specialised models. This includes the:

• Poisson model for count variables,
  e.g. \(Y \in \{0,1,2,\ldots\}\) for votes

• Tobit model for censored variables,
  e.g. \(Y > 0\) for forest loss

• Heckit model for non-random samples,
  which uses the Heckman correction by modeling the sampling probability

• Multinomial probit/logit model for categorical variables,
  e.g. \(Y \in \{\text{agree}, \text{disagree}, \text{unsure}\}\)

Count Data

Count data take non-negative integer values \((0,1,2,\ldots)\) and often include a substantial number of zero outcomes (“zero-inflated”).

To build a model for this kind of data, we could:

• think of a latent normal variable behind \(Y\), or
  
• use a discrete probability distribution.

The Poisson distribution is one example; we can use it to express the probability that a given number of events occurs in a fixed interval.

Poisson Distribution

The probability mass function (PMF) of the Poisson distribution is

\[ \mathbb{P}(Y = y_i \mid \lambda) = \frac{\lambda^{y_i} \exp(-\lambda)}{y_i!}, \qquad y_i = 0,1,2,\ldots \] where the parameter \(\lambda\) is also the expectation \(\mathbb{E}[Y]\) and the variance \(\mathbb{V}(Y)\).

Poisson Distribution(2)

We generally expect that the expectation, i.e. the mean \(\lambda = \mathbb{E}[y]\), depends on other variables. Consider a Poisson model with a dependent mean; let

\[ \lambda = \mathbb{E}[y \mid X; \beta] = \exp(X\beta), \]

where we use the exponential function to ensure that \(\mathbb{E}[y \mid X] > 0\). We obtain

\[ \mathbb{P}(Y = y_i \mid x_i; \beta) = \frac{\exp(x_i \beta)^{\,y_i} \, \exp\!\left(-\exp(x_i \beta)\right)} {y_i!}, \]

describing the probability of each observation.

General Models

• We need a better way to estimate more general models, such as probit and logit, which are non-linear in parameters.

\[ \boldsymbol{y} = G(\boldsymbol{X}, \boldsymbol{\beta}) + \boldsymbol{u} \]

• If we apply the OLS method, we should minimise ( ’ ).

• This will be problematic because we need to consider ( K ) (( ^K )) partial derivatives, and there is no closed-form solution.

• Non-linear least squares estimation is a conceptually straightforward approach. First, we approximate with a linear model, and then refine the estimates iteratively. However, estimates are generally not unique and inefficient — OLS is not BLUE.

Maximum Likelihood Estimation

• Maximum Likelihood estimation is a method for estimating parameters.

• It maximizes a likelihood function, the joint probability distribution of the data as a function of the parameters:

\[ \mathcal{L}(\boldsymbol{\beta}) = \prod_{i=1}^{N} \mathbb{P}\!\left( \boldsymbol{y} \mid \boldsymbol{X} , \boldsymbol{\beta} \right) \]

• Intuitively, we set \(\boldsymbol{\beta}_{ML}\) such that the observed data is most probable within our model

• The resulting estimator is consistent, asymptotically normal, and asymptotically efficient in most cases

• The likelihood \(\mathcal{L}(\theta \mid X)\) itself is not a probability - \(\theta\) can vary, not \(X\)

Maximum Likelihood Estimation (2)

• To make the computation easier we usually work with log-likelihood \[ \ell(\boldsymbol{\beta}) = \log \mathcal{L}(\boldsymbol{\beta}) = \sum_{i=1}^{N} \log \mathbb{P}\!\left( \boldsymbol{y} \mid \boldsymbol{X} , \boldsymbol{\beta} \right) \]

• \(\boldsymbol{\beta}_{ML}\) is then the estimate that maximises the log-likelihood function

• The equation \(\frac{\partial\ell(\boldsymbol{\beta})}{\partial\beta}=0\) generally has no closed form solution, and iterative optimization algorithms are used such as Gradient Descent and Newton’s method

Maximum Likelihood for binary outcomes

• A distributional assumption lies at the center of Maximum Likelihood estimation.

• For binary outcomes, where \(Y \in (0,1)\), we can use the Bernoulli distribution with proability mass function:

\[ f(y_i \mid p) = p^{y_i}(1-p)^{1-y_i} \]

• Given the independence of observations, the joint probability is

\[ f(y_1, y_2, ..., y_n \mid p) = \prod_{i=1}^{N} p^{y_i}(1-p)^{1-y_i} \]

Maximum Likelihood for binary outcomes (2)

• With a Bernoulli outcome, we can use the following likelihood \[ \mathcal{L}(p) = \prod_{i=1}^{N} p^{y_i}(1-p)^{1-y_i} \]

• To find \(p_{ML}\), we maximise the likelihood by solving \(\frac{\partial \mathcal{L}}{\partial p} = 0\)

• The product form of the likelihood is difficult to differentiate — we would prefer a sum.

• Using properties of the logarithm, we instead maximise the log-likelihood.

• We therefore solve: \[ \frac{\partial \ell}{\partial p} = \frac{\partial \sum_i \log \!\left[ p^{y_i}(1-p)^{1-y_i}\right]} {\partial p} = 0 \]

Maximum Likelihood for binary outcomes (3)

To obtain the ML estimate, we first reformulate the log-likelihood as

\[ \begin{aligned} \ell(p) &= \sum_{i=1}^{N} \log \!\left[ p^{y_i} (1 - p)^{1 - y_i} \right] \\ &= \sum_{i=1}^{N} y_i \log p + (1 - y_i)\log(1 - p)\\ &= N \bar{y} \log p + N (1 - \bar{y}) \log(1 - p).\\ \end{aligned} \]

Where the last step relates the summation to the sample mean: \[ \sum_{i=1}^{N} y_i = N \bar{y}. \]

Maximum Likelihood for binary outcomes (4)

We know that

\[ \ell(p) = N \bar{y} \log p + N (1 - \bar{y}) \log(1 - p), \]

which we need to differentiate with respect to \(p\) and solve for \(p_{ML}\).

\[ \begin{aligned} \frac{\partial \ell(p)}{\partial p} &= \frac{N \bar{y}}{p} - \frac{N (1 - \bar{y})}{1 - p} = 0 \\ \frac{N \bar{y}}{p} &= \frac{N (1 - \bar{y})}{1 - p} \\ \bar{y}(1 - p) &= p(1 - \bar{y}) \\ p_{ML} &= \bar{y}. \end{aligned} \]

The maximum likelihood estimate is the average number of occurrences in the sample.

Maximum Likelihood for logit models

With logit models, we have a Bernoulli outcome \(Y\), and model the probability \(p\) using the logistic function. We have the following PMF:

\[ \begin{aligned} \mathbb{P}(Y = y_i \mid x_i) &= p^{y_i} (1 - p)^{1 - y_i}\\ &= \left( \frac{e^{x_i \beta}}{1 + e^{x_i \beta}} \right)^{y_i} \left( 1 - \frac{e^{x_i \beta}}{1 + e^{x_i \beta}} \right)^{1 - y_i} \end{aligned} \] We then set \(\beta_{ML}\) by (numerically) maximising the log-likelihood:

\[ \ell(\beta) = \sum_{i=1}^{N} \left[ - \log\!\left( 1 + e^{x_i \beta} \right) + y_i x_i \beta \right]. \]

Maximum Likelihood for poisson models

With Poisson models, we have a Poisson outcome \(Y\), and model the mean \(\lambda\) using an exponential function. We have the following PMF:

\[ \mathbb{P}(Y = y_i \mid x_i) = \frac{ \exp\!\left( x_i \beta \right)^{y_i} \exp^{- \exp\!\left( x_i \beta \right)} }{ y_i! }. \]

We then set \(\beta_{ML}\) by (numerically) maximising the log-likelihood:

\[ \ell(\beta) = \sum_{i=1}^{N} \left[ y_i x_i \beta - \exp\!\left( x_i \beta \right) \right]. \]

Linear Model and Maximum Likelihood

• Consider the standard linear model with normally distributed errors, given by

\[ \boldsymbol{y} = \boldsymbol{X\beta} + \boldsymbol{u}, \qquad u \sim \mathcal{N}(0, \sigma^2). \]

• This implies that \(y \sim \mathcal{N}(X\beta, \sigma^2)\)

• So far, we have used ordinary least squares to estimate the parameters — now we can also use maximum likelihood estimation.

• The Normal distribution, denoted by \(\mathcal{N}(\mu, \sigma^2)\), has the probability density function

\[ f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\sigma^2 \pi}} \exp\!\left( - \frac{(x - \mu)^2}{2\sigma^2} \right) \]

Linear Model and Maximum Likelihood (2)

We can obtain the likelihood function from the PDF:

\[ \mathcal{L}(\beta, \sigma^2) = \frac{1}{(2\pi)^{\frac{N}{2}} \sigma^N} \exp\!\left\{ -\frac{1}{2\sigma^2} (y - X\beta)'(y - X\beta) \right\}. \] To obtain estimates, we work with the log-likelihood:

\[ \ell(\beta, \sigma^2) = - \frac{N}{2} \log(2\pi) - N \log \sigma - \frac{1}{2\sigma^2} (y - X\beta)'(y - X\beta). \]

We will focus on \(\beta_{ML}\) — notice how the last term measures the squared deviations.

Linear Model and Maximum Likelihood (3)

To find \(\beta_{ML}\), we need to maximise the log-likelihood:

\[ \ell(\beta, \sigma^2) = \frac{N}{2} \log(2\pi) - N \log \sigma - \frac{1}{2\sigma^2} (y - X\beta)'(y - X\beta). \]

When taking the derivative, the first two terms drop out, and we obtain:

\[ \frac{\partial \ell(\beta, \sigma^2)}{\partial \beta} = -2\sigma^{-2} \left( -2X'y + 2X'X\beta \right). \]

• We obtain \(\beta_{ML}\) from \(\frac{\partial \ell(\beta, \sigma^2)}{\partial \beta} = 0\)
• And check whether \(\ell(\beta, \sigma^2)\)is maximal by checking the second derivative.

For the linear model with Normal errors, the OLS and ML estimates of \(\beta\) coincide.

Ridge and LASSO

Two common penalty terms, both added to the sum of squared residuals:

\[ \hat{\beta}^{\text{Ridge}} = \arg\min_{\beta} \left\{ (y - X\beta)'(y - X\beta) + \lambda \sum_{j=1}^{K} \beta_j^2 \right\} \]

\[ \hat{\beta}^{\text{LASSO}} = \arg\min_{\beta} \left\{ (y - X\beta)'(y - X\beta) + \lambda \sum_{j=1}^{K} \lvert \beta_j \rvert \right\} \]

• \(\lambda\) sets the strength of the penalty: \(\lambda = 0\) gives OLS, \(\lambda \to \infty\) pushes all \(\beta_j \to 0\)
• We accept some bias in exchange for a lower variance
• Regressors should be standardised first, and the intercept is not penalised

Why LASSO sets coefficients to zero

The solution is where the RSS contours first touch the constraint region. The LASSO diamond has corners on the axes, so some \(\beta_j\) become exactly zero.

Coefficient paths

As \(\lambda\) grows, Ridge shrinks all coefficients, while LASSO drops them one by one. This is variable selection.

Ridge vs. LASSO

• We choose \(\lambda\) by cross-validation: the value with the lowest out-of-sample error
• The elastic net combines both penalties

Ridge and LASSO in R

library(glmnet)

X <- model.matrix(mpg ~ ., data = mtcars)[, -1]
y <- mtcars$mpg

ridge <- cv.glmnet(X, y, alpha = 0) # alpha = 0: Ridge
lasso <- cv.glmnet(X, y, alpha = 1) # alpha = 1: LASSO

coef(lasso, s = "lambda.min") # some coefficients are exactly 0
plot(glmnet(X, y, alpha = 1), xvar = "lambda")

glmnet standardises the regressors by default and chooses \(\lambda\) by 10-fold cross-validation.

Summary

• ML estimators are based on the probability distribution of \(Y\).
• We learn about the:
  • parameters of this underlying distribution,
  • conditional on the data we observe and the chosen distribution.

To find an ML estimator, we:

1. Model the probability of each observation.
2. Derive the joint probability of all observations.
3. Consider the joint probability as a function of its parameters \(\theta\), conditional on the data \(\mathcal{D}\). This gives us the likelihood function \(\mathcal{L}(\theta \mid \mathcal{D})\).
4. Maximise the log-likelihood \(\ell(\theta \mid \mathcal{D})\) with respect to \(\theta\).

Exercises

• Derive the log-likelihood for the logit model (result slide 11)

• Derive the log-likelihood for the poisson model (result slide 12)