We are interested in finding a relationship between the samples

We can write this relationship as

\[ \boldsymbol{y} = \textcolor{var(--secondary-color)}{f(\boldsymbol{X})}+\boldsymbol{u}, \]

where \(f(\cdot)\) is an unknown function that represents information that \(\boldsymbol{X}\) provides about \(\boldsymbol{y}\). All other relevant information is contained in the error term \(\boldsymbol{u}\).

As you know, there are many names for the dependent variable, such as:

• outcome,
• response, or
• explained variable.

Likewise, we know a multitude of alternative terms for the independent variables, such as:

• explanatory variables, or
• predictors.

You also might come across different ways of denoting the error term, such as

\[ \boldsymbol{u},\qquad\qquad\qquad\qquad\boldsymbol{e},\qquad\qquad\qquad\qquad\boldsymbol{\varepsilon}. \]

We will use \(\boldsymbol{u}\) in the materials of this course, but you can choose whichever you prefer.

You may ask yourself, “what are we doing this for?” This is a very good question (and you should ask these types of questions very often), and there are two answers to it.

In the Kaggle Competition, you will get a training dataset \(\boldsymbol{X}\) and \(\boldsymbol{y}\), which you will use to estimate \(\hat{f}\). You can then predict \(\hat{\boldsymbol{y}}\) and check the predictions against \(\boldsymbol{y}\).

There is a second part of the data, the test dataset, of which you get only \(\tilde{\boldsymbol{X}}\), and we will keep the \(\tilde{\boldsymbol{y}}\). You will try to get good out-of-sample predictions, and in the end we will reveal who fared the best.

For prediction, we do not care about what our \(f(\cdot)\) looks like. As long as we get useful predictions, we can treat \(f(\cdot)\) as a black box.

The accuracy of our prediction depends on the sum of two types of errors:

Let us look at the mean squared prediction error.

\[ \begin{aligned} \mathrm{E}\left((\hat{\boldsymbol{y}}-\boldsymbol{y})^2\right) &= \mathrm{E}\left(\left(f(\boldsymbol{X})+\boldsymbol{u}-\hat{f}(\boldsymbol{X})\right)^2\right)\\ & =\textcolor{var(--tertiary-color)}{\mathrm{E}\left(\left(f(\boldsymbol{X})-\hat{f}(\boldsymbol{X})\right)^2\right)}+\textcolor{var(--quarternary-color)}{\mathrm{Var}(\boldsymbol{u})}. \end{aligned} \]

We have decomposed the mean squared error into a reducible and an irreducible part. We can now split the reducible error once more.

\[ \phantom{\mathrm{E}\left((\hat{\boldsymbol{y}}-\boldsymbol{y})^2\right)\qquad\quad} =\textcolor{var(--tertiary-color)}{\mathrm{Bias}\left(\hat{f}(\boldsymbol{X})\right)^2+\mathrm{Var}\left(\hat{f}(\boldsymbol{X})\right)}+\textcolor{var(--quarternary-color)}{\mathrm{Var}(\boldsymbol{u})}. \]

The reducible error consists of the squared bias of \(\hat{f}\) and its variance.

show decomposition #1 show decomposition #2

We want to minimize the reducible error as far as possible by balancing bias and variance. However, we want to avoid trying to reduce the irreducible error.

When we try to minimize the irreducible error, we will overfit our model.

Why is it bad to overfit? We call it an “irreducible” error for a reason. We can fit something that matches the data in the sample arbitrarily close. But this will lead to poor out-of-sample performance.

Occam’s (or Ockham’s, or Ocham’s) Razor, also called the principle of parsimony, is a simple rule:

We can use this principle to inform our notion of which model is “better” than the other.

In a sense, prediction and inference are opposite approaches. Before, we cared only about the fitted value and treated \(f(\cdot)\) as a black box; now, we care only about \(f(\cdot)\) (or, more precisely, our estimate \(\hat{f}(\cdot)\)).

With knowledge about \(\hat{f}(\cdot)\), we can answer questions like these:

Say I want to answer this question. I now only need to do two things:

It should be apparent that this is not possible. We call this the Fundamental Problem of Causal Inference. The existence of this problem is the reason that you have to take this course.

We need to deal with the Fundamental Problem of Causal Inference in some way if we want to perform causal inference. Just looking at the data and checking correlations (which is what we do with a naive regression) is not enough:

You likely have heard this sentence before:

As we know,

The model (i.e., subway map) on the previous slide is useful for navigating the subway. Of course, it is useless when you use a bike. Models are an approximation of reality that are specific to a certain context and allow us to learn specific things.

To learn about the true \(f\), we need a model that suits our purpose and the data at hand. We can characterize models, e.g., like this:

Parametric models are models that impose a certain parametric structure on \(f\). In case of a linear model, the dependent is a linear combination of \(\boldsymbol{X}\), with parameters \(\boldsymbol{\beta}\in\mathbb{R}^{K+1}\):

\[ \boldsymbol{y} = \boldsymbol{X\beta} \]

Non-parametric models do not impose a structure on \(f\) a priori. Rather, we fit \(f\) to be as cloase as possible to the data under certain constraints.

Supervised Learning includes everything we did so far. We have data on \(\boldsymbol{y}\) on which we can train our model, e.g.

Another example of Unsupervised Learning are Large Language Models. Since ChatGPT was released in late 2022, they have been pretty well known, but their development and public availability predates ChatGPT.

The first Generative Pre-Trained Transformer (GPT) was introduced by OpenAI in mid-2018. The model was called GPT-1, had 117 million parameters, and was trained on 7,000 unpublished books.

The model is able to complete a text prompt with meaningful sentences, but noticeably lacks context awareness.

The successor model, GPT-2, was published in early 2019. It was trained on text from documents and webpages that were upvoted on Reddit, and contained 1.5 billion parameters.

This model is more context-aware, but you can still easily see that it is generating words rather than meaning.

In 2020, OpenAI released GPT-3, which is no longer publicly available; and in 2022, it released GPT-3.5 These models have 175 billion parameters:

Of course, it does not know our syllabus, but this is a pretty reasonable guess for what a course entitled “Econometrics II” could be about.

Current models GPT-4o, GPT-4.5 and GPT-4.1 are rumored to have between 200 billion and 1 trillion parameters. The current product by Chinese competitor DeepSeek, DeepSeek V3, has 671 billion parameters. Claude 4 by Anthropic likely has a comparable number of parameters.

A regression problem is a problem with a quantitative dependent variable (e.g., height, econometrics grade, carbon emissions, …).

In contrast, we refer to a problem with a qualitative dependent variable as a classification problem.

The distinction between the two is not always perfectly clear:

Different methods have different degrees of flexibility, i.e. they can only produce a narrow range of possible shapes of \(f\). For example, linear regression is rather inflexible.

The benefit of choosing a flexible approach is evident. However, there is an important downside: More flexible methods yield results that are less easy to interpret.

We choose a model and estimation method depending on the issue of interest, and the available data. Central questions we may ask ourselves include the following.

Econometrics seeks to apply and develop statistical methods to learn about economic phenomena using empirical data.

Econometrics plays an important role in an empirical shift in economic research, away from pure theory (Angrist et al., 2017; Hamermesh, 2013). Today, economic theories are routinely confronted with real-world data.

“Experience has shown that each […] of statistics, economic theory, and mathematics, is a necessary […] condition for a real understanding of the quantitative relations in modern economic life.” — Ragnar Frisch (1933)

Econometric methods are constantly developing. There is no one-size-fits-all approach that fits any kind of data and research question. Econometrics has seen considerable challenges and developments since its inception. Important milestones concern

You can be sure that the methods we learn today will evolve and change within the next years, during your carreer, and beyond. This gives you an opportunity to go into econometric research if you choose this career path, but it also means that you have to keep up with new developments.

Consider how to transform the following economic model into an econometric model:

\[ \text{wage} \approx f({\text{education}}, {\text{experience}}). \]

A sensible choice might be the following linear regression model:

\[ \textbf{wage} = \textbf{education}\: \beta_1 + \textbf{experience}\: \beta_2 + \boldsymbol{u}. \]

Why is a linear model a sensible choice and why do we choose them so often?

The linear model’s popularity is not surprising, given the classical tasks:

The central task is arguably distilling a causal effect from observational data, since experimental data is rare.

When forecasting, economic theory can provide us with valuable structural information.

The linear model is an essential building block, and linear algebra gives us a very convenient way of expressing and dealing with these models. Let

\[ \textcolor{var(--primary-color)}{\boldsymbol{y}} = \textcolor{var(--secondary-color)}{\boldsymbol{X}} \boldsymbol{\beta} + \boldsymbol{u}, \]

where the \(N \times 1\) vector \(\symbf{y}\) holds the dependent variable for all \(N\) observations, and the \(N \times K\) matrix \(\symbf{X}\) contains all \(K\) explanatory variables.

That is,

\[ \textcolor{var(--primary-color)}{ \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{pmatrix}}= \textcolor{var(--secondary-color)}{ \begin{pmatrix} x_{1 1} & x_{1 2} & \dots & x_{1 K} \\ x_{2 1} & x_{2 2} & \dots & x_{2 K} \\ \vdots & \vdots & \ddots & \vdots \\ x_{N 1} & x_{N 2} & \dots & x_{N K} \end{pmatrix}} \begin{pmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_K \end{pmatrix} + \begin{pmatrix} u_1 \\ u_2 \\ \vdots \\ u_N \end{pmatrix}. \]

The ordinary least squares (OLS) estimator minimises the sum of squared residuals, which is given by \(\hat{\boldsymbol{u}}' \hat{\boldsymbol{u}}\) (i.e. \(\sum_{i = 1}^N \hat{u}_i^2\)). To find the estimate \(\hat{\boldsymbol{\beta}}_{OLS}\), we

\[ \begin{aligned} \textcolor{var(--quarternary-color)}{\hat{\boldsymbol{u}}'\hat{\boldsymbol{u}}} & = (\boldsymbol{y} - \boldsymbol{X} \hat{\boldsymbol{\beta}})'(\boldsymbol{y} - \boldsymbol{X} \hat{\boldsymbol{\beta}}) \\ & = \boldsymbol{y}'\boldsymbol{y} - 2 \hat{\boldsymbol{\beta}}' \boldsymbol{X}' \boldsymbol{y} + \hat{\boldsymbol{\beta}}' \boldsymbol{X}' \boldsymbol{X} \boldsymbol{\beta}. \end{aligned} \]

The estimator \(\hat{\boldsymbol{\beta}}_{OLS} = (\boldsymbol{X}' \boldsymbol{X})^{-1} \boldsymbol{X}' \boldsymbol{y}\) is directly available and a minimum. derivation