Learning Data Science: Why a High R^2 Can Be Misleading

A top can make a regression model look incredibly accurate – but this figure can be misleading. If you want to understand why a high is not always a sign of a good role model, keep reading!

In the article Learning Data Science: Modeling Basics, we built a simple model to predict income based on age. R printed a model summary containing something called R-squaredbut we haven’t yet discussed what this value actually means.

At first glance, a high level that seems very reassuring. In our example, the linear model obtained a almost 90%. This looks impressive.

However, just as high classification accuracy can be misleading — as discussed in ZeroR: The Simplest Classifier Possible, or Why High Accuracy Can Be Deceptive — high accuracy can be misleading. can also create a false sense of confidence.

To understand why, it helps to look at the formula itself, then revisit the three models from the previous post: the medium modelTHE linear modeland the polynomial model.

The meaning of

The coefficient of determination is defined as:

At first glance, the formula seems intimidating, but its basic idea is relatively simple.

The denominator

measure the total variation of the target variable. It quantifies the extent to which observed values ​​differ from their mean.

The numerator

measure the unexplained error remaining after model fitting.

So, measure the proportion of variation explained by the model.

A of:

• 0 means that the model does not explain any variation,
• 1 means that the model perfectly explains all variations.

It seems simple enough. The difficulty is that perfectly explaining observed data is not necessarily the same thing as building a useful predictive model.

The average model

Let’s start with the simplest possible regression model.

Suppose we ignore age completely and simply predict the average income of each individual:

This is effectively the regression equivalent of ZeroR. The model does not learn any relationships.

In this case:

Thus, the residual sum of squares becomes identical to the total sum of squares:

Substituting this into the formula, we get:

The model does not explain any of the variations in the data.

This corresponds to the underfitting case mentioned previously: the model is too simple to capture the underlying structure.

The polynomial model

Now consider the opposite extreme.

Instead of fitting a straight line, suppose we fit a polynomial of sufficiently high degree. In fact, if we have observations with distinct age values, a degree polynomial up to can pass exactly through all observed data points:

In this case :

for all observations, involving:

and therefore:

The model achieves a perfect fit.

At first glance, this seems ideal. In practice, however, such a model often performs poorly on unseen data because it has adapted not only to the underlying relationship, but also to random fluctuations and noise within the training data.

It’s the classic overfitting issue.

A perfect may therefore indicate not a particularly good model, but a model that has become too flexible.

The linear model

The linear model from the previous post falls between these two extremes.

It’s simple enough to avoid remembering every random fluctuation, but flexible enough to capture a meaningful trend in the data.

This balance between simplicity and flexibility is one of the central themes of statistical learning.

The idea was summarized in the previous post with the following plot:

and by the famous observation attributed to George Box:

“All models are wrong, but some are useful.”

The objective of modeling is therefore not to maximize complexity or to maximize but to find a model that generalizes well beyond the observed sample.

For what Alone is insufficient

The main limitation of is that it evaluates the fit only on the observed data.

It does not directly measure:

• predictive performance on invisible data,
• robustness,
• causal validity, or
• ability to generalize.

As the complexity of the model increases, almost always increases as well. A sufficiently flexible model can often achieve values ​​very close to 1 even when its predictions on new data are poor.

For this reason, practical data science relies on additional evaluation methods such as:

• train test splits,
• cross validation,
• regularization,
• adjusted And
• out-of-sample testing.

The goal is not to perfectly reproduce historical observations, but to build models that remain useful in the face of new data.

A top can therefore mean two very different things:

• the model identified a real structure,
• or the model simply overfitted the training data.

Distinguishing between these possibilities is one of the central challenges of machine learning and statistical modeling.

Related