Analytical sophistication and practical reliability are not the same thing.

When faced with a data problem, it’s tempting to reach for the most technical, impressive-sounding method available. However, in many cases, a simpler analysis can produce similar results within a more robust framework. 

To illustrate this point, consider a classic teaching dataset for Machine Learning: the Boston Housing data. This dataset contains 13 variables, some describing the characteristics of the housing units such as room counts and tax rates, while others capture neighbourhood conditions, including crime levels, distance to employment centers and socioeconomic status of households in the area.

Using this dataset, we can predict the median self-reported house values of neighbourhoods using a range of statistical and machine learning techniques. One widely used approach is the Random Forest algorithm, which constructs a large number of decision trees and combines their predictions through a process known as bootstrap aggregation, or bagging. By averaging the predictions of many trees, Random Forest produces more accurate and robust estimates than a single decision tree. The model’s predictions are presented in the chart below.

Clearly, the model does a great job at predicting the median self-reported house values based on all the discovered patterns and correlations between the 13 explanatory variables.  

But this is a pristine dataset. In the real world, data is messy. It is often incomplete, inconsistent, and incorrectly imputed.  

To illustrate the risks of applying complex statistical techniques such as Machine Learning to messy data, a second round of analysis deliberately contaminates the originally pristine data and introduces some real-world complications such as missing values, outliers, duplicates, inconsistent datatypes and a measure of random noise for added realism. 

When we re-run the analysis, the same Random Forest model is significantly less predictive.  

On the clean data, the Random Forest model explained 93% of the variation in median house values (R2R^2 = 0.93). An outstanding result. 

After introducing real-world problems to the data, the R2R^2 dropped to 0.67. 

Even after imputing missing values, tuning hyperparameters, and validating on held-out data, the performance gap remained.

The charts below show how predictive performance deteriorated once the data became messy, even after handling missing values in two different ways. In a perfectly predictive model, every point would sit neatly along the 45-degree line. Instead, the scatter widens, and precision begins to fade. Same model. Very different reality.  

This highlights the importance of robustness, by which we mean he model’s ability to remain reliable despite messy or noisy data.

Complexity is not the only path to insight

An alternative approach uses a far simpler statistical technique. It begins by examining the relationship between each individual variable and the median house value, before selecting only those variables with the greatest explanatory power. In this case, the two most important variables are the average number of rooms per house in the neighbourhood and a composite indicator of neighbourhood socio-economic conditions.

By dividing each variable into quintiles, we then construct a 5×5 grid that cross-tabulates average room count and neighbourhood context. We then calculate the median price (in US$ 000s) within each of the resulting 25 segments to summarise the structure of the data, as illustrated below. From this, we can estimate the median house value in each neighbourhood. 

This approach may not capture every subtle interaction that Machine Learning can detect, but that also means it is far less sensitive to outliers, noise and distortions in the data.

We can see how well each model predicts median house value by taking three random observations from our testing data and inputting it into our models. 

The simple analysis on messy, imperfect data compares very favourably, and often performs better than the Machine Learning models trained on messy and incomplete datasets.

Policymakers, business leaders, and investors rarely have the luxury of waiting for perfect data. Decisions must be made even when the information is incomplete, noisy, or unreliable. 

In these situations, it often pays to reduce complexity: to focus on broad patterns rather than individual data points. Doing so naturally filters out noise and limits the influence of outliers. 

None of this diminishes machine learning, which can be very powerful in the right context. Many problems genuinely call for advanced statistical techniques and sophisticated models. But there is always a trade-off between precision and robustness. It is entirely possible to be precisely wrong, and being roughly right (as Keynes noted) is a far better outcome.  

This is not an argument against machine learning, or for simplicity for its own sake. It is an argument for judgement. The best model is the one that matches the structure, quality, and purpose of the data at hand. When that data is incomplete and volatile, robustness often matters more than sophistication. 

Complexity should be earned, not assumed.