Modelling complex phenomena is as much an art as it is a science. The past twenty years have given us a number of examples where models have gone very badly wrong. These examples serve to demonstrate very clearly the inherent dangers of taking models too seriously. Above all, they have shown us that we must always beware of geeks bearing models. Business and public leaders who put faith in models without questioning their core assumptions are simply begging for trouble.

Always Question Your Assumptions

More years ago than I care to remember, one of my very first managers gave me some unforgettable advice. He taught me what the word assume really means: “making an ASS out of U and ME”. I have been repeatedly reminded of this sage wisdom in the years since. Assumptions are at the core of any complex model, and failing to question them always lands people in soup.

An example or two might be in order. Since my background is in mathematics, economics, and mathematical finance, I’ll start with a well-known one in the world of banking.

The concept of Value-at-Risk is well known and understood within that world. If you want to get the skinny on the concept, read Chapter 22 of this book. The basic idea behind VaR is to ask the question: “how bad could things get?” The answer is presented in the form of a single number. That number essentially says:

I am X% confident that we will lose no more than $Y over the next Z days.

This is an easy number for senior management to understand. That is why most capital markets businesses and regulators use it. Most capital markets businesses use a 95% confidence level and either a 3-day or 10-day time horizon. And, insofar as it measures anything, VaR is a useful measurement of overall portfolio risk.

But it has at several huge problems. And these are not new or surprising. Practitioners have known about these issues for decades.

Modelling Value-at-Risk: A “Crash” Course

For one, there are many different ways of measuring VaR. Let’s go over a few here.

Historical Simulation

One popular method involves using historical returns to model future ones. Essentially, you take the past 500 days’ worth of financial data, calculate returns from it, and then rank those from worst to best. Next, you take the 5th percentile return, or the 25th worst return. (If you want 1% VaR, you take the 5th worst return.) Then you essentially project that return onto the current value of your portfolio.

The keen observer will immediately note the problems with this method. First, some time series for financial data don’t go back 500 days. That’s “only” about 2 years’ worth of trading days, but some products are illiquid or new and lack that much history. And second, the history of financial and market crashes shows us that severe financial instability takes place more often than we think, but less often than once every two years.

That means that a 2-year time horizon is too short. But this leaves us with a severe dilemma. If we take too long a time horizon, we often won’t have enough data. And if we don’t have enough data, how can we be sure that we have captured enough market volatility to really know how bad things can get?

How, then, can we model VaR?

The Normal Approach

This leads to the second preferred approach, which imposes a distribution of some kind on top of financial returns. The one most frequently chosen is the good old Gaussian or normal distribution. Why? Because it is exceptionally easy to work with, very well understood, and mathematically straightforward. You could create a basic VaR model in a spreadsheet with such a distribution in a spreadsheet in about 30 minutes.

The assumption behind such a parametric model is that portfolio returns follow a specific distribution. Figuring out the 5th percentile of returns is then trivial. Just grab your trusty AVERAGE, STDEV, and NORMDIST functions in Excel and you’re off to the races.

But this is no solution either. First, every portfolio is made up of underlying instruments with their own distributions of returns. How can we be sure that a normal distribution captures the dynamics of an entire portfolio accurately? Second, if a lot of positions in that portfolio are tightly correlated, then the actual risk involved will be vastly higher than you think. Correlated instruments, by definition, all move in the same direction during a market movement. Your single number that captures all risk will be garbage in that situation.

You can get around that problem by using a multivariate normal distribution with a correlation matrix. But you are still imposing an assumption on top of the data. And that assumption implies that your portfolio returns are “smooth” and well-behaved. This is usually very unrealistic.

In some desperation, you might then choose to use a more powerful method. Several come to mind, all with fancy mathematical names. For portfolios made up of interest rate instruments, you could use Principal Components Analysis. And for other, more complex positions, you could use full simulation approaches.

None of which will solve your actual problem.

Dancing in the Digital Rain

In the energy risk industry, full Monte Carlo Simulation is often the preferred approach. That is because derivatives in that world move in very, very strange ways. A lot of equity traders would blanch if they came across positions with 50% volatility – but when dealing with electricity contracts, 400% volatility isn’t all that rare. That’s the kind of volatility that requires a fully-stocked bottle of Maalox on one’s desk at all times.

As such, most energy risk management requires some level of simulation modelling using MCS of all portfolio positions and returns. A reasonably modern computer can do this fairly quickly. The algorithms and methods involved are highly complex and technical and you need real systems for a portfolio of, say, power contracts. But you can perform some of the more basic ones for oil or gas forwards in a spreadsheet.

So surely this solves the issue, right? With MCS, you can simulate out the value of any position in your portfolio. You can then simulate your entire portfolio value. And from there, you can calculate the worst returns and figure out how much you could lose.

Not so fast. In order to run a simulation model – note the word there – you once again have to assume certain things. You must make assumptions about how your instruments behave. For reasons of mathematical tractability, we typically assume that the “randomness” process of an instrument basically follows a normal distribution.

And that is where we always run into problems. In the real world, most financial instruments DO NOT actually follow such a distribution very well. Their distributions have what we call “fat tails”. This means that extreme events happen much more frequently than a standard Gaussian distribution would permit.

A number of solutions have been proposed to this problem. All have serious issues of their own. None provide a truly robust practical approach to risk estimation.

An Airbag that Deploys Days AFTER an Accident

As a result, Value-at-Risk is often misleading. One can easily be fooled into thinking that everything is fine. In reality, VaR models cannot account for a great many real-world factors. These include trader behaviours and risk appetites.

Here’s a good example that shows you why you should beware of geeks bearing models. I once did product control for a book of inflation derivatives. Years after I left, the new head trader of that book held the view that the incoming Trump Administration’s policies of tax cuts and loose money would lead to higher inflation. He took up a number of large long positions on the benchmark CPI index. His reasoning was solid and his logic was valid.

But the inflation numbers never took off. Somehow, the economy roared back to life without inflation going up. (This actually isn’t so surprising, but to understand why, you have to treat everything you learn in college about economics as a complete lie – which it is.) And the book bled money throughout the year as a result.

That book once lost over $10 MILLION in a single day. I know the people who closed the book back then. To say that it was a rough night is an EPIC understatement.

Now, at no point did the internal VaR models throw off any signals that anything was amiss. The losses were actually well within limits. But that didn’t stop the losses from stacking up for days on end. By the end of that summer, the book was deep underwater and two traders lost their jobs.

The lesson is clear: there is a human cost to mistaken assumptions and bad or overly tolerant models.

It’s also worth pointing out here that, as the diagram above shows, VaR only tells you what happens when things go south. VaR cannot tell you how bad things could get when the best laid plans aft gang agley. For that, you need something called Conditional VaR – which is much harder to calculate and express.

The Human Cost of Bad Assumptions

Here’s another modern example to beware of geeks bearing models that probably hits a bit on the nose for some. I don’t apologise for this. Some things simply need to be said.

If you take a look at the virological and epidemiological models for the current pandemic, you will notice something important. Virtually every single one of them has WILDLY overestimated deaths and hospitalisation rates.

“Experts” warned us repeatedly back in February and March of the dangers associated with not locking down and “flattening the curve”. We were warned that 2.2 million in the USA and 500,000 in the UK could die from COVID-19. These estimates were based on assumptions of a 1% mortality rate and a certain rate of infection.

Those assumptions have since turned out to be utter garbage.

The best estimates we now have of the true mortality rate of COVID-19 are closer to 0.2% for patients under 70 in age, not 1%, and the real mortality level is almost surely lower than that.

Now I have no expertise whatsoever in epidemiology. So I make no representations of any kind about who did the modelling or how. (I’ll leave that to the always-hilarious British tabloids.) But I do have sense enough to understand the difference between mortality rates of 3-5%, 1%, and 0.2-0.1%. These are HUGE and consequential differences.

Yet those models are used as justifications for locking down entire populations. The catastrophic economic, social, mental, and moral damage done to billions of people has been ignored. Nobody bothered to stop and question the assumptions. And now, people feel as though they have been ripped off and lied to – but when they voice those objections, they are suppressed and called crazy and persecuted.

Conclusion – Always Beware of Geeks Bearing Models

Equo ne credite, Teucri! Quidquid id est, temeo Danaos et dona ferentes.

(Do not trust the horse, Trojans! Whatever it is, I fear the Danaans [Greeks], even when they come bearing gifts.)

— The Trojan priest Laocoön, from Virgil’s Aeneid

All I can say, based on over a decade of experience in watching models go horribly wrong with devastating human costs, is that a little more humility and a little less blind faith in models would be advisable.

And many of the so-called “experts” whose horribly misbegotten advice led us all from disaster to catastrophe with monotonous regularity this year, could do with eating a bit of humble pie. In banking, when your model mispredicts by up to THREE THOUSAND PERCENT and sometimes more relative to observed data, you lose your job. In public service, apparently, you are elevated into a media hero. That’s good work, if you can find it.

We must never forget that the people who create models are just that – people. They are flawed and Fallen just like the rest of us. The fact that they have fancy degrees to their name makes no difference whatsoever. Doctors, whether of the PhD or MD kind, are not gods. And any models that they rely on are imprecise and inexact replications of messy and complex realities.

ALWAYS question your assumptions – or you may wake up one day to find yourself with a tail and long ears, getting whacked with a baseball bat in the court of public opinion.