Pundit Consensus Forecasts

In my last blog, I demonstrated that the weekly EPL predictions of Mark Lawrenson (for BBC Sport) and Paul Merson (for Sky Sports) consistently beat the betting market. Betting a constant amount on their predictions (home win, away win or draw) over the last 3 years would have resulted in an 8% profit, on average, for both pundits. I also showed that a lot of this performance is down to their ability to predict draws, something that the bookmakers – and most statistical models – are quite poor at doing.

So far I’ve treated Lawrenson and Merson’s predictions separately, but what if we combine them? Can we improve their predictive power? They may feature on rival networks, but in this post I’m going to look at what happens when Merson and Lawrenson work as a team.

Consensus Forecasts

From the 2014/15 season through the 2016/17 season there were 1101 EPL matches for which Merson and Lawrenson both made a prediction. But how frequently did they make the same prediction? The grid in Figure 1 shows how their predictions compared over these matches.

Figure 1: The joint distribution of Lawrenson’s and Merson’s predictions since 2014/15.

The rows indicate Lawrenson’s predictions (top row: home win, middle: away win and bottom: draw) and the columns Merson’s predictions (left: home win, middle: away win and right: draw); each cell therefore represents a pair of predictions. The numbers in each cell give the proportion of the 1101 matches in which the pundits made those predictions. For example, the top-middle cell represents those matches in which Lawrenson predicted a home win but Merson predicted an away win. It shows that this particular combination of predictions occurred in only 3% (37) of the 1101 matches.

The shaded cells down the diagonal represent the matches in which the pundits predicted the same outcome. In 41% (450) of the 1101 EPL matches, Lawrenson and Merson both predicted a home win; in 12% (134) they both predicted an away win; and in 10% (111) they both predicted a draw. In total, the pundits made the same prediction for 695 of the 1101 matches, so Lawrenson and Merson agree nearly two-thirds of the time^[1]. The remainder of this post focuses on these matches specifically, which henceforth I’ll refer to as the consensus forecasts.

How accurate are the consensus forecasts? 

Table 1 shows the prediction accuracy – the proportion of their predictions that were correct – for Merson and Lawrenson individually, and for the consensus forecasts (i.e., when they agreed).

Table 1: Proportion of correct predictions for the pundits individually, and combined (‘consensus’).

Individually, both pundits have a success rate of just over 50%: they predicted the outcome correctly in a little over half of all EPL matches. Breaking this down, we see that their home win predictions have been correct in 60% of matches; away win predictions in around 56% of matches; and their draw predictions in 33%. As discussed in my previous post, draws are hard to predict and a 33% success rate is actually pretty good.

However, when both pundits agree  – the consensus forecasts  – their overall success rate increases by nearly 10%, from just over 50% to 60%. Specifically, their home win prediction success increases by a few percentage points to 62%; their away win prediction success by around 8 points to 64%; and their draw predictions by 10 points from 34% to 44%. That’s a significant improvement, particularly for draws.

Aren't the consensus forecasts just matches that were easier to predict?

The obvious criticism is that the pundits may just agree on the matches that are easier to predict and the improvement in prediction success is just a result of that. This is partly the case, at least for home and away wins. The average odds offered on these outcomes for the consensus forecasts tend to be about 10-15% lower (i.e., shorter) than those offered on the individual pundits' home and away win predictions. So the market does view these matches as being easier to predict.

However, there is almost no difference in the odds offered on draws between the consensus predictions and the pundit’s individual predictions. As I demonstrated last time, the betting market is actually quite poor at predicting draws (in particular, see this plot), so by combining the pundits we’re simply increasing our advantage over the bookmakers.

Do the consensus forecasts yield higher betting returns?

Could I have increased my profit by combining their predictions and betting solely on the consensus forecasts? Table 2 shows the average profit per game for the pundits individually, and their consensus predictions, over the 1101 matches. 

Table 2: Average betting return for the pundits individually, and for their combined ‘consensus’ forecast. NB: numbers for Lawrenson are different to Table 3 in my previous post as I’m using predictions from 14/15 onwards; in the last post I used all his predictions since 11/12.

Lawrenson and Merson individually made an 8% profit, on average. This increases to 12% when we consider only those matches that they agreed on (the consensus predictions): a substantial increase in return^[2]. Breaking this down, we see that the increase in profit in the consensus forecast is driven by the draw predictions. Betting on Lawrenson's or Merson’s draw predictions individually has yielded about a 20% profit per match, on average; this increases to 53%(!) when they both predict a draw. As demonstrated above, the odds offered on draws for the consensus forecasts do not differ significantly to the odds offered on draws for the individual pundit predictions, so this whopping increase is entirely generated by the increase in prediction success rate from 33 to 44%.

For home and away wins, the performance of the consensus forecast is less impressive (relative to the individual pundit forecasts), despite the prediction success rate being higher.  As demonstrated above, the odds offered on home or away wins are systematically lower for the consensus forecasts because, when both pundits predicted a winner, they tended to agree for the matches that were easier to predict. So while the consensus forecasts get more matches right, their average profit on the correct predictions is lower.

Comparing betting profits in cash terms for the pundits individually and the consensus forecasts is complicated by the difference in the number of games that you'd be betting on. If you bet £10 on each of Merson's predictions in a single season you would bet a total of £3800 over the 380 matches, with an expected profit of £304 (8%, based on past performance). If you bet the same total amount (£3800) over a season on the consensus forecasts you would bet roughly £16/match over 240 matches (remember: the pundits agree in about 63% of matches), with an expected profit of £460 (12%). Of course, you wouldn't know in advance exactly how many matches the pundits would agree on in that season (although see footnote 1) .

Summary

Mark Lawrenson and Paul Merson predict the same outcome in nearly two-thirds of EPL matches; when they agree their prediction accuracy increases by nearly 10%, from a 51% success rate, to a 60% success rate. Betting solely on these matches would, on average, have yielded a 12% profit per match, a significant increase on the 8% return per match yielded by betting on the predictions of either one of the pundits individually. Much of the improvement is driven by their draw predictions: when the pundits both predict a draw – which occurs in about 10% of matches – they are correct 44% of the time, yielding a whopping 53% profit per match, on average. As described at length in my previous post, draws are tough to predict and the betting market is not particularly good at it.

So where do we go from here? If your goal is to maximize returns you could explore various strategies for determining the optimal bet size. That’s not my main objective though. While the betting market provides a useful baseline for evaluating predictions, my principal interest is how we can improve forecasting by combining human predictions with those from statistical models. Of course, a huge amount of work has been done on the latter -- there are several large media outlets (e.g. FiveThirtyEight, Sky and the FT) and a large number of individual analysts (see, for example, here and here) that regularly publish stats-based predictions. However, I’m convinced that incorporating a human component could make significant improvements; I'm open to collaboration to explore hybrid methodologies. 

For now, I’ll continue to monitor the performance of the pundit’s predictions and their consensus forecast over the current season. You can find latest results here. And maybe I'll wager a little money myself, too. ☺

-------------

[1] The 695 matches were fairly evenly spread over the three seasons, 227 in 14/15, 216 in 15/16 and 251 in 16/17.

[2] The Sharpe ratio – the annualized return per unit risk – for both pundits individually is 1.3. It increases to 1.7 for the consensus forecast.