We’ve tracked Google’s 2018 World Cup predictions for each match and compared them to the actual results. Here’s what we found out.
There are 64 matches, 48 in the group stage and 16 in the knockout stage. We’ve analyzed the total number of matches Google predicted correctly over the total number of matches and we have divided our analysis into three dimensions and the answer to the most important question, who will be the winner of the 2018 World Cup:
- Overall accuracy for the whole tournament;
- Accuracy for the group stage;
- Accuracy for the knockout stage;
- The limitations of the predictions
World Cup Predictions
If the previous World Cups were marked by all sorts of animals “predicting” the winners, the 2018 World Cup will be remembered for the arrival of some serious competition. While Achilles “The Psychic Cat” and Nelly The Elephant are still around, making their predictions, this time they will have to share the stage with some big shots. To start with, remember the yellow from Springfield that predicted Donald Trump’s victory in 2016 US Elections? Well, The Simpsons also apparently predicted that Mexico and Portugal were going play the final.
But what the 2018 World Cup will really be remembered for is the arrival of the robots (scary!?). From Goldman Sachs to UBS, from Google to IBM, this year Wall Street and Silicon Valley behemoths are putting their machine learning teams to work at predicting the most important question human beings have been asking every four years:
Who will win the World Cup?
Google has adopted a different approach. Instead of trying to predict the winner, they have set predictions for every single match in the group stage and have been adding new predictions to the knockout stage. It’s a bettor’s dream. But, how much can you really trust Google’s predictions? In this article we tracked these predictions against reality.
Overall accuracy for Google’s 2018 World Cup predictions
Google’s predictions proved to be right for the first two matches (a 100% accuracy rate), but as the tournament progressed, the prediction accuracy decreased and the overall accuracy rate (total matches accurately predicted/total matches) hasn’t gone above 70% since the second match and ended the tournament at 59.4%.
World Cup Group Stage Predictions
The group stage finished with an average accuracy rate lower than 60% (56.3% to be exact). It’s no surprise that the accuracy rate was close to 50%, provided that there have been lots of unexpected results in this World Cup!!
Which matches did Google accurately predict?
One could say that Google got certain matches more or less right– the draws for example. In order to simplify the analysis, we choose not to consider scores of matches. Rather, we account for wins and losses and approach outcomes from a binary perspective. Did the team that had a higher probability of winning actually win the match? Google was right for the following 27 matches:
Which matches did Google predict inaccurately?
Google was wrong for the following 21 matches…
2018 World Cup Knockout Stage Predictions
Overall accuracy for the knockout stage
If we analyze the knockout stage independently from the Group Stage, Google’s predictions have increased from **56.3% in the group stage to 68.8% in the knockout stage. **
Which matches did Google accurately predict in the Round of 16?
In the Round of 16, Google was right for 5 matches and wrong for 3,** as a result Google got all Quarter-finals matches wrong but one. Our friends from Mountain View predicted Portugal v. France in a revival of 2016 Euro finals (wrong), Brazil v. Belgium (Right)**, Spain v. Croatia (Wrong) and Switzerland v. England (Wrong).
Which matches did Google accurately predict in the Round of 16?
Which matches did Google predict inaccurately in the Round of 16?
Which matches did Google accurately predict in the Quarter-finals?
In the Quarter-finals, Google was right for 3 matches and wrong for only 1 (Brazil v. Belgium). As a result they got one Semi-final right, Croatia v. England and one wrong France v. Belgium.
Which matches did Google accurately predict in the Quarter-finals?
Which matches did Google predict inaccurately in the Quarter-finals?
Which match did Google accurately predict in the Semi-finals?
In the Semi-finals, Google was **right for 1 match (France v. Belgium). As a result they didn’t get the Final right. **
Which match did Google accurately predict in the Finals?
In the Finals, Google was **right both for the third place and the winner of the World Cup. **
The limitations of the predictions
From the elimination of Germany to Argentina having to sweat blood to get past the group stage, this is a World Cup where surprises are the new norm. Not even Achilles “The Psychic Cat” anticipated that Germany would succumb to South Korea. But what’s the real impact of surprises on Google’s predictions? For instance, if Germany and Argentina had won all of their 3 matches, the average accuracy rate would have increased to 64.5%, as shown in the chart below.
You don’t have to be a German supporter flying back home to agree that an accuracy rate close to 50% is not enough to plan your World Cup vacations. We don’t have access to how Google calculated the predictions, but we can speculate that one of the biggest limitations of the model is inherent to the whole World Cup dynamics: the number of times national teams play before the World Cup.
Although the model might take into account team’s performance in the last 18 to 24 months, its FIFA ranking and how well the players have been performing before the World Cup, there aren’t enough data points to predict the interactions between the players and the interactions between the two teams:
- The team: before the World Cup starts, the national teams play a couple of times a year and coaches test several different formations, till they find an “ideal” one. By changing the players, we also change the model. The problem is that once the model is stable, with the ideal team, we don’t have enough observations and the model gets trained in real-life conditions, namely the World Cup.
- The match: the two teams don’t play against each other with the same teams very often, that makes it extremely difficult to create a model that has enough similar observations to train the model.
.svg)
.svg)
