Brier Score Explained PDF

Brier Score enables us to generate a value for each game based on the gap between the probability attributed to the bookmakers, and the actual outcome for each game. This was actually conceived of in 1950 by a statistician called Glenn Brier, who was using it to measure the accuracy of weather forec...

Brier Score enables us to generate a value for each game based on the gap between the probability attributed to the bookmakers, and the actual outcome for each game. This was actually conceived of in 1950 by a statistician called Glenn Brier, who was using it to measure the accuracy of weather forecasts. But in fact, it fits quite nicely for any prediction model. This is a very natural way to evaluate the reliability of the forecast. If we think of a two outcome game, like the MBA, or Major League Baseball, or the Indian Premier League. You could think of the two outcomes are a win, or a loss for one of the teams. If it's a win you have a value of one, if it's a loss, you have a value of zero. The bookmakers have odds associated with each of those possible outcomes, a probability. What you do is you take the difference between the actual outcome, and the probability. If the actual outcome is a win, you get one minus the probability of a win, and then you square that so that it's a positive number. We want the absolute value, which gives us a sense of a measure of distance. We're also going to take the value of the outcome when you lose, which has a value of zero minus the probability of zero, probability of a loss, which would have a value of zero. Then again, we square that because we want to have an absolute measure. We don't want to pluses, and minuses to confuse the issue. That will give us a Brier Score for that particular game that we're looking at. We can then take the average of the Brier Scores across all the games to see how accurate overall the bookmakers predictions are? If you think about what the possibilities are here, so suppose you were completely accurate in terms of your predictions, then you would give a 100 percent probability of the actual outcomes. Suppose the team wins, then you'd have an outcome equal to one, and a probability of that equal to a 100 percent, or one. The gap between the outcome, and the probability would be zero, and zero squared is zero. Then the outcome whereof a loss which has a value of zero, you would have given a probability of zero to that. Again, you have zero minus zero, which is zero, and still zero when squared. Overall, the Brier Score, a perfectly accurate prediction, which generate a Brier Score of zero. Whilst if you were to be perfectly wrong, then you would have given a zero probability to the win the value of zero. So you generated the first part would be one. The outcome is win minus zero, because you gave it zero probability squared, which is one. Then other outcome, which is a loss, which didn't happen, so it has a value of zero. But you gave it a probability of a 100 percent, or one. That has a value of one as well. One squared is one, so we have one plus one. So you're maximally wrong if, when the Brier Score is equal to two. The Brier Score can be equal to, can range between zero, and two. Then if you think about what would happen if you chose a random. If you chose a random, you'd give a 50 percent probability to each outcome in a two-horse race. In that case, you'd have a Brier Score, which would be equal to one minus a half squared, which is a quarter, plus zero minus a half squared, which is also a quarter. So you'd have a Brier score of 0.5. If you chose a random, you'd get a Brier Score of 0.5. This gives us some sense of what the possibilities are. Completely accurate Brier Score is zero, random allocation then the Brier Score should be about a half, and a 100 percent wrong, then you'll get a Brier Score of two. Notice, of course in particular, the lower Brier Scores means more accuracy. We're going to extend that to the three outcome league, and that the logic works exactly the same way when there is possibility of a win, draw, or a loss, or a win overtime loss, or loss. Again, the accuracy, you can think of the outcomes looking the maximum value. When you have a perfect prediction, you're going to have a Brier Score of zero, when you're perfectly wrong, you're going to have a Brier Score of two again. If you pick a random, then now you get a slightly different score, you get a score of 0.666, or two-thirds. That represents the three possible outcomes, which gives us a sense of what to look for when we actually work out some Brier Scores. Let's do that. Let's look at the Briers Scores for the MBA for 2018, 19 season. Here we just take simply the squared difference between the actual outcome win minus the win probability. Here we have one minus the win, which is also a loss times the probability, the bookmakers probability of loss, which is one minus the win probability. When we take the mean of the Brier Score across all of the games in the data. If we do that and run that, so it turns out we get a Brier Score of 0.4, just over 0.4. You might think that's not particularly impressive. Remember, zero would have been perfect. 0.5 is what you'd expect it to go if you'd chosen at random. This is 0.4, which is closer to being random that it is to being perfectly correct. Now, probably a little word of caution here. It's really not very reliable to interpret Brier Scores just on their own. It's useful to take into account these reference values. Really where Brier Scores are really useful is when you're comparing them against other Brier Scores. Just think about how are we going to use this in the future, in the next week of this course, we are going to look at comparing predictions that we're going to generate against bookmakers predictions. We're going to use a Brier Score to evaluate them and there the interest is going to be who has the better, and which would hear mean smaller Brier Score. That's really the value of this. At the moment, you should reserve judgment about how effective you think the bookmakers are based on looking at these Brier scores. Well, as a self-test, you could look at this for major league baseball as well and generate the Brier Scores for them. Let's look at the Indian Premier League and look at what the Brier Score will say. Remember the Indian Premier League seemed to be the league where the bookmakers were least successful. Now this looks rather complicated, but it's actually pretty simple to work out. We have the same problem as before the Data for the Indian Premier League is presented in the format of American odds. Therefore, we have two possibilities depending on whether the odds are expressed as positive or negative. Then we need to scale the odds, but once we've scaled the odds, we can generate the Brier Score based on the difference between the actual outcome and the win probability squared. If we run this, we can calculate the mean. If we run that, we find that actually, so the Brier Score for the Indian Premier League is 0.504. Which I remember what I said. If you picked a random, you'd expect to get a Brier score of 0.5. That suggests that the bookmakers odds in terms of the Indian Premier League are almost completely random, which is again an interesting result. Perhaps surprising that this seems to be a very unpredictable league. Again, you might want to think about why that might be. Finally, what I want do is look at three league outcomes and develop the Brier Scores for that. We'll look at the NHL as an example. Again, this is pretty simple to do once you've seen the logic of the structure. We just need to develop, we load the data, we just need to work out the win probability and the tie probability by, again, the NHL data is expressed in decimal load, so the probability is one over the decimals, but we have to scale by the sum of the probabilities because of the over round. Then we can generate the Brier Score as the sum of the squared differences of the three possibilities, the win, the overtime loss, and the loss. If we run those and then generate the Brier Score as the mean. You can see that the Brier Score here is 0.589. Now remember that in a two outcome league, if you pick a random, you get a Brier Score of 0.5. But in a three outcome league, if you pick a random, you'd expect to get a Brier Score of about 0.666. In fact, the NHL Brier Score is somewhat better than the random result at 0.5, basically 0.59, so a reasonable distance away. Although again, the caveat we really need something to compare it with before we can make a judgment about how reliable this Brier Score is. There we have an examples of how we calculate Brier Scores and the idea behind them. Which as I said, we're going to really put to work in next week when we generate our own forecasts for league results, game results and compare the accuracy of our forecasts with the Brier Scores derived from the bookmaker odds.