PR/ER

When analyzing a match, the bots will calculate and display a value called “PR” or “ER”. These acronyms are interchangeable, standing for either Performance Rate or Error Rate. A PR of 0 means you played “perfectly” as assessed by the bot, both in checker play and cube decisions. Grandmasters play with an average PR of ~3, and most newbies will play around 15-20. As a rule of thumb, you want to make plays that lower your PR to improve your play.

The PR represents the average equity lost per decision in a backgammon match, as assessed by the bots. The best bots play better than any humans today, and so to improve our play, we should strive to imitate them as best as we can. Doing so is equivalent to lowering our PR. This take could be considered slightly controversial, however: some players doubt that the bots really are that strong; alternatively, some believe that the bots have specific weaknesses in their play that can be exploited (such as backgames with large outside primes); even among players that trust the bots, the idea that a single quantity can fully represent a player’s skill negates any “play style” interactions. One can imagine a “rock-paper-scissors” scenario (or, perhaps in keeping with our theme, a “fire-grass-water” scenario) where a player with a lower PR is dominated by others with higher PRs that know how to capitalize on their weaknesses.

Despite these objections, here we will operate under the assumption that PR indeed measures the skill of a player and that a lower PR represents “better” backgammon play. Therefore, we will use it as a tool to monitor our progress while we are learning how to play. At our current level of play, we have so much to learn that none of these concerns apply to us.

How is PR computed?

First, where do these numbers come from? Each of the most popular bots produces slightly different PR values for a few reasons. First, as they are based on different neural nets, they produce different equities for the same positions. In addition to this, they use slightly different equations to calculate PR from the equity. For example, XG computes PR as:

PR = total equity lost × 500 / number of decisions

and for GnuBG:

ER = total equity lost × 1000 / number of decisions

Finally, each bot has slightly different criteria for what it counts as “a decision.” This sounds more cryptic than it really is; mostly, it’s a consequence of how challenging it is for a bot to determine whether it counts as a decision when a player chooses not to double. There’s no “standard” definition of a decision.

All this to say: you can’t compare the PR from different bots. Your PR could be 8 on XG and 11 on GnuBG, etc. If you are using PR to monitor your progress, you should stick to tracking using the same bot.

PR and winning

It’s been shown countless times that a lower PR correlates with more wins in unlimited games. In a recent example, Joseph Heled published a study in a video comparing a dozen bots with different PRs, concluding that lower PR bots win more games. This can be verified relatively easily using your own software at home, and it makes intuitive sense. If you believe that the bot is the best player, you should win more games the more you play like it.

What about the exact value of a PR? What does it really mean to have a PR of 5 or 10? When playing for money, someone who plays with an average PR of 10 would lose 2 cents on the dollar every play when playing against XG. In a more realistic scenario, consider 2 players with a PR difference of 3 (e.g., a PR 11 player and a PR 8 player). If a typical unlimited game has an average of 35 decisions and these 2 players play 10 games, at the end of the session, the lower PR player should expect to be 35 × 10 × 3 / 500 = 2.1 points ahead. Hopefully, this exercise illustrates how small the PR advantage turns out to be, or conversely, how much luck plays a factor in the outcome of a game of backgammon.

When considering matches, there are equations for how to extrapolate the equity loss to match-winning chances. This was computed and published in this lovely table and figure below¹:

This data shows some very reasonable trends, notably that a larger PR difference and a longer match length favour the lower PR player. Still, luck plays a significant role in even the longest matches. Consider a 25-point match where the 2 players’ PR differ by just 3. The table gives the stronger player only a 64/36 edge in any given match. Considering that the Backgammon World Championship consists of single elimination 19-point matches, it should not be too surprising that the players with the lowest PRs (currently around 2.5) don’t often make it to the round of 8.

Improving your PR

Finally, let’s discuss how much work it takes to reduce your PR. For simplicity, let’s assume all errors have the same magnitude. For example, suppose we only make 8-cent blunders (i.e., we only lose equity -0.080 at a time). We can build a model for PR based on this assumption, where we can only lower our PR by reducing the frequency of our blunders. Given this model, it’s trivial to figure out how many blunders we need to eliminate to achieve a certain level of improvement. We plot this result below:

This figure presents the fraction of errors you need to prevent in order to reduce your PR by one. When starting, with a PR of 20, you just need to eliminate 5% of your errors to bring your PR down to 19. When your PR is 10, you need to eliminate 10% of your remaining errors to come down to 9. Therefore, it’s twice as hard to see improvements at that latter stage. (In fact, it’s likely more than twice the challenge, as the typical blunders that are left over are tougher to iron out.) Therefore, it’s typical for new players to see quick improvement that then slowly tapers off as they get better.

1. From what I have been able to find, Wayne Joseph generated this data for a presentation, but doesn’t remember exactly how he did it. It would be nice if we could recreate this using simulation data, or real match data from one of the popular online gaming websites. ↩︎