{"id":437506,"date":"2023-02-25T12:00:00","date_gmt":"2023-02-25T11:00:00","guid":{"rendered":"https:\/\/innovationorigins.com\/?p=437506"},"modified":"2023-02-25T12:00:00","modified_gmt":"2023-02-25T11:00:00","slug":"chatgpt-cant-play-chess-and-heres-the-reason-why","status":"publish","type":"post","link":"https:\/\/ioplus.nl\/archive\/en\/chatgpt-cant-play-chess-and-heres-the-reason-why\/","title":{"rendered":"ChatGPT is great, just not at chess"},"content":{"rendered":"\n

ChatGPT<\/a>  is all the rage right now. So much so that we seem to forget what it actually is: a language model. And that means it isn\u2019t suited for all kinds of tasks that other machines are already capable of. Like playing chess, for one thing. <\/p>\n\n\n\n

ChatGTP’s performance in terms of language is undoubtedly impressive. But that doesn’t mean the system can do anything. Playing chess against ChatGPT is nothing short of hilarious. This Open AI\u2019s Large Language Model <\/em>(LLM) just can\u2019t seem to play either coherently or even abide by the rules, let alone play well.<\/p>\n\n\n\n

An entertaining example is YouTuber and International Master Levy Rozman\u2019s (GothamChess<\/a> on YouTube) game<\/a> against the AI, in which ChatGPT played a sequence of absurd illegal moves, among other shenanigans. It captured its own bishop after three moves, generated material out of thin air and even resorted to moving its opponent\u2019s pieces in the endgame. On some occasions, after an illegal move, when Rozman prompted it to do so, ChatGPT played another move, most of which were just as outlandish and illegal as the previous moves.<\/p>\n\n\n\n

Similarly, Chessily.com<\/a> founder Marc Cressac<\/a> had Stockfish 15.1 – the most advanced chess AI ever developed – play against ChatGPT, which he described in an online article<\/a>. As far as fun was concerned, the results did not disappoint. ChatGPT probably doesn\u2019t find much delight in following strict rules. Maybe, after 1500 years, the game of chess needs a refreshing spin and ChatGPT is here to provide it. A coup d\u2019\u00e9tat <\/em>on one of the most popular games in human history, dethroning the authority of its set of rules<\/a> and revolutionizing the status quo.<\/p>\n\n\n

<\/a>
<\/div><\/div>
Brain training system raises performance levels of gamers<\/div>
For a professional gamer, just one percent of an improvement in their performance can worth millions of dollars. However, there is still no widespread tool for eSports players to measure and help them improve their performance.<\/div><\/div><\/div>\n\n\n

How ChatGPT approaches chess<\/h2>\n\n\n\n

Jokes aside, let\u2019s take a step back and try to understand how ChatGPT actually plays chess. The program, being a Large Language Model –<\/em> meaning an artificial intelligence that can only produce information through written language in response to a prompt – is only able to play by using the algebraic notation system<\/a>. What is that? It is a system that identifies how one of the 32 pieces on the 8 by 8 board can move, assigning a capital letter to indicate a piece (excluding pawns, which are only identified by the square they\u2019re moving to), followed by a lower-case letter and a number which individuate one of the 64 squares on the board. Here is an example.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n
One of the two Knights is moved (“Ng1 -> f3” or “Nf3”)<\/em><\/strong><\/figcaption><\/figure>\n\n\n\n

White\u2019s Knight on the g1 square moved to the f3 square. In algebraic notation, this can be expressed as \u201cNg1 -> f3\u201d or just \u201cNf3\u201d as, in this particular position, the f3 square can only be legally reached this way by one of the two Knights.<\/p>\n\n\n\n

Generally, this is the method ChatGPT can resort to when playing chess, since it does not have access to a built-in board. Although people may think that such a sophisticated Language Model could also be capable of playing this classic game. Moreover, average chess players and grandmasters alike have come to terms with getting categorically crushed by unbeatable AIs, like the all-powerful Stockfish, with no chance of even reaching a draw.<\/p>\n\n\n\n

So, ChatGPT\u2019s chess performances should technically be amazing, right<\/em>? Well, not necessarily. Now, you may think that it just hasn\u2019t been specifically trained to play chess. While this is certainly true, it\u2019s not that simple. In prder to clear this up, we need to understand how a Large Language Model works.<\/p>\n\n\n\n

LLMs<\/h2>\n\n\n\n

\u201cEssentially, an LLM can tell you what is the probability that a bunch of words appears in a [given] sequence,\u201d explains Evangelos Kanoulas, Professor at the University of Amsterdam\u2019s Informatics Institute, \u201cso it can tell you how likely a piece of text is.\u201d Moreover, words from natural language are converted to tokens, where every word corresponds to 1 to 3 tokens. Also, Prof. Kanoulas goes on to say, an LLM can use conditional probability<\/em> to associate words, meaning that on the basis of a given word it can \u201cguess\u201d what the next word (or token) will be. So, the larger the Language Model is, the more it can give a good estimate of what the next word will be.<\/p>\n\n\n\n

\u201cLet\u2019s imagine that this machine has been trained by looking at how humans use words,” Prof. Kanoulas continues. “Most likely it has also seen chess games. So, someone writes \u2018Chess game, player one\u2019 and then the first move, [for example] d4. So now the AI learns that when a game starts, there is a good chance that the first move is d4, because it read it somewhere. [ChatGPT] has no idea about what this is. It doesn\u2019t know what chess is and doesn\u2019t know the rules\u201d. Basically, the AI analyses the language, recognizes the words which indicate the start of a chess game and replies with a probabilistic guess based on what is likely to be an appropriate answer to a specific prompt.<\/p>\n\n\n\n

\"\"
The opening move \u201cd4\u201d<\/strong><\/em><\/figcaption><\/figure>\n\n\n\n

Linguistic predictions<\/h2><\/div>\n\n\n\n

Also, the machine is capable of making generalizations. This is to say that it can produce a sequence of words even though it has never seen a specific prompt. Maybe it has seen similar or correlating ones and it draws information from them. So, with no concept of the rules of chess, and having possibly seen a good number of chess games with differing first moves, it produces a generalized answer, which can result in our \u201cd4\u201d example. However, this guess must not be confused in terms of its nature. It is not a computational analysis of all the games the AI has seen, but only a linguistic prediction.<\/p>\n\n\n\n

In other words, ChatGPT does not play \u201cd4\u201d because it knows that it is a viable move, but only because the popularity of the move in chess games played by humans has made the text \u201cd4\u201d appear numerous times after the start of a game. \u201cOne thing that it knows,” – Prof. Kanoulas points out, “is that, in this case, a letter is probably not followed by a number, so it may not do \u201cdd\u201d. But it could. Maybe something weird happens, like a human playing an illegal move that breaks the probability of what the AI will do next. What will come out of that is something strange.\u201d<\/p>\n\n\n\n

What does the AI know?<\/h2><\/div>\n\n\n\n

Mistakenly, some may think that ChatGPT may be aware of the fact that the playing field is an 8 by 8 checkered board. However, it does not even know that. Again, the reason why, while playing, ChatGPT is not likely to go past the letter \u201ch\u201d and the number 8 is not because it knows it cannot do that. Rather, it has never, or a very small number of times, seen anyone typing a configuration of the text related to a chess move going past \u201ch\u201d or 8.<\/p>\n\n\n\n

Nevertheless, it is still very much capable of generating mistakes. For example, in Rozman\u2019s YouTube video mentioned earlier, ChatGPT, after capturing its own bishop by an illegal castling<\/a> move, moved its Knight from the b1 square to the c3 square, and found itself in the position shown on the following board.<\/p>\n\n\n\n

\"\"
The configuration of the board during Rozman\u2019s game against ChatGPT<\/em><\/strong><\/figcaption><\/figure>\n\n\n\n

For this move, the AI provided this comment: \u201cI\u2019ll play Nc3 [Knight to c3], attacking your bishop and developing a piece\u201d. While the move definitely develops (in chess-speak, \u201cadvances\u201d or \u201cmakes a given piece able to move further\u201d) the Knight, in no way does it attack one of Rozman\u2019s Bishops. ChatGPT may have supplied this line of commentary because Knights often tend to attack Bishops, just not in this position and, normally, only when it is fair and legal to do so within the rules.<\/p>\n\n\n\n

Not bound by hardcore game rules<\/h2><\/div>\n\n\n\n

Here lies the whole premise: ChatGPT\u2019s training and learning process is not bound by hardcore game rules, like a chess engine is. Steven Abreu, MSc, a researcher from the University of Groningen (the Netherlands), clarifies that what works well for predicting the next token in language is to “keep many options open\u201d, and then to select \u201cthe most likely option\u201d. But an LLM model does not learn to invalidate some options, it just assigns a lower probability to those which tend to appear less.<\/p>\n\n\n\n

Ultimately, all things considered, is it accurate to state that ChatGPT actually plays<\/em> chess? In a strict sense, no. ChatGPT responds to language that we, as humans, understand to be related to the game, without any concept of the game being played.<\/p>\n\n\n\n

Will ChatGPT ever be good at chess?<\/h2>\n\n\n\n

Both Prof. Kanoulas and Abreu believe that, if allowed the time needed, ChatGPT could learn to play chess legally, or even well. \u201cI believe that LLMs can be trained to play chess properly\u201d Abreu states. According to him, this could be done through Reinforcement Learning through Human Feedback<\/em>, a process which ChatGPT was trained by, where humans annotate which responses to a question are most suitable and ChatGPT then gradually learns to become better at having conversations, Abreu goes on to explain. Alternatively, the AI could be fine-tuned on existing chess games in order to predict the best moves. Or again, one could train ChatGPT to play chess like a chess engine. The gist is: there are many ways this could happen. <\/p>\n\n\n\n

In this regard, Prof. Kanoulas says that at this stage ChatGPT may not play as well as a chess AI does, because it \u201cknows nothing about rewards, it knows nothing about what it means to win [for example] the Queen\u201d. But if it sees enough times that, in particular occasions, people tend to take the Queen, it will also take it. \u201cProbabilistically speaking, it would kind of understand – not because it can calculate the reward itself, but because the humans did that in their head and the chess AI did in its system – that this is more likely to happen\u201d. In short, it doesn\u2019t know why it has to take the Queen, nor how that\u2019s going to affect the future, as the AI knows nothing about winning. Or losing, for that matter. It is just a high-probability response.<\/p>\n\n\n\n

Still, the conversation remains open as new technologies and scientific methods are rolled out in the artificial intelligence field. Concepts like \u201cneurosymbolic knowledge<\/em>\u201d (a matter for another time!) could potentially revolutionize machine learning. Or maybe not. Time – and research – will tell. For now, we get to laugh at its incapacity to elaborate on tactics and calculations.<\/p>\n