Introduction and Initial Approach
I’m currently working on training an LLM that explains the key ideas in a chess position. Over the years, frontier LLMs have gotten decent at playing moves. However, as a chess learner, I care more about the explanations and rationales behind the moves, which are often wrong: they mislabel pieces on the board, highlight irrelevant ideas, and worst of all, provide no insightful analysis.
The first step for building my model was finding a dataset with high quality chess explanations. While there is an abundance of chess games available online, data for chess explanations is hard to come by.
I had no choice but to somehow get a frontier LLM to generate high quality chess explanation data. I picked Gemini Flash 3.5 over other models because it had the best baseline chess knowledge. But getting explanations I could actually use was much harder than expected.
Trying to Benchmark Failures
My initial approach was to quantitatively measure explanation quality, so I could define my problem concretely. There is no benchmark that measures the natural language description of a chess position and the faithful rationalization of moves. Existing benchmarks simply grade the moves that a model produces, which is trivial. Grading free form text, like explanations, is challenging.
My first idea was to use a less powerful LLM to extract claims that Gemini made about
the position into structured text, and then parse that text and evaluate it with a
chess library. For example, if Gemini said
The bishop on g2 attacks the knight on b7, I would use a less powerful LLM to
convert that into parsable claims like
exists("bishop", "g2"), exists("knight", "b7"), attacks("bishop", "knight"), and
then verify those claims.
But, when reading through the explanations, I realized that Gemini got its basic, easy to check facts right. It could summarize the position on the board and name pieces correctly. Its main failure modes were reasoning about future moves and prioritizing the position’s most relevant ideas for a human player.
For example, Gemini produced
This is the most forcing and strongest move in the position, immediately putting the Black king on the run. It clears the path for the queen.
Converting this into gradable boolean claims is difficult: I can’t easily verify
putting the Black king on the run using if statements, and
clears the path for the queen is vague. I also can’t easily verify if those ideas
are accurate or even relevant to the position. Essentially, the claims that were easy
to grade were mostly correct, but the reasoning and justification claims that I cared
about were non-gradable.
Deriving Context from First Principles
Instead of trying to correct false statements generated by Gemini, I thought about deriving the answer from first principles, assuming Gemini had no chess knowledge. The first step is to visually evaluate the position by counting the pieces on the board and observing their structure. This is called a static evaluation.
A traditional chess engine starts with a static evaluation, then performs a search by efficiently going through millions of positions to find the best future state. Naturally, my first step was to compare the static evaluation of a traditional chess engine like Stockfish to Gemini’s description of the position.
Stockfish simply produces a number indicating what side is winning, and its magnitude. For example, an evaluation of -2.0 means that Black is winning by 2 pawns, which is considered a decent advantage. To calculate the static evaluation, Stockfish currently uses a neural net, making it difficult to break down the atomic components of the evaluation. Luckily, older versions of Stockfish used a human tuned static evaluator (HCE). HCE scores piece types (knights, bishops, rooks) and features like king safety and mobility.
Cataloging Strengths and Weaknesses
I visually graded Gemini’s initial positional explanation against the HCE features and noticed that the two correlated highly. Gemini was already great at evaluating positions statically: its qualitative assessment roughly matched the numerical scores in each category.
This suggested a useful distinction. Gemini could describe positions whose evaluation was visible from the board, but might struggle when the evaluation only emerged after calculating a sequence of moves.
In chess, each position can be classified as “quiet” or “sharp”. Quiet positions have at least two good moves and multiple future ideas, and the position doesn’t change significantly in the short term. Sharp positions have a precise sequence of moves that completely changes the position in the short term. Stockfish is perfect at solving quiet and sharp positions: it can visually evaluate a quiet position, and perform an exhaustive search to find the correct sequence of moves in a sharp position.
LLMs, being naturally good at describing and summarizing data, are also naturally good at solving quiet positions. However, they do not possess the chess specific machinery to perform an exhaustive search like Stockfish. Therefore, they often hallucinate in sharp positions and miss the key idea of the position.
Fortunately, I had stumbled upon a heuristic for classifying a position as sharp or quiet: comparing the search and static evaluation. If the static evaluation is similar to the search evaluation, the position doesn’t change significantly in the short term, making it a quiet position. On the other hand, if the static evaluation differs from the search evaluation, there exists a specific sequence of moves that transforms the position immediately, making it a sharp position.
Now, I could effectively predict whether the LLM would describe a position correctly based on the position’s search and static evaluations. LLMs worked out of the box for quiet positions, but they needed the right guidance and context for sharp positions.
I came across Understanding Reasoning from Pretraining to Post-Training, which gave me a useful way to think about this gap. The paper pretrains models on human chess games, then post-trains with synthetic reasoning traces to teach them how to calculate chess positions.
I don’t know if Gemini underwent similar chess-reasoning post-training. But from what I saw, it felt like Gemini lacked that capability. It had enough knowledge to evaluate many positions statically, yet struggled when it had to calculate a precise variation. The paper’s solution was to generate reasoning traces by rolling out many possible continuations before selecting a move.
But a broad reasoning trace is not necessarily a useful chess explanation. Human players do not need every plausible calculation; they need the critical line and the natural responses that reveal why the idea works. If I could give human-relevant lines to Gemini, maybe I could compensate for its missing calculation at test time.
Generating a Human-Relevant Reasoning Trace
A human-relevant reasoning trace builds on a strength Gemini shares with human players: static evaluation. Like Gemini, human players can often understand a quiet position from the board’s visible features alone. The difference is that human players can calculate through the sharp positions, and chess books and commentary make that calculation explicit. They focus on the critical sequence of moves that resolves the sharp position into a favorable, quiet one, and include refutations to natural opponent responses.
Drawing on this structure, my next idea was to give the LLM the same compact sequence of moves. Essentially, I needed a way to generate:
- The main sequence of moves, or the “main line” of the position:
19. Rxe7+ Qxe7 20. Re1(highlighted in red) - Any relevant sequence of moves, or “side lines”, that address natural opponent
responses:
19...Kxe7 20. Qe3+ Kf8 21. Qxa7(highlighted in orange).
These lines give a human reader what they need to understand the position. I could then ask the LLM to verbalize:
- The main idea of the position:
An attraction draws a piece, usually the king, to a square leaving it vulnerable to another tactical idea. - Why the main idea works:
If Qxe7 as a response, we have Re1, pinning the queen to the king and winning materialandIt's important that the knight defends the rook or the queen can take it with mate! - Why another natural refutation fails:
If Kxe7, we have a follow-up fork, with Qe3+ winning the undefended a7 rook.
To generate a human-relevant version of the rollout inspired by the paper, I went back to static and search evaluations. Rather than sampling many possible continuations, my version of the rollout played the best moves until the static and search evaluations matched. The resulting line carries the position through its sharp, tactical phase into a quieter position that Gemini can evaluate and explain reliably.
I hit one snag: the best Stockfish moves were not always the moves a human player would actually consider. Stockfish often produces theoretically perfect, “alien-level” responses that make the explanation less useful to a human reader.
To resolve this, I used Maia, a neural network built to generate human-like chess moves. At every step, I compared Maia’s move to Stockfish’s. If their evaluations were roughly similar, I used Maia’s move in the main line. If they differed, I kept Stockfish’s theoretically correct move in the main line and added Maia’s move as a side line.
This gave the rollout the structure of a good chess explanation: follow the best line with human-like moves while addressing the natural responses a human reader would expect.
For the position above, the generated rollout is:
19. Rxe7+ Qxe7 ( 19... Kxe7 20. Qe3+ Kf8 21. Qxa7 ) 20. Re1 Qxe1+ 21. Nxe1. This
matches the human picked moves almost perfectly: the side line is an exact match, the
main line matches perfectly with two extra moves. The implementation can be found in
this GitHub repo.
Verifying a Compact, Human-Relevant Trace
The paper’s reasoning traces are built for broad search: for each puzzle, the authors sample eight first-move candidates, then 2–10 continuations from each candidate, with each continuation running for 1–20 moves. My rollouts had to be selective: they had to surface the lines a human reader would care about without burying Gemini in unrelated continuations.
I still needed to verify that this procedure did not explode into too much context. So I generated rollouts for ~2000 positions, with 50% sharp positions and 50% randomly sampled positions. Sharper positions had 8 moves on average, while quieter positions had 3, and the combined average was 5 moves. There was also limited branching, with only 1.2 lines per position on average. Most of the time, Stockfish and Maia chose similarly evaluated moves, so the rollout simply followed Maia’s more human-like move.
I also clicked through the rollouts to check whether I would have chosen the same lines myself. After some tuning, I was satisfied that they captured the core idea and natural refutations without adding irrelevant calculation. In several cases, the rollout made the position clearer to me as well. I was confident that the rollout was the right context to surface, and could lead to the LLM explanation being genuinely insightful.
Mapping Observed Failures to Benchmarks
The rollout gave Gemini the right lines to explain, but it did not solve a separate problem: tracking the board accurately through those lines. The LLM can summarize a critical line, but it can’t confidently make a specific claim about a future board state like “the knight defends the rook or the queen can take”. This is a well-documented failure mode, visible both in ChessQA, a chess benchmark, and the thinking traces of models.
In ChessQA, one of the five tasks is solving short tactical puzzles, which requires reasoning about future positions. It was the hardest category, with an average accuracy of 17.4%. ChessQA’s #1 “recurrent failure” family was board-state hallucination/misrecognition.
Looking at thinking traces about chess positions, it is clear that LLMs are ill-suited to tracking board state across positions. Below is an example:
Okay, hold on a second. I need to retrace this position. Where in the world *is* that black rook? This is key. Alright, let's see... the move was 31. Bxg6. Okay, so that means a bishop took something on g6. Did the bishop capture the rook specifically? No, not necessarily. The bishop was on d3, and it captured on g6. Right. Now, where *was* the rook before that? Did I miss something? Wait, is the black rook actually on d6?
When evaluating LLMs, ChessQA turned off agentic functionality like tool calling and code execution. I proceeded to design tools specifically to address board-state hallucination and the clunky spatial reasoning observed in thinking traces.
Giving Tools to Bridge the Gap
My starting point was to go to ChessQA’s repository and convert all of their verifiers to tool calls for Gemini. If an LLM is observed to be bad at something, it should “cheat” its way to a correct answer using a tool call. I also experimented with different prompts to compel Gemini to use tools instead of guessing.
The fix for board-state hallucination was a tool called get_piece_positions(moves),
which provides piece arrangements given a future state. To enhance Gemini’s spatial
reasoning about future states, I introduced check_motif(moves, motifs, square).
I implemented the same motifs that ChessQA checked for: pin, skewer, fork,
battery, discovered_check, double_check. These motifs are widely recognized and
standard across chess literature. I rounded out my motifs with hanging: the
thinking traces showed that the LLM could not detect hanging pieces.
After some prompt and context engineering, I was able to get Gemini Flash 3.5 to generate accurate, high quality, and insightful chess explanations.
