Introduction and Initial Approach
I’m currently working on training an LLM that explains the key ideas in a chess position. In my previous post, I surveyed the literature on distilling chess explanations from a frontier model into a smaller one, and concluded that distillation was ill-suited for chess reasoning. The more important realization came from staring at the teacher’s outputs along the way.
Over the years, frontier LLMs have gotten decent at playing moves. However, as a chess learner, I care more about the explanations and rationales behind the moves, which are often wrong: they mislabel pieces on the board, highlight irrelevant ideas, and worst of all, provide no insightful analysis.
The teacher’s own explanations were bad, so there was nothing worth distilling. I went back to basics: train a model directly, on a dataset of explanations worth learning from. While there is an abundance of chess games available online, data for chess explanations is hard to come by.
I scoured the internet for an existing dataset and came up empty. The closest thing I found was ChessBase’s MegaDatabase, which was unusable: the database is a licensed product stored in ChessBase’s own file format, and its annotations are game commentary rather than structured explanations of positions.
Synthetic data was my only option. I picked Gemini Flash 3.5, the best teacher model to come out of my previous post. The goal this time was not to copy Gemini, but to get Gemini to produce high-quality chess explanations.
Trying to Benchmark Explanation Quality
My initial approach was to quantitatively measure explanation quality, so I could define my problem concretely. There is no benchmark that measures the natural language description of a chess position and the faithful rationalization of moves. Existing benchmarks grade the moves that a model produces: I did the same in my previous post. Grading free form text, like explanations, is challenging.
My first idea was to use a less powerful LLM to extract claims that Gemini made about the
position into structured text, and then parse that text and evaluate it with a chess
library. For example, if Gemini said The bishop on g2 attacks the knight on b7, I would
use a less powerful LLM to convert that into parsable claims like
exists("bishop", "g2"), exists("knight", "b7"), attacks("bishop", "knight"), and then
verify those claims.
But, when reading through the explanations, I realized that Gemini got its basic, easy to check facts right. It could summarize the position on the board and name pieces correctly. Its main failure modes were reasoning about future moves and prioritizing the position’s most relevant ideas for a human player.
For example, Gemini produced
This is the most forcing and strongest move in the position, immediately putting the Black king on the run. It clears the path for the queen.
Converting this into gradable boolean claims is difficult: I can’t easily verify
putting the Black king on the run programmatically, and clears the path for the queen is
vague. I also can’t verify if those ideas are accurate or even relevant to the position.
Essentially, the claims that were easy to grade were mostly correct, but the reasoning and
justification claims that I cared about were non-gradable.
Deriving Context from First Principles
Instead of trying to correct false statements generated by Gemini, I thought about deriving the answer from first principles, assuming Gemini had no chess knowledge. The first step is to visually evaluate the position by counting the pieces on the board and observing their structure. This is called a static evaluation.
A traditional chess engine starts with a static evaluation, then performs a search by efficiently going through millions of positions to find the best future state. Naturally, my first step was to compare the static evaluation of a traditional chess engine like Stockfish to Gemini’s description of the position.
Stockfish simply produces a number indicating what side is winning, and its magnitude. For example, an evaluation of -2.0 means that Black is winning by 2 pawns, which is considered a decent advantage. To calculate the static evaluation, Stockfish currently uses a neural net, making it difficult to break down the atomic components of the evaluation. Luckily, older versions of Stockfish used a human tuned static evaluator (HCE). HCE scores piece types (knights, bishops, rooks) and features like king safety and mobility.
Cataloging Strengths and Weaknesses
I manually graded Gemini’s positional explanations against the HCE features for a few dozen positions and noticed that the two correlated highly. Gemini was already great at evaluating positions statically: its qualitative assessment roughly matched the numerical scores in each category.
This suggested a useful distinction. Gemini could describe positions whose evaluation was visible from the board, but struggled when the evaluation only emerged after calculating a sequence of moves. That matched the failure mode I had seen earlier: Gemini named the pieces correctly and summarized the board well, but its claims about future moves were often incorrect. In engine terms, Gemini had a solid static evaluation and no search.
In chess, each position can be classified as “quiet” or “sharp”. Quiet positions have at least two good moves and multiple future ideas, and the position doesn’t change significantly in the short term. Sharp positions have a precise sequence of moves that transforms the position and shifts the balance decisively. Stockfish handles both: it can statically evaluate a quiet position, and it can search exhaustively to find the critical sequence in a sharp one. Gemini handles only the quiet ones.
Fortunately, I had stumbled upon a heuristic for classifying a position as sharp or quiet: compare Stockfish’s static evaluation to its search evaluation. If the two agree, the position is quiet. If they disagree, the search has found something the board alone doesn’t show, and the position is sharp.
Now, I could effectively predict whether Gemini would describe a position correctly based on the position’s search and static evaluations alone.
I came across Understanding Reasoning from Pretraining to Post-Training, which gave me a useful way to think about this gap. The paper splits chess ability into two stages. Pretraining on human games teaches a model what positions look like and which moves are typical. Post-training on synthetic reasoning traces, generated by rolling out many possible continuations of a position, teaches the model to calculate. That split lined up with what I had been seeing: whatever chess Gemini absorbed from the internet looked like the first stage. It was unclear whether Gemini had been post-trained on chess reasoning, but from what I saw, Gemini lacked it.
But a broad reasoning trace is not necessarily a useful chess explanation. Human players do not need every plausible calculation; they need the critical line and the natural responses that reveal why the idea works. If I could give human-relevant lines to Gemini, maybe I could compensate for its missing calculation at test time.
Generating a Human-Relevant Reasoning Trace
Human players calculate through sharp positions, and chess books and commentary make that calculation explicit. They focus on the critical sequence of moves that resolves the sharp position into a favorable, quiet one, and include refutations to natural opponent responses.
I needed a way to generate the same compact structure:
- The main sequence of moves, or the “main line” of the position:
19. Rxe7+ Qxe7 20. Re1(highlighted in red) - Any “side lines” that address natural opponent responses:
19...Kxe7 20. Qe3+ Kf8 21. Qxa7(highlighted in orange).
Given those lines, I could ask Gemini to verbalize:
- The main idea of the position:
An attraction draws a piece, usually the king, to a square leaving it vulnerable to another tactical idea. - Why the main idea works:
If Qxe7 as a response, we have Re1, pinning the queen to the king and winning materialandIt's important that the knight defends the rook or the queen can take it with mate! - Why another natural refutation fails:
If Kxe7, we have a follow-up fork, with Qe3+ winning the undefended a7 rook.
To generate these lines, I went back to static and search evaluations. Rather than sampling many possible continuations as the paper did, my version of the rollout played the best moves until the static and search evaluations matched. The resulting line carries the position through its sharp, tactical phase into a quieter position that Gemini can evaluate and explain reliably.
I hit one snag: the best Stockfish moves were not always the moves a human player would actually consider. Stockfish often produces theoretically perfect, “alien-level” responses that make the explanation less useful to a human reader. To resolve this, I used Maia, a neural network built to generate human-like chess moves. At every step, I compared Maia’s move to Stockfish’s. If the two moves’ evaluations were roughly similar, I used Maia’s move in the main line. If they differed, I kept Stockfish’s theoretically correct move in the main line and added Maia’s move as a side line.
For the position above, the generated rollout is
19. Rxe7+ Qxe7 ( 19... Kxe7 20. Qe3+ Kf8 21. Qxa7 ) 20. Re1 Qxe1+ 21. Nxe1. The side line
is an exact match with the commentary; the main line matches with two extra moves. The
implementation can be found in this GitHub repo.
The paper’s reasoning traces are built for breadth: eight first-move candidates, several continuations per candidate, each run for a handful of moves, which works out to 300 moves of calculation per position. My rollouts had a much tighter budget. I needed sharp positions to resolve within 5 to 15 moves, the range I had seen in books and enough context to explain a position without projecting too far into the future. I also needed the Stockfish/Maia divergence to produce only a few side lines rather than a branching tree.
To verify this, I generated rollouts for ~2000 positions, with 50% sharp positions and 50% randomly sampled positions. Sharper positions had 8 moves on average, while quieter positions had 3, and the combined average was 5 moves. There was also limited branching, with only 1.2 lines per position on average. Most of the time, Stockfish and Maia chose similarly evaluated moves, so the rollout simply followed Maia’s more human-like move.
I also clicked through the rollouts to check whether I would have chosen the same lines myself. After some tuning, I was satisfied that they captured the core idea and natural refutations without adding irrelevant calculation. In most cases, the rollout made the position clearer to me as well.
Mapping Observed Failures to Benchmarks
The rollout gave Gemini the right lines to explain, but it did not solve a separate problem: tracking the board accurately through those lines. The LLM can summarize a critical line, but it can’t confidently make a specific claim about a future board state like “the knight defends the rook or the queen can take”. This is a well-documented failure mode, visible both in ChessQA, a chess benchmark, and the thinking traces of models.
In ChessQA, one of the five tasks is solving short tactical puzzles, which requires reasoning about future positions. It was the hardest category, with an average accuracy of 17.4%. ChessQA’s #1 “recurrent failure family” was board-state hallucination/misrecognition.
Looking at thinking traces about chess positions, it is clear that LLMs are ill-suited to tracking board state across positions. Below is an example:
Okay, hold on a second. I need to retrace this position. Where in the world *is* that black rook? This is key. Alright, let's see... the move was 31. Bxg6. Okay, so that means a bishop took something on g6. Did the bishop capture the rook specifically? No, not necessarily. The bishop was on d3, and it captured on g6. Right. Now, where *was* the rook before that? Did I miss something? Wait, is the black rook actually on d6?
When evaluating LLMs, ChessQA turned off agentic functionality like tool calling and code execution. I proceeded to design tools specifically to address board-state hallucination and the clunky spatial reasoning observed in thinking traces.
Giving Tools to Bridge the Gap
My starting point was to go to ChessQA’s repository and convert all of their verifiers to tool calls for Gemini. If an LLM is observed to be bad at something, it should “cheat” its way to a correct answer using a tool call. I also experimented with different prompts to compel Gemini to use tools instead of guessing.
The fix for board-state hallucination was a tool called get_piece_positions(moves), which
provides piece arrangements given a future state. To enhance Gemini’s spatial reasoning
about future states, I introduced check_motif(moves, motifs, square).
I implemented the same motifs that ChessQA checked for: pin, skewer, fork, battery,
discovered_check, double_check. These motifs are widely recognized and standard across
chess literature. I rounded out my motifs with hanging: the thinking traces showed that
the LLM could not detect hanging pieces.
After extensive prompt and context engineering, Gemini Flash 3.5 began producing explanations I would trust as a chess player. Below is a representative one.
Closing Thoughts
As I noted at the start, no benchmark measures explanation quality. So I spent weeks reading agent traces, grading explanations by hand, and reworking the prompt and context until the output held up to my own standards as a chess player. The pipeline combines every lever I found: rollouts to supply the calculation Gemini lacks, tools to plug the failure modes ChessQA documented, and a prompt tuned against hundreds of traces. I do not see another technique to add. These are the best chess explanations I can get out of a frontier model.
Future Work: Owning the Intelligence
A fair question at this point is why train a model at all. If the pipeline already produces good explanations, why not serve those directly?
Generating explanations are slow and expensive. Each position costs 7 to 10 cents and can take a couple of minutes, most of it spent round-tripping tool calls. When I described the setup to Rakshit, the founder of Chessiro, his reaction was that the latency was unusable for a product and that even one cent per position was more than he could spend. As a chess player, I know what the bar is: Stockfish answers in seconds, runs in a browser tab, and is free.
The entire pipeline is fragile. It rests on prompt and context engineering against a model I do not control; Google can change the weights under me and break everything.
The rollout and tools bolted chess-specific machinery onto a language model from the outside: search, board tracking, motif detection. That machinery already exists as a single neural network. Chess transformers like Leela play at a grandmaster level, carry search and spatial reasoning in their weights, and were built with interpretability in mind. Instead of probing a black box with prompts and hoping it cooperates, the next step is to graft Leela onto a small language model directly, so that a grandmaster-level explanation comes out of one forward pass.