Distilling Reasoning: Exploring the Boundaries

I’m currently working towards training a task-specific LLM for chess reasoning: a model that can identify the key ideas in a chess position and explain them clearly in natural language. As a chess learner, my goal is to build something that I can learn from, rather than a model that simply outputs the best move.

A natural first approach was knowledge distillation: using a stronger teacher LLM to transfer broad chess-reasoning ability to a smaller student. This was attractive because distillation can, in principle, recover much of a teacher’s capability with far less training data than conventional methods. To understand how effective and viable this approach might be for chess reasoning, I started with the literature.

What Does Distillation Actually Require

Hinton et al. (2015) introduced knowledge distillation as a way to train a smaller model with guidance from a larger, more capable one. The larger model acts as the teacher, and the smaller model as the student. In one of the paper’s most striking results, a student trained with the teacher’s outputs on only 3% of the original dataset recovered nearly all the teacher’s performance. The key was not merely having fewer examples, but having access to the teacher’s soft targets: its full probability distribution over possible outputs, rather than only a single correct label.

Hard target: just one label

BMWGarbage truck

100%

Teacher's soft targets: more learning signal

BMWGarbage truckCarrot

90%

9%

1%

A model’s full output distribution provides a richer training signal than a hard label alone. Consider a classifier shown an image of a BMW. The training example says only that the correct answer is “BMW.” A teacher, however, might assign 90% probability to BMW, 9% to another vehicle, and 1% to an unrelated object. Those lower-probability predictions still matter: they reveal which classes the teacher considers similar. By learning from that distribution, the student receives a more informative signal than it would from conventional supervised learning.

This initially appeared to settle an important design decision. If effective distillation required access to the teacher’s full output distribution, then I needed a teacher whose logits I could inspect. That constraint pushed me toward open-weight models and away from closed-weight frontier models. Kim and Rush (2016) complicated that conclusion.

Hinton et al. studied classification models. Kim and Rush, by contrast, focused on sequence generation, which was closer to my setting with language models. They showed that a student could achieve comparable performance to its teacher without access to the output distribution. Instead, they introduced sequence-level knowledge distillation: use the teacher to generate an entire target sequence, then train the student with ordinary supervised learning. Much of their method focused on constructing a strong training target from the teacher.

If a high-quality teacher-generated sequence was enough, then distillation no longer required access to the teacher’s logits. Closed-weight frontier models were back on the table as potential teachers.

Choosing the Models

I turned to ChessBenchLLM to identify strong closed-weight and open-weight candidates. The top of the leaderboard was dominated by closed-weight models, with the Gemini models clearly above the rest. For the open-weight side, I chose Gemma, partly because Google DeepMind’s models had a strong reputation for chess capability.

I chose Gemma 4 E2B as my initial student candidate. I wanted a model small enough to fine-tune affordably, and I figured a few billion parameters could be enough for a narrow domain like chess. I then evaluated the student and both teacher candidates on the same set of 100 randomly selected chess positions. Gemma 4 E2B had a mistake rate of 56%, compared with 29% for Gemma 4 31B and just 3% for Gemini Flash 3.5. The benchmark code and results are available here.

For context, human mistake rates are roughly 10–15% for beginners, 6–9% for intermediate players, 3–5% for advanced players, and 1–2% for super-grandmasters. By that scale, Gemini was playing at roughly an advanced-human level, while both Gemma models performed substantially worse than a beginner. My real target was reasoning and explanation quality rather than raw playing strength, so I treated move quality as a rough proxy rather than a definitive measure.

Gemini was clearly the stronger teacher candidate, but I didn’t know whether I could distill that level of performance into E2B using only teacher-generated sequences. Gemma 4 31B was much weaker, but its open weights gave me access to the teacher’s full output distribution. I kept both options open, to see whether Gemma 4 31B’s open access would win over Gemini’s superior performance.

With the basic design space in place and a few plausible teacher-student choices, I turned to the frontier of LLM knowledge distillation to find the right training method. One approach kept appearing across the recent literature: on-policy distillation.

On-Policy Distillation

On-policy distillation offers an elegant idea: instead of simply imitating the teacher, the student generates its own sequence and then learns how to correct it using the teacher’s output distribution. The “on-policy” part comes from reinforcement learning, where a model learns from trajectories generated by its current policy.

Distillation with access to the teacher’s logits, and therefore its full output distribution, is known as white-box distillation. Distillation from a closed-weight teacher, where only generated outputs are available, is known as black-box distillation. White-box OPD is much more established, while black-box OPD remains a newer area of research. In my setup, distilling Gemma would be a white-box problem, while distilling Gemini would be a black-box one.

White-Box OPD

For the white-box path, one of the foundational papers was MiniLLM. MiniLLM took the on-policy intuition and formalized it as a sequence-level distillation method: the student samples complete responses from its own distribution, while the teacher provides the learning signal for those responses. The key advantage was that the student could learn from its own sequences. When it generated a weak sequence, the teacher’s distribution showed how the model should have behaved instead, giving the student a way to correct its own output space.

The method showed strong results across several language-generation tasks, but implementing that idea introduced an important complication: once the student generates its own training data, optimizing the objective is no longer as straightforward as ordinary supervised learning.

In supervised learning, the training examples are fixed. Changing the model parameters changes how the model responds to those examples, but it does not change which examples are in the dataset. In an on-policy setting, the model parameters also affect the training examples: the dataset is derived from the model itself. In principle, calculating the exact gradient would mean accounting for every possible sequence the model could generate. For a language model, that space becomes combinatorially large almost immediately.

MiniLLM uses a trick from reinforcement learning: policy gradients. Rather than accounting exactly for every sequence the model could generate, the gradient is estimated from a manageable number of sampled sequences. That makes the problem tractable, but introduces other issues. As the MiniLLM authors note, “policy gradient suffers from high variance and reward hacking.” Correcting for these problems adds additional machinery to the objective and gradient calculation, making the final implementation considerably more complicated.

Generalized Knowledge Distillation (GKD) offered a much simpler way to preserve the core idea of on-policy distillation. The student still generates its own sequences and learns from the teacher on the mistakes it actually makes. The key difference from MiniLLM is that GKD uses a token-level objective instead of a sequence-level one. Once the student samples a sequence, GKD treats the sequence as fixed and compares the teacher and student distributions token by token. This allows it to avoid differentiating through the sampling process, removing the need for policy gradients and much of the stabilization machinery that came with them. The authors describe the result as closer to supervised learning than reinforcement learning, while still preserving the central benefit of training on the student’s own generations.

GKD gave me a practical white-box approach, but before implementing it, I wanted to understand on-policy distillation more deeply. That led me to Rethinking On-Policy Distillation, which analyzes GKD-style white-box OPD, and asks why some runs succeed while others fail.

The paper identifies two conditions for successful OPD. The first is that the teacher and student need compatible “thinking patterns.” In practice, this means their high-probability next-token distributions need enough overlap on the student’s own generations. Successful OPD runs showed the teacher and student becoming progressively more aligned on these high-probability tokens, while failing runs remained mismatched. Gemma 4 31B looked like a sensible initial teacher choice: a teacher from the same model family as E2B seemed like a likelier source of compatible distributions than an unrelated open-weight model.

The second condition was more problematic. The teacher needed to offer genuinely new capabilities beyond what the student had already learned. Gemma 4 31B was larger and substantially stronger than E2B, but both models shared the same lineage. Rethinking OPD directly challenged the idea that a larger version of the same model would make a good teacher. In its closest analogue to my proposed Gemma setup, distilling R1-Distill-1.5B from the stronger R1-Distill-7B produced only limited gains. A parallel Qwen experiment made the contrast clearer: an RL-post-trained 4B teacher transferred more effectively to the 1.7B student than the standard 4B teacher. This gave me a more specific picture of a good OPD teacher: it needed to be similar enough to the student for their distributions to be compatible, but different enough to have actually learned something new.

That weakened the case for Gemma 4 31B, but it made Gemini more interesting. Its much stronger chess performance gave me more reason to believe it had new capabilities to offer. The problem was that Gemini was closed-weight, so I could neither use GKD nor directly measure the distributional compatibility studied in Rethinking OPD. If I wanted to use Gemini as the teacher, I needed a black-box approach.

Black-Box OPD

Black-box on-policy distillation is a remarkably recent area of research. Generative Adversarial Distillation (GAD), one of the first methods to bring OPD to closed-weight teachers, appeared in late 2025, with Semi On-Policy Black-Box Distillation (SODA) following only months later in 2026. GAD starts from the central problem I had with Gemini: without access to the teacher’s output distribution, the student loses the direct feedback signal that makes white-box OPD work.

GAD borrows an idea from adversarial training, where two models are trained in opposition to each other. In this case, the second model is a discriminator whose job is to tell teacher responses apart from the student’s responses. The discriminator learns to score teacher responses more highly, while the student is trained with reinforcement learning to beat the discriminator. The authors argue that the discriminator effectively replaces the missing teacher distribution as the feedback signal for on-policy learning.

SODA followed a similar progression to GKD after MiniLLM: it tried to preserve the core idea while removing much of the machinery. Rather than continuously training a discriminator and updating the student with reinforcement learning, SODA turns the teacher-student difference into a simpler preference-learning problem. The student is trained to prefer the teacher’s responses over its own, without the adversarial training loop. The simplification was substantial: the authors report roughly a 10× training speedup over GAD while matching or exceeding it on most of their evaluations.

Both methods seemed primarily optimized for teacher-like responses rather than the teacher’s underlying reasoning process. GAD’s own analysis says that, compared to Kim and Rush’s sequence-level distillation, it better captures the teacher’s “global stylistic characteristics.” For my use case, that distinction mattered: a response could sound like the teacher and follow the same general structure while still missing the critical idea or calculation that determines the position.

The evaluations did little to address whether these methods could transfer reasoning. Neither GAD nor SODA reported results on dedicated reasoning benchmarks such as BBH, AGIEval, or GSM8K; both focused primarily on conversational datasets evaluated with an LLM judge. This lowered my confidence that they would transfer the chess capability I cared about.

At this point, I was less interested in whether the on-policy idea could be made to work with a closed-weight teacher. The broader question was whether other forms of black-box distillation could transfer the reasoning capability I wanted. So I widened the search beyond OPD.

Beyond On-Policy Distillation

Proxy-KD gave me the reasoning benchmarks that GAD and SODA were missing. The method trained a large open-weight proxy to approximate the closed-weight teacher, then used that proxy to recover some of the probability-level feedback unavailable through the API. It resembled pieces of both earlier approaches: like GAD, it introduced an additional trainable model, and like SODA, it used preference optimization to align that model with the teacher.

The reasoning gains were real, but they did little to close the gap to GPT-4. On AGIEval, Proxy-KD improved a vanilla black-box student from 34.7 to 36.6, while GPT-4 scored 56.4. On GSM8K, it improved from 49.5 to 53.1 against GPT-4’s 92. BBH showed a larger gain, from 46.7 to 53.4, but still remained far behind GPT-4’s 88. Even with a 70B proxy and a considerably more elaborate training pipeline, the student recovered only a small fraction of the teacher’s reasoning advantage.

Lion took a different approach. It used the teacher to identify where the student performed poorly, then generated new training examples targeting those weaknesses. Over successive iterations, these “hard” examples increasingly concentrated on complex tasks such as math and coding, which made the approach feel more relevant to chess. The method produced meaningful gains over other open-source baselines, but it still fell well short of the teacher on reasoning. In the paper’s limitations section, the authors write that “Lion still lags behind its teacher model ChatGPT in handling intricate reasoning tasks.”

A pattern was starting to emerge: black-box distillation seemed much more effective at transferring the teacher’s style and response behavior than its reasoning capability. Proxy-KD and Lion showed that reasoning could improve, but both still left a large gap to the teacher. The False Promise of Imitating Proprietary LLMs made that distinction much more explicit.

What Can Distillation Actually Transfer?

The False Promise of Imitating Proprietary LLMs examined a more fundamental question: whether distillation from model-generated outputs transfers the teacher’s underlying capabilities or primarily its surface behavior. The models quickly learned ChatGPT’s surface characteristics, including response structure, length, use of lists, and authoritative tone. But they showed much less evidence of learning the knowledge and problem-solving ability behind those responses. On benchmarks designed to test those capabilities, including MMLU, Natural Questions, and HumanEval, broad distillation produced little improvement over the base models and left a large gap to ChatGPT.

One example made the distinction especially clear: when asked about actor-critic reinforcement learning, an imitation model produced a polished answer with nearly the same structure as ChatGPT’s while getting much of the substance wrong. It had learned the form of a correct answer without reliably learning the substance. The authors summarized the result as “mimicking ChatGPT’s style but not its factuality.”

But False Promise also contained one result that pointed in a much more promising direction. When the authors trained on a dataset constructed specifically for Natural Questions, distillation worked far better. The 13B base model started at 20 on Natural Questions, broad imitation reduced it to 15, and the specialized dataset improved it to 27, much closer to ChatGPT’s score of 31.

I had spent all this time exploring increasingly sophisticated ways to distill Gemini through a black-box API, only to keep running into the same limitation: transferring broad reasoning capability was hard. But I did not need generic reasoning, just chess reasoning. False Promise had shown that distillation could work much better when the target was narrow and the training data was built specifically around it. That pointed to a different path: a highly specific chess dataset, evaluated on an equally specific chess task.

Distilling Chess-Specific Reasoning

Grounded Chess Reasoning in Language Models via Master Distillation proposes a chess-specific distillation pipeline without introducing a new distillation objective. Instead, its contribution is largely in the data: a 4B student is fine-tuned on tactical chess puzzles paired with carefully constructed reasoning traces. At the training-objective level, it is essentially the sequence-level distillation proposed by Kim and Rush.

And it worked. On the paper’s chess-puzzle benchmark, their model C1-SFT-4B scored 40.9%, essentially matching the 40.8% accuracy of Gemini-3-Flash, the teacher model. For the first time in my search, I had found a distillation result that closed the teacher gap on its target benchmark. It was exactly the kind of narrow success False Promise had suggested was possible.

The success, however, was extremely specific to the task it was trained on. Tactical puzzles represent a very small slice of chess positions: they are positions that only emerge immediately after an opponent has made a mistake and contain a single correct continuation. I needed to know whether C1-SFT-4B could reason about ordinary chess positions, so I reran my mistake-rate benchmark on 100 randomly sampled positions while keeping the prompting and context identical. Gemini-3-Flash, their teacher model, made a mistake on just 2% of positions; C1-SFT-4B made a mistake on 42%. At that point, the result was unusable for ordinary chess positions: a 42% mistake rate was more than twice the upper end of the beginner range. Whatever the model had learned from the puzzle dataset had not transferred to chess positions in general.

Arguably, C1-SFT-4B’s general chess reasoning could be improved by scaling the dataset to cover a wider variety of chess positions. But that brings me back to why distillation was attractive in the first place. I had started with Hinton et al.’s result that a student could recover nearly all of the teacher’s performance using only 3% of the original dataset. That result captured the magic of distillation: sample-efficient transfer of broad capability from the teacher to the student. For chess reasoning, that magic did not apply.

References

• Hinton et al. (2015), Distilling the Knowledge in a Neural Network
• Kim and Rush (2016), Sequence-Level Knowledge Distillation
• Gu et al. (2023), MiniLLM: On-Policy Distillation of Large Language Models
• Agarwal et al. (2023), On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Ye et al. (2025), Black-Box On-Policy Distillation of Large Language Models
• Chen et al. (2026), SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Chen et al. (2024), Knowledge Distillation of Black-Box Large Language Models
• Jiang et al. (2023), Lion: Adversarial Distillation of Proprietary Large Language Models
• Gudibande et al. (2023), The False Promise of Imitating Proprietary LLMs
• Tang et al. (2026), Grounded Chess Reasoning in Language Models via Master Distillation