One Night of AutoResearch on Reasoning Distillation

What a 0.6-billion-parameter student learns in 20 minutes, and what an AI agent found out about it overnight.

Code, logs, and results: https://github.com/alessoh/ai-distillation

Summary

We took the distillation recipe from chapter 8 of Sebastian Raschka's *Build a Reasoning Model (From Scratch)*, wrapped it in Andrej Karpathy's AutoResearch loop, and let a coding agent run experiments on an H100 for a night. The student is Qwen3-0.6B. The teachers are DeepSeek-R1 and Qwen3-235B, whose written solutions to 12,000 MATH problems come with the book. Each experiment trains for a fixed 20 minutes of wall-clock time and is scored by accuracy on 100 held-out problems.

In 31 experiments the score went from 26 to 39. The next day, controlled repeats with three seeds per teacher and a clean evaluation on MATH-500 sorted the real effects from the noise. The findings:

1. The student's dominant failure was not reasoning badly but never finishing. Every change that shortened what it imitated helped, and deleting the teacher's reasoning trace entirely helped most.

2. Trained on final solutions only, for 20 minutes, the student reaches 36.0% on MATH-500 with DeepSeek-R1 data, above the book's 33.6% after three full epochs with the traces kept. With Qwen3-235B data it reaches 38.1%, well below the book's 45.0% with traces.

3. The Qwen3 teacher's advantage therefore lives mostly in its reasoning traces, not in its final answers. Without traces the gap between teachers shrinks from 11 points to 2.

4. The noise floor of a 100-problem score is about plus or minus 4 points. Roughly half of the agent's keep-or-discard decisions were coin flips, which is why the seed repeats mattered.

Background

Knowledge distillation trains a small model to imitate a large one (Hinton, Vinyals, and Dean, 2015). For reasoning models the usual form is to have the teacher write out solutions, including its chain of thought, and fine-tune the student on that text. DeepSeek-R1's release made this mainstream: the R1 paper's distilled Qwen and Llama students were trained on 800,000 teacher-written samples.

Chapter 8 of Raschka's book implements exactly this at hobby scale. The student is the Qwen3-0.6B base model, the teachers are DeepSeek-R1 and Qwen3-235B-A22B, and the data is 12,000 problems from the MATH training set with the MATH-500 test set removed. The book reports MATH-500 accuracy of 15.2% for the base model, 33.6% after three epochs on the DeepSeek-R1 data, 45.0% after one epoch on the Qwen3-235B data, and 48.2% for Qwen's own 0.6B reasoning model.

Karpathy's AutoResearch is a layout for letting an agent do research unattended. One file holds the fixed rules and the metric and is never edited. One file holds the training recipe and is the only thing the agent changes. A third file is the agent's instruction sheet. The agent edits, commits, trains for a fixed time, reads the metric, keeps the commit if the metric improved, and otherwise resets, forever.

The kit

The repository has the same three files. `prepare.py` downloads the teacher data, holds out 100 problems with a fixed seed, loads the model, and scores the student by generating an answer to each held-out problem with greedy decoding and grading the boxed result with the book's chapter 3 verifier. `train.py` reproduces the chapter's batched recipe by default, so the first run is the baseline. `program.md` is adapted from Karpathy's, with a results table and a noise rule.

Two design choices mattered. The score is accuracy that the student actually achieves, not validation loss, because the book's own exercise shows the two can move in opposite directions. And MATH-500 is never touched inside the loop, so it stays a clean test set for the write-up.

Hardware was one H100 on a Lightning AI studio. Calibrating the harness took two runs. The chapter's recipe with four 2048-token sequences per step needs about 77 GB and crashed the 80 GB card, which gradient accumulation fixed. With a 1536-token generation cap, 72 of the 100 held-out responses never finished, so the cap was raised to 2048 to match the training length. The baseline then scored 26, with 63 of 100 responses still truncated, and of the 37 that finished, 26 were correct.

Night 1: the loop

The agent ran 31 experiments at about 28 minutes each. Four were kept.

| Change | Held-out score |

|---|---|

| Baseline, chapter 8 recipe | 26 |

| Batch 16, learning rate 5e-5, warmup, cosine decay | 33 |

| Strip backtracking sentences from the reasoning traces | 36 |

| Drop the reasoning trace, train on the teacher's final solution only | 39 |

| Remove now-inert code, identical data and recipe | 36 |

The last row is the important one. Nothing functional changed and the score moved by 3 points. The agent noted this itself and called it the noise floor. Everything it tried afterward, including three weight decay values, two learning-rate schedules, and a partial reasoning trace, landed within that band.

Several ideas from the literature failed clearly at this scale and budget. Dropping teacher solutions whose final answer was wrong, about 15% of the DeepSeek data, cost 6 points. An exponential moving average of the weights cost 7. Label smoothing cost 3. A learning rate of 1e-4 cost 8. Extra loss weight on the final-answer tokens cost 8. A 200-word reasoning budget was worse than no trace at all.

The thread that worked was about length. Capping solutions at 1024 tokens cut truncation sharply but lowered accuracy per finished attempt, a wash. Stripping backtracking helped. Deleting the trace, so the student learns only the clean written solution, took truncation from 47 responses to 1, memory from 36 GB to 11 GB, and the score to its peak.

Day 2: seeds and MATH-500

Single runs cannot separate a 3-point effect from noise, so the winning recipe was run three times per teacher with different seeds, the final weights saved, and every checkpoint evaluated on all 500 MATH-500 problems with the same greedy decoding and a 2048-token cap.

| Student | Held-out, 3 seeds | MATH-500, 3 seeds | MATH-500 mean |

|---|---|---|---|

| DeepSeek-R1 data, solutions only | 34, 42, 35 | 36.2, 36.8, 35.0 | 36.0 |

| Qwen3-235B data, solutions only | 45, 39, 38 | 38.4, 38.0, 38.0 | 38.1 |

| Untrained base model | | | 24.6 |

Three things stand out. The held-out score of the same recipe spread 8 points across seeds while MATH-500 spread under 2, so a 100-problem score is simply too small to steer a loop. Every Qwen3 run beat every DeepSeek run, but by 2 points on MATH-500 rather than the book's 11, so most of the stronger teacher's advantage was carried by its reasoning traces. And the DeepSeek student beat the book's DeepSeek result while training for 0.58 of an epoch instead of three, whereas the Qwen3 student fell 7 points short of the book's, so DeepSeek's traces hurt this student and Qwen3's help it.

The base model's 24.6% is higher than the book's 15.2% because the protocol differs: the evaluation here uses the chat template and a 2048-token cap rather than the book's 512. The numbers within this study are consistent with each other, and the gains over base, 11 and 13 points, are the fair comparison.

What this says

The result is consistent with the 2026 literature on trace compression, which finds that compressed or truncated traces cut training cost by several times while raw traces keep the highest accuracy, and with the observation that small students struggle to imitate long, intricate reasoning. Here the effect is stronger than usual because the student is tiny and the budget is short: at 0.6B parameters and 20 minutes, the reasoning trace from DeepSeek-R1 is a liability, and from Qwen3-235B it is an asset the student cannot yet afford.

It also says something about AutoResearch as a tool. The loop found the three real effects in one night, unattended. But its metric was too noisy for the search it was asked to do, and once the obvious length effects were exhausted it spent the rest of the night dithering within the noise band. The loop suits cheap experiments with quiet metrics. Here the fix is a 200-problem score, a keep threshold of 5 points, and a mandatory repeat of every apparent winner.

The most interesting question is untested. Every experiment here is off-policy: the student imitates text the teacher wrote. On-policy distillation, where the student writes and the teacher grades each token, is now standard in large-model post-training and has a growing 2026 literature on its failure modes, but almost none of it is measured on a student this small. A live Qwen3-4B teacher shares the student's vocabulary and would fit beside it on one GPU. That is the next stage of this project.

Reproducing

```bash

git clone https://github.com/alessoh/ai-distillation

cd ai-distillation

uv sync

uv run prepare.py # downloads data and weights, smoke-tests evaluation

uv run train.py # one experiment, about 28 minutes on an H100

bash run_designed.sh # 3 seeds x 2 teachers, then MATH-500 for each checkpoint

```

The night 1 results table and every experiment log are under `results/sep2/`, the seed runs and MATH-500 logs under `results/designed-sep2/`, and the experiment branch itself is `autoresearch/sep2`.

References

Project

- This repository: https://github.com/alessoh/ai-distillation

- Night 1 experiment branch: https://github.com/alessoh/ai-distillation/tree/autoresearch/sep2

- Night 1 results and logs: https://github.com/alessoh/ai-distillation/tree/main/results/sep2

- Seed runs and MATH-500 logs: https://github.com/alessoh/ai-distillation/tree/main/results/designed-sep2

Starting points

- Andrej Karpathy, autoresearch: https://github.com/karpathy/autoresearch

- Sebastian Raschka, *Build a Reasoning Model (From Scratch)*, Manning: https://mng.bz/lZ5B and https://sebastianraschka.com/reasoning-from-scratch/

- Book code repository: https://github.com/rasbt/reasoning-from-scratch

- Chapter 8 distillation training scripts and reported results: https://github.com/rasbt/reasoning-from-scratch/tree/main/ch08/04_train_with_distillation

- Chapter 8 teacher-data generation: https://github.com/rasbt/reasoning-from-scratch/tree/main/ch08/02_generate_distillation_data

- Teacher datasets (DeepSeek-R1 and Qwen3-235B solutions): https://huggingface.co/datasets/rasbt/math_distill

- Qwen3-0.6B weights and tokenizers used by the book: https://huggingface.co/rasbt/qwen3-from-scratch

- MATH-500 test file used by the book: https://raw.githubusercontent.com/rasbt/reasoning-from-scratch/main/ch03/01_main-chapter-code/math500_test.json

Models and benchmarks

- Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset (2021): https://arxiv.org/abs/2103.03874

- Lightman et al., Let's Verify Step by Step (2023), the source of the MATH-500 subset: https://arxiv.org/abs/2305.20050

- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025): https://arxiv.org/abs/2501.12948

- Qwen Team, Qwen3 Technical Report (2025): https://arxiv.org/abs/2505.09388

Distillation

- Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network (2015): https://arxiv.org/abs/1503.02531

- Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023): https://arxiv.org/abs/2306.13649

- Thinking Machines Lab, On-Policy Distillation (2025): https://thinkingmachines.ai/blog/on-policy-distillation/

- A Survey of On-Policy Distillation for Large Language Models (2026): https://arxiv.org/abs/2604.00626

- Rethinking On-Policy Distillation of Large Language Models (2026): https://arxiv.org/abs/2604.13016

- Entropy-Aware On-Policy Distillation of Language Models (2026): https://arxiv.org/abs/2603.07079

- Trust Region On-Policy Distillation (2026): https://arxiv.org/abs/2606.01249

- Teach Small Models to Reason by Curriculum Distillation (EMNLP 2025): https://aclanthology.org/2025.emnlp-main.376/

- Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation (2026): https://arxiv.org/abs/2606.05988

- When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation (2026): https://arxiv.org/abs/2606.21704

- Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces (2026): https://arxiv.org/abs/2605.29288

Tools

- Lightning AI Studios: https://lightning.ai

- uv, the Python package manager used by the kit: https://github.com/astral-sh/uv

- PyTorch CUDA memory management notes: https://pytorch.org/docs/stable/notes/cuda.html#environment-variables