The Overnight Recursive Researcher

Here is an experiment you can run tonight. Give an AI agent a Connect Four training script, a fixed evaluation function, and one instruction document. Tell it to modify the script, train a network for five minutes of self-play, check whether the agent plays better, keep or discard the change, and repeat. Then go to sleep.

In the morning there is a ledger — results.tsv, one row per experiment, roughly a hundred rows — and a game-playing network measurably stronger than the one you left. You did no research overnight. Research happened anyway.

That is the premise of Karpathy's Autoresearch, which pointed the loop at GPT pretraining. My project, autoresearch-connect4, adapts the same pattern to reinforcement learning on a toy game. The shrinking is the point. This post is about why a deliberately small substrate turns out to be a serious instrument for studying recursive AI — systems that improve the systems that learn.

Three files and a sacred number

The Autoresearch pattern is austere. There are exactly three files that matter.

prepare.py is the immutable harness. In the Connect Four version it holds the game engine, a fixed suite of four opponents — random, one-step lookahead, minimax at depth 3 and depth 5 — and one sacred function, evaluate_winrate, which plays a hundred games against each opponent and produces a single weighted number. The agent may never touch this file.

train.py is the single mutable file: a convolutional network that evaluates board positions, a REINFORCE training loop, and every hyperparameter. Anything in it is fair game — architecture, optimizer, exploration strategy, board encoding, inference-time search. The only constraints are that it runs without crashing and finishes inside the five-minute budget.

program.md is the human's sole lever: the instruction document the agent reads before it starts, containing the experiment loop, the logging format, a simplicity criterion, and the directive that keeps it running until morning.

Everything interesting about the design flows from one discipline: the researcher cannot grade its own homework. The metric lives outside the file the agent edits. There is no partial credit, no narrative justification, no "the loss looked better." A change either moved the number or it didn't.

commit b71section 2keeptwo nested loops

The loop inside the loop

Moving the pattern from language modeling to a game added something the original doesn't have: two fully separated levels of optimization.

The inner loop is ordinary reinforcement learning. A network starts from random weights, plays roughly 39,000 games of Connect Four in five minutes, and learns a strategy from nothing.

The outer loop is the research process. The agent edits train.py, launches a run, reads one number out of the log, and decides keep or discard — about twelve times an hour.

The agent in the outer loop never plays Connect Four. It never watches a game. It never learns strategy. It sees one number per experiment, and from that number alone it learns which blueprints produce the best learners.

This separation is what makes the project a study of recursion rather than a study of Connect Four. The outer loop is not optimizing a policy; it is optimizing the process that produces policies. That is the same shape as an AI improving its own training code, scaled down to where every consequence is visible by breakfast.

commit c04section 3keepwhat the loop actually found.

What the loop actually found

The naive baseline — a textbook REINFORCE implementation — scored a weighted win rate of 0.063. It beat the random opponent 82% of the time, lost everything else, and its training loss diverged to −800,000. Textbook correct, practically broken.

Getting to the current 0.227 required five fixes, and they are worth listing because none of them is exotic. Advantage normalization, so gradient magnitudes stay bounded as the policy grows confident. Advantage clipping for outlier batches. Log-probability clipping, so forced exploration moves can't dominate the gradient. An entropy bonus, so the policy doesn't collapse to playing one column forever. And opponent mixing — the single biggest jump came from making the network spend 30% of its games against a fixed one-step opponent instead of pure self-play. Two bad players playing only each other never learn to block a threat. That one change took the one-step win rate from 0% to 100%.

random         : 0.950 win_rate

one_step       : 1.000 win_rate

minimax_d3     : 0.000 win_rate

minimax_d5     : 0.000 win_rate

The current model crushes the reactive opponents and loses every game to minimax. That cliff is not a failure; it is the frontier. The fixed opponent ladder gives the outer loop a staircase of measurable challenges, and "learn to beat depth-3 search in five minutes of training" is the step it is standing under right now.

commit d92section 4keepwhy small substrates matter.

Why this advances recursive AI

It is fair to ask what a toy game can possibly say about recursive self-improvement, a topic usually discussed in the vocabulary of frontier labs. I think it says quite a lot, for four reasons.

It makes recursion legible. When an agent modifies the training code of a large model, cause and effect are separated by days of compute and a fog of confounds. Here the full causal chain — edit, train, evaluate, decide — closes in about five minutes and is recorded in a ledger a human reads in one sitting. Every claim about how the research loop behaves can be checked against the receipts.

It isolates the researcher from the player. Because the outer loop only ever sees a scalar, you can study the research process as its own object. Does it hill-climb into local optima? Does it re-discover known techniques? Does it favor complexity or fight it? The Connect Four ledger answers these empirically. Overnight, the loop's job is precisely the one a human RL researcher would do — and its keep/discard trail is a dataset about machine research behavior, not just about the game.

The sacred metric is a miniature alignment mechanism. The single most load-bearing design decision is that evaluate_winrate is off-limits. An agent that could edit its own evaluator would trivially "improve" — the toy version of reward hacking. Keeping the metric outside the blast radius of self-modification, and studying how much optimization pressure that boundary can absorb, is a small, concrete rehearsal of a question that matters enormously at scale.

It exposes the weaknesses of greedy recursion. The keep/discard rule is hill-climbing on one noisy number. Some genuinely good changes look flat for a while before they pay off; a greedy loop discards them. Some improvements are within evaluation noise; a greedy loop keeps them anyway. These are not bugs in my repo — they are fundamental limits of scalar-feedback self-improvement, and a substrate this cheap lets you map them experimentally instead of arguing about them.

commit e18section 5keepthe frontier

Where it goes next

Connect Four is the first substrate, not the last. Each new game stresses a different joint of the recursion. Nim has an exact mathematical solution, so it asks whether the loop can select for architectures that grok an algorithm — and whether greedy selection kills changes that would have grokked with more time. Simplified poker replaces win rate with exploitability and asks what happens to hill-climbing when the game has no best pure strategy. Ultimate Tic-Tac-Toe is recursive in its very board, a chance to see whether problem structure gets reflected into blueprint structure. And running Gomoku warm-started with the findings of the Connect Four run — not the weights, the findings — measures whether research itself transfers, which is recursion across runs.

The deeper bet behind all of this: before machines can do important research autonomously, we need instruments for studying how machine research behaves — where it is reliable, where it is greedy, where it fools itself. Grand versions of that question are expensive and slow. A Connect Four board, an immutable judge, and one long night turn out to be enough to start answering it.