Yesterday, Rikiya Takehi and Jayden Teoh et al. posted LLMs can natively read & write neural network weights. One part is that GPT-6 Astra wrote the weights of small game-playing networks by hand without testing them, and they beat PPO on three MinAtar games.
The authors released Astra's full response, so you can see that it doesn't "learn" anything but comes up with a strategy, like figure out where the ball will land and move the paddle there, and then writes that strategy into the weights directly so neurons can behave like on/off switches. To make neurons behave like switches, Astra adds huge values in the weight. The Breakout network, as the most extreme one, has weights of 1,000 and biases of −415. It's nearly a finite-state controller compiled as a neural network for all three games.
The post suggests these networks could be a cheap starting point preceding normal training. My first reaction was that this wouldn't work, because huge weights mean huge activations, and huge activations push tanh units and the softmax into their flat regions, where gradients nearly vanish. I expected PPO either to get no signal at all, or to get a messy one and quickly wreck a network that was clearly built on exact switching thresholds.
So I took Astra's three networks and kept training them with PPO to find out, using the same settings the original post used for its baselines, with 5 random seeds per game. As a reference, I also trained from random weights with the exact same setup. Each run was 1 billion steps.
The result is not the way I expected: PPO took over the training just fine. After 1 billion steps, Seaquest went from about 53 points to between 88 and 116, roughly double. Asterix went from about 54 to between 78 and 85. Nothing collapsed along the way, and all five Astra-start runs outperformed every random-start run.
Breakout is the exception. PPO didn't change it at all: after 1 billion steps, all five seeds ended at exactly the same score. Here my worry about flat gradients was right, just not for the reason I expected. The hidden units aren't the problem: all three networks have heavily saturated hidden units, and the other two still learned. The problem is that Breakout's network is completely sure of itself. In every state I checked, its chosen action has probability 1 minus about 10−13, and PPO's update shrinks with how much probability is left to move.
However, Breakout also barely needs help. Astra's network scores over 500 in a game where published RL agents usually score in the tens. My best random-start run reached 127 after a billion steps. The scoring caps each game at 10,000 steps. Astra's Breakout policy already lasts about 7,400 steps per game on average because it almost never misses the ball, so there's not much room left to improve.
I really want to see what exactly happens to the gradients and try bigger networks and more complicated policies, but I am short of time to dig further (so I made this post!). This was still a small, hands-on test -- there may well be cases where a network is highly certain but still has room to improve, and PPO may not take over it well as I originally feared. I will push the code today or tomorrow, just in case someone would like to test it further.
Details: PPO settings follow the original post, with unpublished details (like the value network size) set to CleanRL defaults. Training ran in JAX: gymnax for Breakout, and Seaquest and Asterix ported from the official MinAtar code after I found small differences in gymnax. All ported games were checked state by state against the official code. All scores come from the official game: 1,000 games per seed at the end of training, 200 games per seed at intermediate checkpoints, and 5,000 games for Astra's untrained networks. Full float32 precision was forced on the GPU (RTX A5000). Code coming soon. Astra's networks are from the authors' release.