Astra Lifted Qwen’s Math Score From 7.9% to 48.1% by Editing Weights, but the Gain Came From Fixing a Loop

MIT CSAIL researchers published a blog post on 2 November 2026 claiming a frontier language model can read and write other neural networks’ weights directly, without gradient descent. GPT-6 Astra was given raw numbers from small networks and asked what they do and how to change them.

In the most striking experiment, Astra edited 131,072 weights in Qwen3.5-0.8B, about 0.016% of the model, and raised its MATH-500 score from 7.9% to 48.1%. The same edit lifted GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%.

The catch is the diagnosis. Qwen3.5-0.8B was not weak at maths because it lacked the ability, but because it got stuck in reasoning loops, and Astra removed a direction in the weights that caused them. When the team tried the same approach on Gemma 3 270M, a model that rarely loops, the gains disappeared.

Quick answer: The MIT CSAIL result is real in the narrow sense that a general-purpose model edited the weights of a public 0.8B model and improved its scores without training. It is not evidence that Astra can install new reasoning ability. The big gain came from fixing a known looping defect, the model is small, and the work is a research blog, not a peer-reviewed paper.

What the MIT team actually did

The authors call this weight-space programming. Instead of writing prompts or training code, it inspects the numbers inside a network and changes them directly, from a 17,253-weight policy network to a model with roughly 800 million weights.

Astra named the correct game for all four PPO policies from their weights alone. It also read the middle layers of three small language models, identified each training language, and found every copying head without running the network.

Astra also fixed a car controller stuck in a local minimum, changing 3.5% of weights and taking success from 0% to 100% across 1,024 episodes. It wrote MinAtar networks that beat PPO policies trained for 100 million steps, and a Go network that beat an equal-sized PPO policy trained on 20 million games with 50 forward passes.

Those results are striking but small-scale. The authors call the work early and frame weight-space programming as a direction, not a finished replacement for training.

The Qwen result is real, but the setup matters

The Qwen experiment will travel furthest because the numbers look like a step change. Astra got a Python notebook with the weights loaded and could run forward passes on prompts from GSM8K and MATH, but no training or gradients.

It found that the model looped, sometimes for tens of thousands of tokens, and traced the behaviour to a direction written by the MLPs in layers 15 and 19. It subtracted that direction from 64 output columns in each layer. The edit touches 131,072 weights, or 0.016% of the model, and the blog says it generalises to instruction following without extra work.

The size of the gain depends on how many forward passes Astra gets. With 10 passes, MATH-500 rose only from 7.9% to 12.3%, and 87.2% of answers still hit the 32,768-token cap. With 25 passes, it reached 14.1%. The headline 48.1% arrives at 50 passes, when the share of answers hitting the cap falls to 33.1%.

Qwen3.5-0.8B setting	MATH-500	GSM8K	IFEval	Answers hitting 32,768-token cap
Base model	7.9%	13.8%	21.0%	93.0%
Edited, 10 forward passes	12.3%	25.7%	49.0%	87.2%
Edited, 25 forward passes	14.1%	25.1%	39.9%	86.4%
Edited, 50 forward passes	48.1%	67.1%	62.3%	33.1%

The first 25 passes mostly did not fix the loop, so the model kept burning tokens. The jump at 50 passes is where it finally broke. Treat 48.1% as the result of a specific search budget, not a free upgrade that arrives with one prompt, and remember the test covers one small model and two maths benchmarks, the kind of narrow measure that GDPx was built to question.

Why fixing a loop is not the same as teaching maths

The Gemma control is the most useful part of the post. Gemma 3 270M rarely loops, with only 8.5% of answers hitting its token limit, so it lacks the same defect. Astra found ordinary arithmetic and algebra mistakes, with no single component to switch off.

Its edits stayed small and targeted, such as doubling one neuron that responds to numbers, and none moved MATH-500 by more than about a point. Astra itself reported the gain as not established. The base score was 6.8%, and the edited versions scored between 7.0% and 7.7%.

That is the difference between removing an obstacle and adding an ability. Qwen already had enough maths capability to answer many questions, but a loop stopped it finishing. Gemma lacked that hidden ability, and weight editing could not supply it. The post says this plainly, but the headline number is easy to misread.

Reading weights is not the same as understanding them

The sycophancy experiment adds another caution. The team fine-tuned two copies of Gemma 3 270M, one to answer correctly and one to agree with a suggested answer, and asked Astra to identify the sycophant from the weights alone. It failed in all four attempts.

It could list differences but could not map them to behaviour. Only when allowed to inspect attention patterns on text it wrote itself did it pick the sycophant, in all three pairs. In the sycophantic model, head 10.2 sent 85.7% of attention from the answer position to the user’s suggested answer, against 0.75% in the honest model.

So the model can find structure in weights, but it does not automatically understand what that structure means. The authors present weight-space programming as a tool for interpretability and optimisation, not as a finished auditing method.

Fascinating finding! Astra can interpret the function of smaller models just by looking at the weights. And it can fix flaws in smaller models. Positive update both for mechanistic interpretability / AI monitoring, and for RSI.

Caveat: The models Astra improved were 1,000 – 58… https://t.co/2nuih9zIrn

— Ramez Naam (@ramez) November 2, 2026

The caveat from Ramez Naam is the right one. The models Astra improved were between 1,000 and 58 million times smaller than Astra itself. A result on a 0.8B model does not scale to a frontier model just because the method is interesting.

What it means for Indian developers

For teams building on small open models, the practical lesson is to check for degenerate behaviour before blaming the base model. A model that loops can look weak on a benchmark when it actually has the ability to answer. Fixing the loop can produce a larger gain than another round of instruction tuning.

The second lesson is that weight editing is not free. It needs a frontier model, a harness and repeated passes, and this result depended on 50 passes over training prompts. Teams with limited GPU budgets should compare that cost with quantisation, better sampling settings or a small supervised fine-tune.

The third lesson is reproducibility. The authors released the logs, the edited model and a demo. Hugging Face’s Ahsen Khaliq rebuilt the edited model from the logs and re-ran its evaluation, according to the post. That is useful, but not independent peer review, and it does not test another model family.

Confirmed vs not confirmed

Claim	Status	Evidence
Astra edited 131,072 weights, or 0.016% of Qwen3.5-0.8B	Confirmed	Method and code logs in the primary post and GitHub repository
MATH-500 rose from 7.9% to 48.1% with 50 forward passes	Confirmed for this setup	Reported benchmark table, sampling settings and released logs
The gain came mostly from fixing a reasoning loop	Confirmed by the authors	Token-limit rate fell from 93.0% to 33.1%; Gemma control showed no gain
Astra can identify a sycophantic model from weights alone	Not confirmed	It failed in all four attempts without attention patterns
The method works at frontier scale or adds new capability	Not confirmed	No larger-model results; Gemma edits did not move the score
The work has passed peer review	Not confirmed	It is a research blog post dated 2 November 2026

FAQ

What is weight-space programming?

Weight-space programming asks a language model to read and write another network’s numerical weights using reasoning rather than gradient descent, then edit them around a natural-language goal.

Did Astra really improve Qwen without training?

Yes, no gradients or training loop were used. Astra ran forward passes on prompts and wrote a targeted weight edit. That is still optimisation, just not gradient-based training.

Does this mean AI can improve itself?

Not from this result. It improved a much smaller model with a known looping defect. The post discusses recursive self-improvement as a future direction but does not demonstrate it at frontier scale.

Can I reproduce the result?

The logs, edited model and demo are public. Reproducing the search would need GPT-6 Astra and the same harness. The blog does not publish a full cost estimate for the 50-pass run.

Is this peer reviewed?

No. It is a blog post from four MIT CSAIL researchers dated 2 November 2026. The logs and Hugging Face rebuild make it checkable, but peer review is still missing.

Sources

• 2 November 2026: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, “Sparks of Weight-Space Programming: LLMs can natively read and write neural network weights,” MIT CSAIL.
• 2 November 2026: Experiment logs for the weight-space programming post, GitHub.
• 2 November 2026: Edited Qwen3.5-0.8B model card, Hugging Face.
• 2 November 2026: Weight-space programming demo, Hugging Face Spaces.
• Qwen model card: Qwen3.5-0.8B README, including the warning about thinking loops.