MIT CSAIL proposes weight-space programming: GPT-6 Astra directly rewrites neural network weights

The MIT CSAIL team proposed a method called "Weight-Space Programming": without training or computing gradients, it lets GPT-6 Astra directly read and modify the weights of neural networks.

In experiments on Qwen3.5-0.8B, Astra modified only about 0.016% of the weights, yet raised the model's MATH-500 score from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and the IFEval instruction-following score from 21.0% to 62.3%.

This means that large models are beginning to have the ability to directly understand and edit neural network parameters.

The work was co-led by MIT CSAIL doctoral students Rikiya Takehi and Jayden Teoh, with collaborators including Alex Zhang and Omar Khattab. The research team published the results in the research blog "Sparks of Weight-Space Programming."

1. What is Weight-Space Programming?

Weights are the core parameters in a neural network, determining how the model processes input information and generates the final output.

Traditional model training usually requires preparing data, computing loss and gradients, and then continuously adjusting the weights through an optimizer. The entire process relies on a large amount of computation, and the internal parameters of the model are also difficult for humans to understand directly.

Weight-Space Programming offers a different approach:

treating the weights of a neural network as an object that can be read, analyzed, and modified by a large model.

A general-purpose large model directly determines the network's function, locates anomalous structures, and modifies the corresponding parameters based on a natural-language goal.

In this study, Astra mainly completed three types of tasks: 1. reading weights to determine what task a neural network was trained to perform; 2. modifying weights to repair specific faults in the network; 3. generating weights from scratch to construct small networks with specific functions.

The research team first tested Astra's ability to understand neural network weights.

In the first set of experiments, the researchers used PPO to train four small neural networks with completely identical structures, respectively for playing Breakout, Pong, Pac-Man, and Freeway.

Astra was given all the original weights, but did not know the candidate game names and could not observe behavior by running the networks. In the end, it correctly identified the game corresponding to each of the four networks.

In the second set of experiments, the researchers trained three diffusion models to generate circles, triangles, and squares, respectively. Again using only the weights, Astra successfully determined the generation target of each model.

The third set of experiments further extended to language models. The researchers trained three small Transformers with 7.2 million parameters, respectively learning Python code, Romanized Japanese, and English. Even when only the middle-layer weights were provided, with the input and output layers removed, Astra was still able to identify the type of training corpus and find the copy attention heads in the network.

These experiments show that general-purpose large models may be able not only to process text and code, but also to extract functional information about neural networks from weights.

2. Modifying only 0.016% of the weights to fix a reasoning loop

The Qwen3.5-0.8B experiment is the most closely watched part of this research.

After enabling thinking mode, this model easily falls into reasoning loops, repeatedly outputting similar content and failing to give a final answer in time. In baseline tests, 93% of MATH-500 responses hit the 32,768-token length limit.

Astra accessed the model weights through a Python interface and used practice problems from the GSM8K and MATH training sets to check model behavior, but could not use training or gradient information, nor could it access the final test set.

After analysis, Astra localized the problem to parts of the MLP structures in layers 15 and 19, and deleted a direction related to repetitive output.

This modification involved a total of 131,072 weights, about 0.016% of the model's total weights. After the modification, the proportion of MATH-500 responses hitting the token limit dropped from 93% to 33.1%.

Therefore, this score improvement was mainly not about injecting new mathematical knowledge into the model, but about repairing a fault that prevented the model from completing its reasoning. Some problems that could originally be solved could output complete answers normally after the modification.

However, although the final patch was very small, the analysis process was not simple. The best-performing experiment used 50 rounds of evaluation requests, 158 rounds of dialogue, and 80 Python runs, taking 20.3 hours.

3. Writing neural network weights from scratch

The research team also tested Astra's ability to generate network parameters from scratch.

In three MinAtar games—Breakout, Seaquest, and Asterix—Astra was given the game rules, source code, and network structure, but could not use training data, loss functions, gradients, or environment playtesting.

The new networks it generated directly surpassed the corresponding networks trained with 100 million steps of PPO in all three games.

Compared with PPO networks trained for 500 million steps, the networks generated by Astra achieved about 20 times the score in Breakout, performed comparably in Asterix, and reached about 60% of the latter's score in Seaquest.

In a 9×9 Go experiment, Astra generated a network with about 1.48 million parameters. As the number of allowed forward evaluations increased, the network's performance continued to improve; under the condition of 50 forward evaluations, it defeated an equally sized PPO network trained on 20 million games.

These results suggest that large models may already be able to translate some algorithmic logic directly into neural network weights based on task descriptions.

However, the relevant experiments are still mainly focused on networks with simple structures and small scale, and whether this can be extended to larger and more complex models still needs further verification.

To distinguish between "repairing faults" and "improving capabilities," the research team also conducted additional experiments on Gemma 3 270M.

Gemma 3 270M rarely falls into reasoning loops, and its main errors come from arithmetic and algebraic reasoning. Astra tried multiple rounds of targeted modifications, but the improvement in MATH-500 scores never exceeded about 1 percentage point, and it could not be confirmed that the model had achieved stable improvement.

This shows that Weight-Space Programming is currently better suited to handling clear, concentrated model faults, such as abnormal loops or specific behaviors, and cannot directly add knowledge and reasoning capabilities that the model did not originally possess.

At the same time, Astra failed in another experiment to identify, using only the weights, which model had been trained for sycophantic behavior, which also shows that its weight-understanding ability is not stable and still has task boundaries.

At present, these results are early results released by the research team and still require more independent reproduction and systematic evaluation.

If this method can be further extended, it may be used to locate model faults, remove abnormal behaviors, improve interpretability, and even generate specific networks based on natural-language requirements.

But the current experiments are not yet sufficient to prove that it can replace training, nor do they mean that large models can already generally "hand-craft" other models.

The more important significance of this research lies in the fact that it demonstrates a new way of interaction:

not only generating text and code, but also beginning to understand and write neural networks themselves.

This article comes from the WeChat public account "Big Data Digest" and is published by 36Kr with authorization.