The MIT CSAIL team has proposed a "weight-space programming" method that enables GPT-6 Astra to directly read and modify neural network weights. In the Qwen3.5-0.8B model, adjusting just 0.016% of the weights increased the MATH-500 score from 7.9% to 48.1%, significantly resolving reasoning loop issues. Experiments also demonstrated Astra’s ability to understand weights, repair faults, and generate networks from scratch; however, its effectiveness remains limited to simpler models and it cannot yet replace training or universally generate complex models.Article author and source: Big Data Digest
The AI circle has struck again!
The MIT CSAIL team has proposed a method called "Weight-Space Programming": without training or computing gradients, GPT-6 Astra directly reads and modifies the weights of the neural network.
In experiments with Qwen3.5-0.8B, Astra improved the model's MATH-500 score from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval instruction-following score from 21.0% to 62.3%, by modifying only about 0.016% of the weights.
This means that large models are beginning to acquire the ability to directly understand and edit neural network parameters.
This work was led by MIT CSAIL PhD students Rikiya Takehi and Jayden Teoh, with collaborators Alex Zhang and Omar Khattab. The research team published their findings on the Sparks of Weight-Space Programming research blog.
What is weight space programming?
Weights are the core parameters of a neural network, determining how the model processes input information and generates the final output.
Traditional model training typically requires preparing data, computing loss and gradients, and iteratively adjusting weights through an optimizer. The entire process relies on substantial computational resources, and the model’s internal parameters are difficult for humans to understand directly.
Weight space programming offers a different approach:
Treat the weights of the neural network as objects that can be read, analyzed, and modified by large models.
General large models directly judge network functionality, identify anomalous structures, and adjust corresponding parameters based on natural language objectives.
In this study, Astra primarily accomplished three types of tasks: 1. Reading weights to determine what task the neural network was trained to perform; 2. Modifying weights to fix specific faults in the network; 3. Generating weights from scratch to construct small networks with specific functionalities.
The research team first tested Astra's ability to understand neural network weights.
In the first set of experiments, researchers trained four identical small neural networks using PPO to play Breakout, Pong, Pac-Man, and Freeway.
Astra received the full original weight but did not know the candidate game names and could not observe behavior by running the network. In the end, it correctly identified which game corresponded to each of the four networks.
In the second set of experiments, researchers trained three diffusion models to generate circles, triangles, and squares, respectively. Astra again successfully identified the target shape generated by each model, using only the weights.
The third set of experiments extended to language models. Researchers trained three small Transformers with 7.2 million parameters each to learn Python code, romanized Japanese, and English. Even when only intermediate layer weights were provided and the input and output layers were removed, Astra was still able to identify the type of training corpus and locate duplicated attention heads within the network.
These experiments suggest that general-purpose large models may not only be able to process text and code, but also extract functional information about neural networks from their weights.
Second, fix the inference loop by adjusting only a 0.016% weight.
The Qwen3.5-0.8B experiment is the most prominent part of this study.
After enabling thought mode, this model tends to fall into reasoning loops, repeatedly outputting similar content and failing to provide final answers in a timely manner. In baseline testing, 93% of MATH-500 responses reached the 32,768-token length limit.
Astra accesses model weights via a Python interface and evaluates model behavior using practice problems from the GSM8K and MATH datasets, but cannot access training or gradient information, nor the final test set.
After analysis, Astra identified the issue in parts of the MLP structure at layers 15 and 19 and removed a direction related to duplicate outputs.
This update involved 131,072 weights, approximately 0.016% of the model's total weights. After the update, the percentage of MATH-500 responses hitting the token limit decreased from 93% to 33.1%.
Therefore, this improvement in performance was primarily not due to injecting new mathematical knowledge into the model, but rather fixing the issues that were hindering its reasoning ability. Some problems that previously could not be solved now produce complete, correct answers.
However, although the final patch was small, the analysis process was not simple. The best-performing experiment used 50 evaluation requests, 158 dialogue rounds, and 80 Python runs, taking 20.3 hours.
III. Writing Neural Network Weights from Scratch
The research team also tested Astra’s ability to generate network parameters from scratch.
In the MinAtari games Breakout, Seaquest, and Asterix, Astra is given the game rules, source code, and network architecture, but cannot access training data, loss functions, gradients, or environment playthroughs.
It generated a new network that outperformed the corresponding network trained with 100 million steps of PPO in all three games.
Compared to a PPO network trained for 500 million steps, the network generated by Astra achieves about 20 times the score in Breakout, performs similarly in Asterix, and reaches about 60% of the score of the latter in Seaquest.
In the 9×9 Go experiment, Astra generated a network with approximately 1.48 million parameters. As the number of allowed forward evaluations increased, the network's performance continued to improve; under 50 forward evaluations, it defeated a PPO network of the same size trained on 20 million games.
These results suggest that large models may already be capable of directly converting part of the algorithmic logic into neural network weights based on task descriptions.
However, related experiments have still primarily focused on simpler, smaller-scale networks; whether they can be extended to larger, more complex models requires further validation.
To distinguish between “fixing bugs” and “enhancing capabilities,” the research team also conducted additional experiments on Gemma 3 270M.
Gemma 3 270M rarely encounters reasoning loops; the main errors stem from arithmetic and algebraic reasoning. Astra attempted multiple rounds of targeted modifications, but the MATH-500 score improved by no more than approximately 1 percentage point, making it impossible to confirm that the model achieved stable improvement.
This indicates that weight space programming is currently better suited for addressing well-defined, focused model failures, such as anomalous loops or specific behaviors, rather than directly enhancing the model’s original knowledge or reasoning capabilities.
Meanwhile, Astra’s failure in another experiment to identify which model was trained with flattery based solely on weights also indicates that its weight understanding is not stable and still has task boundaries.
Currently, these results are early findings released by the research team and require further independent replication and systematic evaluation.
If this approach can be further scaled, it could be used to identify model failures, remove anomalous behaviors, improve interpretability, and even generate specific networks based on natural language requirements.
However, current experiments are not sufficient to prove that it can replace training, nor does it mean that large models can generally "build" other models from scratch.
The more significant implication of this study is that it demonstrates a new way of interaction:
It's not just generating text and code anymore—it's also beginning to understand and write neural networks themselves.
Source:Big Data Digest
Risk Disclosure and Disclaimer
The market carries risks; invest with caution. This article does not constitute personal investment advice and has not taken into account individual users’ specific investment objectives, financial situations, or needs. Users should consider whether any opinions, views, or conclusions presented herein are suitable for their particular circumstances. Investment decisions based on this content are made at your own risk.