In two weeks, 100,000 domestic AI accelerators began optimizing their own models.Author: APPSO
Source: Wall Street Journal
Just now, Tang Jie, Chief Scientist at Zhipu GLM, shared a study on GLM-5.3-Flash inference system optimization on the X platform.
He revealed that it took only two weeks from the first run of GLM-5.3-Flash on a domestic AI accelerator to fully handling all production traffic. During this process, the system’s end-to-end throughput increased by 3.2 times.
What impressed Tang Jie the most was that the extensive optimization work was not done solely by infrastructure engineers, but also by an Infra Agent powered by GLM-5.3.
“A model that helps optimize its own services,” Tang Jie described this change.
In his view, this means AI is beginning to participate in optimizing the systems that host its own operation. Although still far from true recursive self-improvement (RSI), an early form has already emerged.
The Zhipu team also noted that over the past year, they have observed a shift in GLM's role.
Initially, the team explored GLM's capabilities in code understanding and cybersecurity, hoping to leverage the model to help identify vulnerabilities in complex code. However, as the model's capabilities improved, GLM began participating in the construction of AI systems themselves.
The team said they have observed the model completing tasks that previously required senior infrastructure engineering teams weeks to accomplish, and these tasks directly impact how next-generation models are trained and deployed.
Deploy domestic computing cluster within two weeks
Significant systems engineering effort is required to move a large model from its first run on new hardware to a stable inference service capable of handling production requests.
GLM-5.3-Flash is deployed on a cluster comprising over 100,000 domestic AI accelerators. The Zhipu team stated that this was a large-scale deployment without prior mature experience to reference.
The team needs to address multiple challenges, including limitations in domestic chip memory capacity and interconnect bandwidth, adaptation to new model architectures, support for 1 million token long contexts, and processing of multimodal requests. Meanwhile, the software ecosystem for domestic AI accelerators is still under development, with insufficient Kernel support and much of the necessary information requiring independent exploration by the engineering team.
Tang Jie stated that, under these constraints, the Infra Agent powered by GLM-5.3 participated in the reasoning system optimization process, helping to analyze performance bottlenecks, propose optimization solutions, and implement certain code modifications.
The entire optimization process is not simply about increasing computational resources, but rather about finding a new balance among computing power, memory, communication, and scheduling.
The Zhipu team has implemented a series of optimization strategies. For example, they use ReplaySSM to trade computation for memory space, reduce GPU memory pressure through intra-node tensor parallelism, improve capacity utilization via mixed-precision caching with INT8, FP8, and BF16, and introduce the Encode-Prefill-Decode (EPD) modular architecture to enable more flexible scheduling across different inference stages.
Ultimately, the end-to-end service performance of GLM-5.3-Flash has improved by approximately 3 times compared to the initial version, with hardware utilization and cost per token reaching levels comparable to mainstream NVIDIA GPU platforms.
After deployment, GLM-5.3-Flash entered real-world usage testing. The Zhipu team noted that the model previously operated under the anonymous name Ox-Alpha on the OpenCode and OpenRouter platforms, becoming one of the most widely used models on both platforms within a week, processing over 62 trillion tokens in six days.
However, the real breakthrough of this experiment was not just having AI write code, but enabling it to understand why performance changes occur in complex systems.
For example, when the system reports a "20% drop in throughput," it can only indicate that an anomaly has occurred, but cannot directly tell the agent which layer is at fault, whether the current assumption is incorrect, or what should be verified next.
For experienced engineers, such judgments rely on years of experience. Engineers know when to examine execution timelines, when to run microbenchmarks, and which module outputs to compare.
The Zhipu team aims to transform this engineering experience into a feedback mechanism callable by AI, which they refer to as "dense feedback."
The core of this mechanism is to provide the Agent with information that more closely resembles the engineering judgment process.
When the model needs to verify whether a calculation is correct, it can obtain corresponding correctness feedback; when it needs to analyze the cause of performance degradation, it can review the time consumption during system operation; when it tries new optimization strategies, it can evaluate whether the strategy is suitable for the current scenario through experimentation.
The Zhipu team believes that truly effective feedback must meet several conditions: it must be closely tied to the specific issue, quickly obtainable, and verifiable through objective experiments. Otherwise, large volumes of logs and metrics may make it difficult for the agent to determine the next course of action.
Model optimization system, system service model
With the help of dense feedback, the Infra Agent begins to identify hidden issues in the reasoning system.
In the context of KDA's parallel paths, the agent identified an issue affecting the accuracy of long-context computations.
Due to the continuous merging of state matrices across different context shards, rounding errors from TF32 computations accumulate with increasing sequence length, ultimately leading to computational inaccuracies.
The agent identified the issue as precision handling during state propagation and merging by comparing the outcomes of different execution paths. The related fix has been merged into Flash Linear Attention PR #1180.
Another issue arises between KV Transfer and DeepEP scheduling.
During testing, the Agent found that KV Transfer did not effectively overlap with DeepEP Dispatch, resulting in transmission overhead exceeding 30% in certain scenarios.
Subsequently, the Agent continued analyzing the Python and C++ call chain and discovered that the Python GIL was not released promptly within the node, preventing the transmission task from progressing in a timely manner.
After the modifications, the additional overhead introduced by KV Transfer has been reduced from over 30% to less than 1%.
In addition, the Agent also optimized a decode kernel.
It found that due to the kernel's splitting method, the same normalization computation was executed four times. By reorganizing the computation structure to reduce redundancy, the kernel achieved a 1.71x performance improvement.
This optimization approach is derived from the Agent’s learning of existing kernels in projects such as SGLang, Flash Linear Attention, and DeepGEMM. It distills the optimization insights from these codes into an "optimization skeleton" and then evaluates their applicability based on current system feedback.
In the past, similar optimization experience primarily relied on individual engineers' accumulated knowledge.
During this process, the Agent begins to learn optimization methods from existing code and then validates these methods through experiments to determine their suitability for new hardware and model environments.
Tang Jie stated that human engineers remain responsible for setting goals, establishing feedback environments, and reviewing high-risk changes.
But the role of engineers is shifting from directly solving every problem to designing feedback systems.
The Zhipu team believes that a verifiable feedback environment built on real infrastructure tasks may also serve as a crucial foundation for training the next generation of models. Each time an agent completes an engineering task, it could become data for learning in the next generation of models.
Currently, the GLM-5.3-Flash case is still distant from true recursive self-improvement. However, Tang Jie believes that a minimal loop has already emerged.
The model optimizes the system, and the system serves the model.