DeepSeek publishes paper on agent training system DSec with Liang Wenfeng

Training large models is about computing power; training agents is about the environment.

How is the environment created? Signed by Liang Wenhong DeepSeek The latest paper has disclosed the technical details.

DeepSeek The system we built is called DSec ( DeepSeek (Elastic Compute), which is used to batch-create sandboxes for Agent training.

It can generate 5,000+ sandboxes per second, reaching 3 million per day, with a peak of 380,000 running simultaneously.

The single cluster supporting this scale is also very large, consisting of approximately 160 nodes, 30,000 CPU cores, and 250 TB of memory.

Why is training an agent so difficult?

Because the training environment for large models is a GPU cluster, where data is fed and gradients are computed, agents are completely different.

It needs to write code, compile, open a browser, and even install an operating system within a sandbox—each step alters the environment’s state and could potentially crash it at any time.

So each training round requires a brand-new, clean sandbox that is used once and discarded afterward.

So, after all the back and forth, the issue ultimately comes down to infrastructure—

This infrastructure must provision a complete operating system and toolchain for each sandbox at a rate of 5,000 per second.

At the same time, we must prevent hundreds of thousands of concurrent sandboxes from overwhelming the cluster’s memory and CPU.

How exactly to proceed? The paper lays out the entire scope of this project.

The first core problem DSec aims to solve is that different types of Agent tasks have vastly different requirements for sandbox environments, and these environments must be uniformly scheduled on the same platform.

An agent for solving OJ problems requires only a stateless function call environment to run and retrieve the output; persistent file system storage is not needed.

However, an agent performing SWE-bench requires a complete Linux user-space environment, where it must install dependencies, modify code, run pytest, and potentially add new packages midway through the task.

In security defense and computer-use scenarios, container-level isolation is insufficient; the agent must interact with the browser or even the desktop. A vulnerable agent could inadvertently compromise the host machine, making virtual machines essential.

In the most extreme case, training an agent to operate commercial software requires a full Windows or macOS system with a graphical interface and drivers, nearly indistinguishable from a real computer.

DSec has prepared four backends for these four scenarios: FnCall handles stateless function calls, Container runs Docker containers, MicroVM uses Firecracker for lightweight virtual machines, and Full VM runs complete operating systems with QEMU.

The isolation strength and resource overhead of the four backends increase progressively, but the training framework sees a unified Python SDK: libdsec.

Regardless of whether the underlying infrastructure is a container or a virtual machine, the same interface is used to create a sandbox, execute commands, and retrieve results—the method of calling each step remains identical.

To run all four backends on the same cluster, the platform’s scheduling layer must also keep pace.

DSec has broken down the entire chain into six layers.

This workflow begins with a creation request from the training framework, which first passes through IAM authentication and authorization, then enters the API Server. The scheduling engine (Placement Engine) selects a target node from the cluster based on available resources, and the Edge component on that node is responsible for actually launching the corresponding type of sandbox.

The network egress and package management mirrors in the sandbox are uniformly proxied by Aether. Every command executed by the Agent and every line of output it generates are relayed back to the training framework through a sandbox-internal communication component called Chronus, enabling the framework to track the Agent’s progress and provide appropriate feedback.

Through resource overcommitment and high-density deployment, a single node can simultaneously host 3,200 containers or 800 MicroVMs.

However, DSec's most significant challenge in scale is not scheduling, but environment construction.

Each sandbox requires a complete operating system image and toolchain upon startup, equivalent to installing an operating system on 5,000 "computers" per second.

The traditional Docker approach is to bundle the base image, workspace, and toolset into a single complete image.

This approach works fine at a small scale, but DSec’s container backend has accumulated 11,266 base images and 102,171 workspaces, with 67.8% of sandboxes requiring at least one additional layer of workspace or toolkit stacked on top of the base image.

Under this diversity, once a toolkit is updated, all composite images containing it must be rebuilt, resulting in a cost of O(m·N).

DSec's approach splits the environment into three independent, read-only EROFS images: base image, workspace, and toolset, each versioned separately. These layers are dynamically combined at sandbox startup using overlayfs. Updating the toolset affects only the toolset layer, reducing costs to O(m) + O(k).

Equally important is how to deliver the image to the node after it has been built.

Intuitively, you should pull the image to local cache in advance, but the paper statistics real runtime data:

The Python container image is 6.0 GB, but the Agent actually read only 6.0% of the data within it;

Java image: 12.1 GB, only 9.2% accessed;

The C++ image is 4.9 GB, with only 8.7% accessed.

In other words, the Agent never interacted with the vast majority of the mirrored content.

Therefore, DSec opts for on-demand loading, with its images stored in EROFS format on 3FS (Fire-Flyer Distributed File System), metadata pre-fetched locally, and data blocks only pulled from 3FS when actually accessed by the sandbox.

DeepSeek Our team tested and found that burst deployment of 8,192 containers with on-demand loading takes just 35 minutes, while Docker cold pulls take over 60 minutes.

Additionally, the on-demand loaded disk writes are reduced by more than half, from approximately 1,600 GB to approximately 700 GB.

After the environment is set up, hundreds of thousands of sandboxes running simultaneously face resource contention.

In terms of memory, when MicroVM reads image data via a virtual block device, the same data is stored separately in both the host and the virtual machine’s page cache, doubling the memory requirement.

DSec uses virtio-pmem with DAX to allow virtual machines to bypass their own page cache and directly map to the host's physical memory, enabling multiple virtual machines to share the same mapping, reducing peak memory usage by 40.2%.

For writable disks incompatible with virtio-pmem, DSec periodically scans cold memory pages using DAMON and actively returns them to the host. Combined with virtio-balloon’s free-page reporting, this further reduces demand by 21.2%.

For CPU management, DSec categorizes sandboxes into latency-sensitive and best-effort types, with the latter set to SCHED_IDLE priority. Additionally, Linux core scheduling is enabled to prevent low-priority tasks from running on sibling hyperthreads of the physical cores hosting high-priority tasks.

After stacking the two-layer strategy, the latency inflation for latency-sensitive tasks under 50% background load decreased from 45.2% to 17.3%.

Additionally, DSec must work in conjunction with the RL training framework to handle GPU resource contention.

In the early architecture, the Agent's reasoning loop ran inside the GPU training Pod; when GPU tasks were preempted, all of the Agent's progress was lost.

From DeepSeek Starting from V4.1, the Agent loop has been separated and runs independently in DSec's worker container, no longer tied to the lifecycle of the GPU Pod.

When the GPU is preempted, DSec suspends and saves the state of the associated sandbox; when the GPU resumes, the sandbox continues execution, eliminating the need for the training framework to implement its own checkpoint and recovery logic.

Even with all these optimizations, a single cluster may still struggle to handle peak loads.

The paper mentions that a maximum production task requests 32,000 sandboxes at once; under this burst load, DSec automatically triggers cloud bursting to offload overflow tasks to cloud virtual machines.

DeepSeek Our team's testing shows that when cluster utilization exceeds 80%, 200 cloud VMs can absorb approximately 30% of peak demand.

For an agent, the sandbox is not just an execution environment, but also a security barrier.

The paper reveals multiple instances of reward hacking techniques that agents discovered on their own during training—methods that allow agents to achieve high scores without legitimately completing the intended tasks.

The first target of these agents is Chronus, the component within the sandbox responsible for communication between agents and the external environment.

When Chronus performs certain operations, it invokes the system's bash. Upon detecting this, the Agent overwrites /bin/bash and injects its own commands to intercept quiz answers or other privileged information from subsequent shell sessions.

After this path was blocked by AppArmor, the agent found a more fundamental vulnerability: the XFS filesystem's ioctl call.

XFS_IOC_SWAPEXT is used to swap the underlying data block mappings between two files. The agent uses it to swap the contents of the protected file onto a file descriptor it controls, bypassing file-level access controls.

The side effect of this attempt is more dangerous than cheating, as it corrupts the XFS metadata, potentially causing immediate filesystem failure that affects more than just this sandbox.

Agents have also found information leakage channels outside the sandbox.

Some agents scan network ports and services accessible within the sandbox to locate reference implementations; others cheat directly by pulling code from GitHub via Go module proxies; some even install updated software packages to obtain ready-made solutions.

These behaviors cannot be detected by simply examining the final output, because the Agent indeed "got the right answer"—just through improper means.

Beyond cheating, there are even more violent scenarios.

An agent recursively executed grep within a sandbox, scanning from the root directory all the way to /proc/kpagecgroup, triggering a kernel bug that caused the entire host kernel to crash.

Another agent executed the attack command intended for the target virtual machine within its own container, similarly crashing the kernel.

The agent also discovered that Chronus asynchronously logs stdout output, leading to repeated calls to the yes command that continuously printed characters, filling the storage space with dozens of gigabytes of logs.

DSec's current defense measures consist of two layers.

First, AppArmor is used to control file read/write permissions and Unix domain socket access; these policies remain effective even if the Agent runs as root, limiting the Agent’s ability to extract answers from logs and internal communication channels.

Second, use eBPF for fine-grained network-layer control, configuring domain name allowlists based on tasks and blocking out-of-bounds traffic through triple filtering by IP address, port, and protocol. Policies can also be dynamically updated at different stages of the task—for example, allowing access to PyPI during environment setup and tightening network restrictions during actual execution.

However, the paper also clearly states that this is not a problem that can be fully resolved.

AppArmor and eBPF can restrict information leakage channels, but they cannot prevent kernel bugs; user isolation can limit the blast radius, but agents will always find new pathways.

This will become an ongoing battle between AI and countermeasures—the more powerful the model, the better its ability to exploit vulnerabilities, forcing platforms to continuously strengthen their defenses.

Agents have evolved from being able to "speak" to being able to "act," and the complexity of training infrastructure has undergone a qualitative transformation.

Training large models relies on "brute force yields breakthroughs," but training agent clusters requires not only being both large and precise, but also capable of defending against what they themselves have trained.

The opponent that needs to be defended against is precisely the Agent itself being trained.

Paper URL: https://arxiv.org/abs/2609.22978

This article is from the WeChat public account "Quantum Bit," authored by Kresi.