Cursor's MoK Delivers a 2.37x Performance Improvement on NVIDIA GB300 NVL72

Cursor open-sourced MoK, integrating token scheduling, cross-GPU communication, and expert computation into a single GPU kernel. This solution achieves up to 2.37x faster forward pass and 1.78x faster backward pass on GB300 NVL72, increasing training throughput across 512 GB300 GPUs by 41% to 1070.2 tokens/s. Signaling latency drops from 103 μs to 18 μs. Analysis indicates that, with NVLink bandwidth now reaching 130 TB/s, data movement time on the GPU has become the new performance bottleneck—this breakthrough marks the AI competition’s entry into the “full-stack sovereignty” phase, where companies that bring code closer to GPU memory and registers gain greater pricing power.

Author and source: Leiphone

On July 21, NVIDIA announced the latest results for training DeepSeek-V3 using the GB300 NVL72: on 256 GPUs, each GPU achieves a performance of 1,648 TFLOPS.

Less than two weeks later, Cursor open-sourced Mixture-of-Kittens, or MoK. Instead of continuing to optimize faster matrix multiplication, it completely rewrote the execution layer of MoE, combining token scheduling, cross-GPU communication, and expert computation into a single GPU kernel.

This is somewhat counterintuitive. Since the GB300 NVL72 has already placed all 72 GPUs within the same NVLink domain, achieving a total NVLink bandwidth of 130 TB/s across the entire rack, data transfer between GPUs should already be fast enough according to these specifications.

In large-scale MoE training, communication still bottlenecks expert computation.

The issue lies in the MoE data flow. At each step, the router reassigns tokens to different experts, which are distributed across multiple GPUs. Tokens must first be sent across devices, computed, and then sent back; before transmission, their positions must be organized, and upon arrival, the system must wait for all data to be complete. As MXFP8 and Blackwell Tensor Cores have significantly reduced expert computation time, these previously hidden waiting periods are now becoming increasingly noticeable.

Cursor tackles MoK by starting here. Instead of focusing solely on how fast a single dispatch can transmit, it reorganizes how tokens reach experts, when computation begins, and how communication and computation simultaneously utilize the GPU.

Today, with computing power pushed to its limits by Blackwell and NVLink, developers have realized: no matter how fast the hardware runs, it cannot save inefficient software orchestration.

The open-sourcing of MoK is not just a victory for the kernel; it marks the beginning of an era where "application-layer-defined operators" take center stage: AI startups are waging a war for底层sovereignty to squeeze out the final 30% of computational power.

However, to understand why Cursor’s design works, you first need to see where the MoE incurs its “communication tax.”

01

MoE's "communication tax"

In a standard dense FFN, the weights a token passes through are largely fixed. With MoE added, each token dynamically selects a few experts. When using Expert Parallelism, the experts are distributed across many GPUs, so at least two rounds of cross-GPU communication are required during a single forward pass.

The first round is called Dispatch, sending the token to the GPU where the expert is located; after the expert completes the computation, the result is sent back to the token’s original location via Combine. Training also involves backpropagation, which requires two additional rounds of communication in the opposite direction.

If you're simply moving a large block of contiguous data from A to B, NVLink is already fast enough. The challenge with MoE is that the data distribution provided by the router differs at each step.

An expert might receive many tokens in one step and very few in the next. The system must first count how many tokens each expert has, then determine where to place those tokens on the target GPU, while grouping data from the same expert together as much as possible. Grouped GEMM requires well-structured input to efficiently feed the Tensor Cores.

Even after communication ends, computation cannot begin immediately. The target GPU must confirm that all remote writes have been completed and the data is truly visible. Additionally, expert workloads are not perfectly balanced—some GPUs may finish early but still must wait for the busiest expert to complete.

So within a single "communication" phase, multiple tasks are mixed together: data movement, layout generation, synchronization, and load imbalance. The 130 TB/s figure represents the peak bandwidth the entire rack can provide, but it does not mean that every dynamic, fragmented MoE communication can simultaneously saturate all these links.

DeepEP has optimized its data movement to be extremely fast. As a high-performance communication library designed for Expert Parallelism, it provides specialized Dispatch and Combine kernels, supports FP8, and allows control over the number of SMs used for communication. The latest version can maintain high communication throughput using significantly fewer SMs.

Even a single-layer MoE still requires constant handoffs between Dispatch, Grouped GEMM, and Combine.

At this point, a practical contradiction arises: if you wait for enough tokens to arrive before computing, the matrix becomes large, allowing Tensor Cores to operate efficiently, but computation starts late; if you compute as soon as even a few tokens arrive, communication and computation can overlap earlier, but the matrix becomes too small, leaving many GPU SMs underutilized.

Multiple CUDA streams can enable concurrent communication and computation, but it is difficult to consistently ensure that both sides receive the optimal amount of GPU resources.

The design after MoK primarily addresses this "rhythm" issue.

02

One of the most interesting changes in MoK is switching the forward dispatch from Push to Pull.

Traditional Push is straightforward: the source GPU, having the token, actively writes it into the target GPU. The complication arises with the target address—a single GPU may simultaneously receive tokens from many other GPUs.

Each sender must know in advance which section to write to, avoiding overlap; tokens from the same expert should also be arranged consecutively, otherwise the subsequent GEMM operation will need to be reorganized.

As more GPUs participate, this scheduling process becomes increasingly heavy. Pull adopts a different approach: experts save the target GPU and fetch the required tokens themselves. They only need to know which source GPU the token is on and its position in the source data; where to store it locally is up to them.

This eliminates the need for coordination among multiple senders regarding the target address. The received data can also be directly organized according to local experts.

Interestingly, Pull did not transmit less data. In the microbenchmark for Cursor, for the same 256×256 BF16 data block, Push moved approximately 159.6 KB over NVLink, while Pull reached 172.0 KB due to the additional requests required for reading.

Pull excels in another aspect: MoE communication is fragmented and load-imbalanced. NVLink has independent channels in both directions, allowing Pull to utilize both the request and data return paths simultaneously. In tests with uneven expert loads, Cursor achieved up to a 29% increase in NVLink utilization.

The synchronization gap is more pronounced. After a Push operation completes, the target GPU must wait for completion signals from other GPUs; with Expert Parallelism at scale, a single rank may involve up to 71 peers. With Pull, the local GPU initiates the read itself, and data can be used immediately upon arrival.

In Cursor's multi-node microbenchmark, this signaling latency decreased from approximately 103 microseconds with Push to 18 microseconds with Pull.

MoK also doesn't use Pull everywhere. For the forward pass, it uses Pull for Dispatch and Push for Combine; for the backward pass, it uses Pull for Reverse-Combine and Push for Reverse-Dispatch. During the Dispatch phase, tokens from many sources need to be reorganized into expert inputs—Pull is more efficient for coordination. During Combine, it’s already clear which token each result should return to, so Pushing directly is simpler.

After changing the communication direction, MoK still combined communication and expert computation within the same Megakernel.

It divides the GPU's SM into two parts. One part handles dispatch, combination, and state management, while the other is dedicated to executing expert FFNs. After the communication side receives a batch of complete tokens, it notifies the computation side via a local GPU counter; once computation is complete, it notifies the communication side to send the results back.

This allows you to directly determine how many SMs are allocated for communication versus computation, rather than leaving it entirely up to multiple CUDA streams to compete. The most critical parameter here is the minibatch, which defines how many tokens are processed by the experts at once.

It can't be too large. Too large means the first batch of computations will take a long time to complete. It also can't be too small. The expert GEMM ultimately breaks down into numerous computational tasks assigned to SMs; if there are too few tokens, there won't be enough tasks, leaving many SMs idle.

Cursor uses waves to determine this boundary. A complete wave can be simply understood as all compute SMs being assigned work. MoK aims for at least two complete waves per minibatch to ensure Tensor Cores have sufficient tasks for continuous execution.

The actual results speak for themselves. On the Kimi 2.5 architecture with a hidden size of 7168 and expert intermediate dimension of 2048, Cursor estimates that a minibatch requires at least approximately 2,368 tokens. At 512 tokens, the MoK forward pass takes 5.981 ms; when increased to 2,560 tokens, it drops to 3.425 ms. Further increases yield no significant improvement in speed.

In other words, breaking communication into smaller pieces doesn't always make it faster. True efficient overlap requires delivering data as early as possible while not splitting the GEMM operation too finely.

But MoE has another issue: before the router finishes its work, you simply don’t know how many tokens each GPU will ultimately receive.

Preparing a buffer for the worst-case scenario would waste a lot of VRAM. If we first let the GPU count the tokens and then notify the CPU to allocate the corresponding space, the GPU would have to stop and wait for the CPU.

MoK uses a fixed-size Ring Token Buffer. Space is first filled with dispatched tokens; once experts finish processing and the Combine unit transfers the results, that space is immediately reused for the next batch of tokens. The Combine phase of one macrobatch can occur simultaneously with the Dispatch phase of the next macrobatch.

The ring buffer acts like a buffer layer here: when communication runs faster, data accumulates within it; when computation consumes faster, it waits for the next batch of tokens. The entire process is driven by state progression on the GPU, eliminating the need for the CPU to intervene at each step to determine the next action.

Cursor also integrated the MXFP8 activation quantization into the data paths of Dispatch, Grouped GEMM, and SwiGLU, eliminating the need for a separate quantize kernel and reducing one round of intermediate data reads and writes to HBM.

When Pull, minibatch, SM partitions, and Ring Buffer are combined, MoK truly becomes a continuous MoE pipeline.

03

Cursor's benchmark evaluates the complete MoE layer, including scheduling, dispatch, expert FFN, combining, and the final weighted aggregation, with comparison targets including NCCL + PyTorch, DeepEP, TransformerEngine, and HybridEP + Megatron.

On the GB300 NVL72, compared to the fastest public baselines for each scenario, MoK achieves up to 2.37x faster forward and 1.78x faster backward performance with MXFP8, and up to 1.92x faster forward and 1.58x faster backward performance with BF16.

More importantly, end-to-end training is crucial. Cursor’s original production system already used DeepEP; after switching to MoK on 512 GB300 GPUs, the per-GPU throughput increased from 760.9 tokens per second to 1,070.2 tokens per second, representing a roughly 41% improvement.

You should also be clear about the boundaries of these results. Since Cursor has not published a complete itemized ablation study, it is not possible to accurately determine how much of the 2.37x improvement comes from Pull, how much from Megakernel, and how much from Ring Buffer.

The improvement in NVLink utilization and signaling latency can be individually attributed to Pull; the remaining benefits stem from the combined effect of the entire execution approach.

MoK also has strong hardware dependencies. It is designed for high-speed NVLink domains such as Blackwell and NVL72. Remote reads, communication, and fine-grained interleaving of computation all rely on GPUs being able to access each other’s memory with low latency. When the model’s hidden size, top-k, or expert scale changes, the appropriate minibatch size and number of communication SMs also adjust accordingly.

This is also the most compelling aspect of MoK. In the past, discussions about MoE optimization often focused solely on two metrics: how many TFLOPS the GEMM achieves and how many GB/s the All-to-All transfers. With the GB300 generation, simply pushing these two numbers higher no longer fully explains the overall performance.

When the token arrives, how to arrange it into the layout required by experts, how many to accumulate before calculation begins, how much SM to allocate for communication, and when to release the buffer—these execution details directly determine training speed.

A rack with 130 TB/s NVLink bandwidth still requires rewriting GPU kernels for MoE—precisely because the links are already fast; what needs to be saved now is the time GPUs spend waiting for data.

Cursor's rewriting of GPU kernels marks the competition in the AI 2.0 era entering a new phase of "full-stack sovereignty."

In the past, we firmly believed in specialization—applications should focus on applications (Cursor), and infrastructure should focus on infrastructure (NVIDIA). But today’s AI competition has entered a new phase of disintermediation: Cursor isn’t building its own kernel because it wants to, but because it has no choice.

DeepSeek has ushered in a new era of engineering extraction, while Cursor has taken this fire to the application layer. This trend of “disintermediation” is reshaping the pricing power of AI: in the future, what determines an AI company’s valuation will no longer be how many tokens it owns, but how close its code is to VRAM and registers.

AI companies that cannot penetrate the underlying black box will ultimately remain stuck in the quagmire of the "mediocrity tax."