Ultraman revealed on the podcast why OpenAI prioritizes Codex over Sora. Sora’s video generation requires substantial continuous compute, with each task consuming GPU time that is difficult to reuse; in contrast, Codex leverages mechanisms such as KV cache, continuous batching, and tool calling to distribute compute across multiple interleaved stages, enabling a single GPU to support more concurrent workflows. The ability to reuse GPU time is beginning to determine the pace of expansion for AI products.

Author and source: Leiphone

Why did Sora lose to Codex?

On August 23, Otomachi mentioned Sora while discussing resource trade-offs within OpenAI on David Senra's podcast.

He was straightforward: Sora is a good product and could become a solid business if continued, but it’s extremely compute-intensive; at the same time, Codex had higher priority, so computing resources and team efforts began shifting toward Codex.

But interestingly, Codex doesn’t save GPU resources either. Sora generates a video by running multiple rounds of Transformer computations on massive spatiotemporal latents; when Codex receives a command like “fix this bug,” it may repeatedly run inference, read code, call tools, execute tests, and then return with new logs and context to continue inference.

One concentrates all computational power into a single video generation, while the other distributes computational power across an Agent workflow that may last tens of minutes or longer. Thus, the true divergence between Sora and Codex begins to manifest within the data center.

Why is it harder to amortize compute for video generation with the same batch of GPUs, while Coding Agents can reuse compute across more concurrent tasks through KV cache, continuous batching, prefill/decode scheduling, and tool waiting?

In fact, Sora did not lose due to the absolute amount of computational power consumed, but rather due to its workload architecture: Sora’s computational power is continuous and exclusive, while Codex’s is fragmented and reusable. It is this difference in scheduling mechanisms that accounts for the disparity in their expansion speeds.

Looking further down this line, the resources Sora lost may reflect a deeper choice about how GPU time should be allocated.

The cost of Sora may begin to escalate as soon as the video enters the model. It first compresses the raw video into latent space, then slices it into spacetime patches, allowing the Transformer to compute over these patches.

Text tokens primarily grow along the sequence direction, while video patches are simultaneously spread across time, height, and width, making video inherently a three-dimensional state within the model.

On a high level, the number of visual tokens can be understood as N_video ≈ T × H × W, where T, H, and W have already been compressed and patched, but the three-dimensional multiplication relationship still holds.

Extending the video duration increases patches along the temporal dimension, while increasing the frame size expands the spatial patches. In other words, video length and spatial dimensions do not independently increase costs—they jointly expand the latent grid.

After this grid enters the Transformer, it must still contend with the second layer of computation introduced by diffusion. Sora starts with a noisy latent and, in each iteration, updates the video representation based on the current state, then feeds the new latent into the next iteration.

The computational cost of a single video can be roughly understood as C_video ≈ D × C_transformer(N_video), where D is the number of sampling iterations. The larger the video latent, the heavier each iteration becomes; increasing the number of sampling iterations means running the network additional times for the same video.

Here you can see the key difference between video diffusion and LLMs. When a language model generates subsequent tokens, the previous Keys and Values can be stored in the KV cache, so the model does not need to reconstruct the entire historical state at each step.

After each round of video diffusion, the main latent has changed, so the next round faces a new spatiotemporal state, requiring continued computation for the video subject.

Therefore, the cost of Sora is difficult to significantly reduce by “reusing history.” It’s more like a visual effects shot undergoing multiple rounds of processing, where each round must handle an entirely changed frame. The longer the video, the higher the resolution, and the more samples taken, the heavier this generation path becomes.

Precisely because each task is sufficiently heavy, Sora’s GPU utilization may appear very high. Large matrix computations keep the Tensor Cores busy for extended periods, resulting in GPU usage that remains nearly constant on monitoring charts.

High utilization only indicates that the chip has been continuously working, not that it has delivered many tasks per unit of time. If a single video occupies a group of GPUs for an extended period, the utilization may appear excellent, but the GPU-seconds consumed per request will still be high.

Video serving is still hindered by shape differences. Variations in duration, resolution, and aspect ratio result in different tensor shapes. To improve batch efficiency, the server groups requests with similar dimensions into the same bucket. Waiting a bit longer can yield a fuller batch, but increases queue latency; executing immediately reduces wait time, but the batch may remain underfilled.

Therefore, most of Sora's computational cost is already locked into the generation path of each individual video. The number of sampling steps can be reduced, latents can be further compressed, the model can be distilled, and kernels can be further optimized, but what the scheduler can primarily adjust is “how to schedule these heavy tasks”—it’s hard to change the fact that “a single video inherently requires substantial continuous computation.”

This is also the entry point to understanding Codex. Codex is similarly expensive, but instead of placing all the cost into a single continuous computation block, it breaks the task into many stages that can be paused, resumed, and recombined.

The user gives Codex the instruction "fix this bug," and the task does not end with a single model call. The agent may first read the repository, prompt the model to determine the next step, then execute a shell command; after receiving an error, it adds the logs to the context and calls the model again; subsequently, it modifies the code, runs tests, and continues reasoning based on the new results.

So a Codex task is closer to the accumulation of many roundsPrefill + Decode + Tool, where after each tool call, the context seen by the model in the next round is often richer than in the previous one.

At the beginning of the task, the model may only have the user's request and a small amount of code. After running for some time, additional files, diffs, terminal outputs, test logs, and tool results continuously enter the prompt.

What the user ultimately sees may be just a few hundred words of completed instructions, but the content processed internally by the GPU could already be extremely large. The pressure of Agent token consumption is hidden within this ever-growing trail of work.

If the entire history is reprocessed in each inference round, long tasks will quickly be slowed down by repeated prefilling, so prompt caching is crucial for Codex.

Assume an agent already has 100K tokens of context, and tool execution adds only 3K tokens of logs. If the stable prefix earlier in the sequence hits the cache, the additional computation for this round focuses primarily on the latter portion; however, if changes to the beginning of the prompt cause a cache miss, the system may once again face a heavy prefill.

An important change has occurred: the number of logical tokens no longer directly represents actual GPU costs. Both requests show 100K input tokens, but one consists mostly of cached content while the other requires recomputation—resulting in vastly different GPU workloads. Agent load therefore depends on the rate of context growth, cache hit rates, and how many times a task re-enters the model.

After entering single inference, prefill and decode have different hardware requirements. Prefill processes many input tokens at once, resulting in larger matrix dimensions and more easily forming compute-heavy workloads; decode generates only a few tokens per sequence per step but repeatedly accesses model weights and KV cache, thus relying more heavily on HBM bandwidth and concurrency scale.

This means that if decode runs a single sequence independently, it would be extremely inefficient. The model weights remain large, and generating a single token still requires a full forward pass. Only by batching multiple sequences together on the server side can a single weight access advance more requests simultaneously.

However, increasing the batch size further is limited by the KV cache. The longer each sequence’s context, the more HBM it consumes. After increasing the number of agents, the GPU may still have computational headroom, but the memory may already be full of active states.

Designs like PagedAttention reduce GPU memory fragmentation by managing the KV cache in a paginated manner, effectively increasing the number of active sequences a GPU can accommodate simultaneously.

Tool calls further distribute the load on Codex. When the Agent runs tests, compiles code, or waits for I/O, the GPU no longer needs to work on it—CPU, containers, and the file system take over. Once the results return, the Agent proceeds to the next inference round.

An Agent running for 60 minutes does not mean it continuously occupies the GPU for 60 minutes. Its task time is split into model computation and external execution, giving the scheduler a level of flexibility that Sora struggles to provide: when an Agent is running a tool, the GPU can immediately serve another sequence.

Of course, this also creates new VRAM issues. Should the waiting tool's Agent retain the KV cache? Retaining it allows faster recovery but occupies HBM long-term; evicting it frees up space, but the task incurs recovery costs when it returns. The more Agents there are, the more this trade-off resembles an operating system managing a large number of processes that sleep and wake up.

At this point, the difference between Codex and Sora is no longer about which is “heavier,” but whether the computational cost has been decomposed. Sora concentrates its compute power along a single continuous generation path, while Codex distributes its compute across multiple stages. Because it has been decomposed, Codex can move to the next level of optimization: allowing the scheduler to determine how these stages share the same batch of GPUs.

During online inference of large models, weights typically remain permanently resident on GPUs, and tensor parallelism, node communication, and caching states must be maintained. Therefore, competition for resources between Sora and Codex occurs primarily at the fleet level: some GPUs are permanently allocated to the video serving pool, while others are permanently assigned to the LLM pool, with the upper-level capacity system deciding where to scale up or down.

The real complexity occurs inside the Codex pool. Imagine the system simultaneously handling 200 Agent sequences, some decoding, others waiting for tools, and dozens returning from tool environments that need to process new long contexts. The scheduler faces constraints not only on FLOPs, but also on HBM capacity, memory bandwidth, KV cache residency, and latency budgets.

Continuous batching first addresses the utilization issue of decode. Traditional static batching ties a group of requests together, so even after short sequences finish, the remaining long requests still occupy the batch.

Continuous batching dynamically swaps sequences at the token iteration level: once a sequence is complete, it is removed, and new requests are immediately added. The larger the batch, the more sequences can be processed in a single model computation round, allowing the costs of model weight access and memory bandwidth to be more effectively amortized.

But here we quickly hit the memory wall. Large numbers of long-agent KV caches continuously occupy HBM, and a GPU’s memory may already be full before its Tensor Cores are fully utilized. At this point, adding more compute power is meaningless—the real bottleneck is cache capacity and memory management.

Another conflict exists between prefill and decode. Suppose dozens of sequences are steadily being decoded, when an Agent returns with a new context of 100K tokens requiring a large prefill. If this prefill occupies a prolonged execution window, the TPOT of nearby requests will noticeably degrade.

Chunked prefill splits long inputs into smaller chunks to interleave prefill and decode phases; an even more advanced approach is to separate prefill and decode onto different GPU pools.

The reason is that the two stages inherently favor different hardware bottlenecks: prefill is more compute-throughput oriented, while decode relies more on HBM bandwidth, KV cache, and consistent per-token latency. Separating them allows resources to be configured independently according to each stage’s specific needs.

This shows that the core of Agent serving has moved beyond simply “making the model kernel faster.” Much of the capacity improvement comes from reorganizing when and where tasks run, which states are worth keeping in GPU memory, and which inputs should be included in the current batch.

Therefore, GPU utilization is no longer sufficient here. The capacity team needs to monitor GPU-seconds per task, TTFT (time to first token, which determines user-perceived responsiveness), TPOT (time per output token, which determines how fast the model "speaks"), queueing latency, prefix cache hit rate, KV cache occupancy, and SLO goodput (effective throughput, representing the compute power that can actually be monetized).

Together, these metrics answer one question: How many effective tasks can a GPU sustain per hour within acceptable user latency?

This is where Codex's schedulability shines. The stable prefix reduces redundant prefill, decode supports continuous batching, KV cache can be paged and evicted, and agents can yield the GPU while waiting for tools. Its workload is highly fragmented, but these fragments can be reordered by the scheduler.

This naturally pushes the question to the resource layer: if the same batch of GPUs can interleave service for more long-term agents, then one hour of GPU time could actually support more than one hour of work.

Suppose a Codex Agent runs for 60 minutes from task initiation to completion, but only a portion of that time is spent on model prefill and decode; the rest is used for compilation, testing, reading/writing files, or waiting for tools. The exact proportion varies by task, but the structure is critical: the agent’s wall-clock time is not one-to-one with GPU compute time.

If there are many agents in the system, they won’t all require GPU resources in the same second. Some are in prefill, others in decode, some running tests, and others waiting on the file system. As long as the scheduler can interleave these phases, a limited number of GPUs can sustain far more active workflows than the number of GPUs available.

This relationship can be roughly understood as: Agent-hours depend on GPU-hours, model inference utilization, and scheduling efficiency. The more tool execution time, the larger the batch size, and the higher the cache hit rate, the more Agent wall-clock work one hour of GPU can support.

This directly changes the meaning of adding new GPUs. Adding a batch of GPUs to Codex doesn't just make individual tasks faster—it may also allow the system to sustain more agents simultaneously. An engineer can launch multiple tasks in parallel: one to modify the backend, one to add tests, one to handle another repository—so long as these tasks have no strong dependencies, machine workload time can scale in parallel.

Sora's capacity curve is more direct. The large wall-clock time of a video inherently involves running diffusion on the GPU, tightly coupling individual tasks with GPU utilization. Adding more GPUs can directly increase video throughput, but it's difficult to significantly widen the gap between GPU hours and video computation time.

A single software task in Codex moves between GPU, CPU, containers, file systems, and tool environments. The GPU handles model inference, while other systems manage execution, and multiple Agents share inference capacity through a scheduler. As a result, the GPU transforms from a mere generation device into a scarce "thinking resource" within the entire Agent system.

This is why Codex, despite being able to consume large amounts of compute,更容易获得新的 compute. OpenAI needs to look not just at the cost of a single inference, but also at whether new capacity can be quickly converted into additional parallel workloads.

When a batch of GPUs can support more long-term agents, and these agents can continuously receive new software tasks, resources easily continue to flow in this direction.

The technical implication of Altman’s statement thus becomes clear. Sora’s substantial computing power is locked into a single generation path, while Codex’s computing power is divided into multiple interleavable stages. Both are equally expensive, but their resource return curves differ.

The resource transfer instructions for Sora and Codex introduce a new variable to AI products that directly impacts expansion speed: workload architecture.

Using the same expensive GPUs, one type of task locks substantial computing power into a single generation path, while another type can interleave the same batch of inference capacity across more workflows through caching, batching, tool execution, and scheduling—resulting in naturally divergent resource curves.

So in the future, some seemingly low-level issues will become increasingly close to product decisions: where to place the KV cache, how to split the prefill, how thick to make the decode batch, whether to evict cache for waiting-agent tools—these choices will ultimately determine how many tasks a batch of GPUs can sustain simultaneously.

The difference between Sora and Codex is not just the difference between video and code.

They are competing for the same hour of GPU power—just how much work can it handle?