OpenAI’s self-designed inference chip, Jalapeño, outperforms NVIDIA in efficiency.

Woke up to a flood of news about OpenAI.

First, X user Leo@synthwavedd exclusively claimed: OpenAI has completed a new pre-training, codenamed "Bel," with an alleged parameter scale exceeding 10 trillion, potentially serving as Doug's successor and forming the foundation for Astra/GPT-6...

After searching around, no accurate information was found. But it is certain that OpenAI’s first in-house developed inference chip has delivered results.

Just now, OpenAI released the latest test results for its first in-house inference chip, Jalapeño: a major breakthrough. Designed specifically for large language model inference, the chip achieves both higher throughput and lower latency through a novel architecture, all while maintaining high energy efficiency. In multiple model tests, it outperformed NVIDIA’s GB200 and GB300 systems in efficiency.

Shortly after, OpenAI CEO Sam Altman also said on X: "We built a chip that's very fast."

Note that Jalapeño, as OpenAI’s first in-house designed inference chip, is not intended for training the next generation of GPT models, but rather optimized for the model’s deployment and inference phase.

In simple terms: the training phase teaches the model capabilities, while the inference phase enables the model to quickly respond to user requests. Jalapeño primarily addresses the second issue.

OpenAI states that traditional hardware systems often require a trade-off between two metrics:

• Higher throughput: Processes more requests per unit of time;
• Lower latency: Provides users with faster responses.

For example, large-scale batching can improve overall efficiency but may increase the wait time for individual requests; conversely, pursuing ultra-low latency might compromise resource utilization. Jalapeño aims to optimize both simultaneously within a single architecture.

In terms of specific performance, OpenAI stated that Jalapeño outperforms NVIDIA GB200 and GB300 systems across multiple large model inference benchmarks, including the OpenAI GPT-OSS 120B model. DeepSeek R1 670B; Moonshot AI Kimi K2.5 1T.

Particularly noteworthy is DeepSeek R1 and Kimi The appearance of K2.5 suggests that Jalapeño is not optimized solely for OpenAI's own models, but can adapt to various large model workloads.

Interesting—Jalapeño can be translated as "Mexican pepper," and immediately after this message was posted, it sparked widespread jokes and discussions among netizens.

Are you planning to name your chip after chili? I can’t wait to try it.

For OpenAI, the launch of Jalapeño indicates that the company is attempting to integrate the entire chain from models to software, chips, and data centers.

Additionally, it is worth noting that OpenAI’s Jalapeño chip was developed in collaboration with Cerebras on the hardware side.

Tibo posted: "This capability is made possible by our deep collaboration with Cerebras and their unique hardware architecture. In the future, this partnership will further push the boundaries of the 'ultra-fast' experience."

I am very excited for both sides to continue collaborating, pushing the limits on the Cerebras platform to explore how we can run our most powerful models at the fastest speeds and deliver this experience to our most performance-demanding customers.

In fact, OpenAI is not the first AI company to develop its own chips. Many major AI companies have already entered the chip-making space—for example, Google launched the TPU, Amazon developed Trainium and Inferentia, Microsoft advanced Maia, and Meta has also invested in developing its own AI accelerators.

The underlying logic is very similar: general-purpose GPUs are flexible but not optimized for all AI workloads. For companies running massive volumes of AI requests daily, designing chips tailored to their own models, software systems, and service approaches presents an opportunity to further reduce costs...

Now let’s take a closer look at Jalapeño’s capabilities.

Jalapeño’s core advantage is performing more AI work with the same power consumption while delivering faster responses. Existing systems typically require a trade-off between throughput and latency, but Jalapeño aims to achieve higher throughput and lower latency simultaneously with a single architecture.

OpenAI emphasized that this advantage is not limited to its own models but also extends to externally developed models, demonstrating that Jalapeño is a more general-purpose reasoning architecture.

In GPT-OSS 120B, DeepSeek R1 670B and Kimi On three public models—K2.5, 1T—Jalapeño achieves 1.5 to 1.9 times higher AI workloads per watt at peak throughput, reduces end-to-end latency to 1/1.7 to 1/3.6 of competing systems; for highly interactive workloads, performance advantages reach 2.1 to 4.1 times.

In GPT-OSS 120B, DeepSeek R1 670B and Kimi On K2.5 1T, Jalapeño achieves peak throughput per watt that is 1.9x, 1.7x, and 1.5x that of existing best-in-class systems, respectively.

Among them, the largest Kimi On K2.5, Jalapeño achieves approximately 1.5x higher peak performance per watt and reduces end-to-end latency by approximately 3.4x.

At Kimi On the K2.5 1T, as single-user decoding speed increases, Jalapeño's watts-per-throughput advantage over GB300 expands from a peak of 1.5x to as high as 56.1x.

OpenAI did not focus solely on comparing the peak performance of individual chips, but instead prioritized a key metric: how much useful AI work can be accomplished per unit of power while meeting user and interactive agent latency requirements.

The reason is also related to the Agent. Agents often need to complete multiple steps in sequence, and even a small latency in a single request can accumulate throughout the entire task execution.

This is also where Jalapeño shows its most noticeable performance in low-latency ranges. With DeepSeek For example, with R1, Jalapeño achieves 12,258 mixed TPS/kW per kilowatt, compared to GB300’s 118, representing an approximately 104.3-fold improvement.

At DeepSeek On the R1 670B, Jalapeño achieves approximately 1.7 times the throughput per watt of the GB300; at the same decoding speed, this advantage can expand up to 104.3 times.

OpenAI tested using SemiAnalysis’s public benchmark, InferenceX, and compared it against today’s leading commercial systems. Jalapeño achieved superior performance-per-watt and latency combinations across three public models, from high-throughput services to highly interactive, low-latency scenarios, placing it on the Pareto frontier.

At DeepSeek At various operating points of the R1 670B, Jalapeño achieves a superior balance between throughput per watt, interaction speed, and end-to-end latency, positioning itself on the Pareto frontier.

The Jalapeño has a rated power consumption of 700W, but during this test workload, the actual sustained power consumption remained at 550W or below.

Jalapeño was designed from the start around current and future large language models, particularly interactive agents.

Different stages of LLM inference have different bottlenecks: Prefill is more dependent on computational power, while Decode is more limited by memory bandwidth, and data movement between cores and chips also increases latency.

Therefore, OpenAI co-designs chips, memory, networking, software, and rack-level systems. Model states, including the KV Cache, can be explicitly placed and kept as local as possible, enabling Jalapeño to simultaneously handle both Prefill and Decode phases and adapt dynamically to changes in their workloads.

OpenAI believes this is a key characteristic of agent workloads.

Interestingly, AI is not only the target service of Jalapeño but also directly involved in the development of this chip.

By leveraging models at different stages, the OpenAI team completed the journey from initial design to tape-out in just nine months. The models were used to explore various implementations, shorten design, measurement, and verification cycles, and assist in optimizing the chip’s arithmetic circuits.

Jalapeño was also designed as a clear, predictable programming goal for both humans and models. Using the GPT-Astra-powered Codex, the team enabled high-performance operation of three open-weight models not originally included in Jalapeño’s initial production plan—all within two months.

In certain GPT-OSS Attention and MoE modules, the implementations generated by Codex are 1.5 to 1.8 times faster than the original human-expert handwritten versions.

OpenAI plans to begin deploying Jalapeño on its own computing infrastructure by the end of 2026. The team is currently conducting production validation, refining the software, preparing for large-scale operations, and continuing to verify performance across additional models.

Subsequent products have also been launched; Gen 2 is in advanced development, and Gen 3 has already begun taking shape.

OpenAI summarized the efficiency improvements brought by Jalapeño into three tiers: Ultra-fast mode achieves the efficiency previously only possible with Fast mode; Fast mode achieves the efficiency previously only possible with Batch mode; and Batch mode itself has further improved efficiency.

However, developing in-house chips does not mean OpenAI is preparing to move away from NVIDIA.

OpenAI has clearly stated that meeting the growing demand for AI requires more computing power from all available sources, and the company will continue to deploy NVIDIA and other partners' accelerators at scale, covering both training and inference workloads.

What do you think? Feel free to comment and share your thoughts!

Reference link:

https://x.com/OpenAI/status/2092300846675505602

https://x.com/sama/status/2092339694210040187

https://openai.com/index/jalapeno-first-results/

https://techcrunch.com/2026/08/25/openais-jalapeno-chip-is-built-for-fast-inference-at-scale-benchmarks-show/

This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), authored by Machine Heart focused on AI, edited by Shan Hui and Youli.