The competitive landscape for AI chips is undergoing profound changes. OpenAI has launched the Jalapeño chip focused on LLM inference, NVIDIA is combining GPUs with Groq LPUs to enable heterogeneous computing, and Google has split training and inference into two distinct chips: TPU 8t and TPU 8i. These three approaches reflect the differing hardware demands of training and inference: training emphasizes matrix computations and large-scale interconnectivity, while inference requires higher HBM bandwidth, larger SRAM, and shorter network paths. As the chip design requirements for these two workloads diverge further, FLOPS is no longer the sole metric—token cost is emerging as the new benchmark in AI hardware competition.

Article author and source: LeFeng.com

Token cost has become a new benchmark in the hardware competition for large models.

Embodied intelligence enters the smoky kitchen

A Robot's Work Diary Under a Million Visitors

The former Vice President of Covariant AI and Zhoupu Shuzhong directly summarized the role of LPU as filling the gap in Vera Rubin's low-latency decode region.

The chip design of Groq is also almost entirely centered around this. A single LPX rack contains 256 LPU chips, totaling only 128 GB of SRAM, a capacity that cannot be compared on the same scale as the HBM in GPU racks, but the aggregated SRAM bandwidth can reach 40 PB/s.

This design prioritizes "proximity." HBM can store many model states but is farther from the compute units; SRAM is expensive and difficult to scale in capacity, but data resides within the chip, enabling extremely high bandwidth and very low access latency.

Each step involves limited computation and frequent data access; placing data closer to the ALU will directly reduce the waiting time for the next token.

Groq has further reduced dynamic hardware control. The LPU is a deterministic execution architecture where instruction scheduling is primarily completed in advance by software. Each chip simultaneously serves as both a processor and a router, with the compiler coordinating compute and network resources together, even eliminating traditional hardware flow control and virtual channel mechanisms.

The trade-off is clear: this architecture is not as general-purpose as a GPU, and the on-chip SRAM cannot hold the full state of a large model. Therefore, NVIDIA did not let Groq 3 run the entire model independently, but instead built a more complex heterogeneous system.

Prefill is performed on the GPU, while most of the Decode is offloaded to the LPU; the Attention within Decode can return to the GPU. The GPU and LPU maintain their own KV Caches, primarily exchanging draft tokens, and overlap computation and communication using micro-batches. Since the LPU is a synchronous domain while the GPU and external KV Cache belong to an asynchronous system, NVIDIA has even incorporated FPGA as an asynchronous bridge between the two.

This architecture clearly illustrates how far AI chip specialization has come. It no longer just separates “training chips” and “inference chips,” but even different parts within a single decode operation can be executed on distinct architectures.

However, the data presented by NVIDIA also defines the boundaries of this approach: when the focus is solely on overall throughput and higher latency is acceptable, the Rubin GPU remains highly efficient; as requirements for individual user token speed increase, LPX gradually gains an advantage, and overall throughput efficiency decreases when more LPU are used.

The emergence of Groq 3 does not mean GPUs are being phased out for inference. It illustrates something else: it’s difficult for the same GPU to excel at both high throughput and ultra-low latency; by letting each type of hardware operate in its own optimal range, the system can more effectively widen the performance gap.

Google placed this cut at a higher level.

Training and inference use two different chip configurations.

Google introduced both TPU 8t and TPU 8i in the TPU 8 generation,t for training,i for inference. This distinction is directly reflected in the chip’s memory configuration.

During the Hot Chips live demonstration, the TPU 8t uses 6 groups of HBM, while the TPU 8i uses 8 groups. Google explained that inference requires more HBM per unit of computation, along with a higher proportion of SRAM, so the 8i allocates more resources to SRAM, memory capacity, and bandwidth.

This difference is worth contemplating. If AI chips were merely competing on matrix computational power, there would be no reason for the inference version to allocate so much die area and packaging resources to memory. The design of the TPU 8i suggests that Google has identified the bottleneck as having shifted to data supply.

During training, large batches amortize the weight retrieval cost across many tokens; during decoding, however, each generated token is minimal, yet weights and KV cache are still frequently accessed. Thus, the amount of HBM bandwidth required per unit of FLOPS differs between these two tasks.

Google has even applied this difference to network topology. The 3D Torus traditionally used by TPUs is better suited for training, emphasizing overall throughput in large-scale clusters. The TPU 8i supports BoardFly, with shorter network paths: BoardFly has a maximum of 7 hops, while the 3D Torus reaches 16 hops.

For training, after a few additional network hops, there is typically a large matrix computation, allowing communication time to be amortized. The computation window for each decode step is short, making network latency across a few hops more likely to directly fall within the token interval.

MoE makes this issue even more apparent. MoE allows a token to activate only a subset of experts, which is computationally efficient, but the router sends tokens to different experts. When these experts are distributed across different chips, while computation decreases, all-to-all communication increases.

Therefore, the TPU 8i also incorporates the Collective Acceleration Engine, which handles certain collective operations on the I/O die located near the network interface. This eliminates the need to first move data into the Compute Die and then process it through HBM, thereby reducing some internal chip data movement.

The resource allocation for the training version of the TPU 8t is clearly skewed toward the other side. Training requires massive FLOPS and a large scale-up domain to synchronize parameters and gradients. The TPU 8t Superpod can scale up to 9600 chips, featuring approximately 2 PB of shared HBM and 121 EFLOPS FP4 aggregate computational power. Google has also introduced the Virgo network to create a dedicated infrastructure for connecting larger-scale training workloads.

Google also mentioned a very practical chip design issue: dark silicon.

The chip area and power budget are limited. If a single chip simultaneously incorporates the large matrix computation resources needed for training, along with the additional SRAM, HBM, and low-latency networks required for inference, a portion of the circuitry will remain idle during either workload.

Since the two tasks require different resource ratios, it’s cleaner to build two separate chips.

From FLOPS to Token Economics

Putting the three routes together, the differences are actually clear.

OpenAI goes with Jalapeño, specializing chips at the LLM inference layer while retaining flexible scheduling of Prefill and Decode on homogeneous hardware; NVIDIA goes further down, splitting the stages of a single inference between GPU and Groq LPU; Google goes up, directly separating training and inference into two distinct TPU chips.

There is no mysterious new computational principle behind these paths—what changes is the allocation of resources. Training aims to allocate more transistors to matrix computation and large-scale interconnects, because high batch sizes spread out the data movement cost; low-latency inference requires higher HBM bandwidth, larger SRAM, better KV cache locality, and shorter network paths, because much time is spent waiting for data.

As the "chip recipes" required by the two types of workloads diverge further, the efficiency loss from using a single general-purpose chip to serve both becomes increasingly apparent.

This is why the key metrics in this round of hardware competition are changing. FLOPS remain important, but alongside them are now emerging Tokens/s/user, TBT, TTFT, Tokens/kW, HBM bandwidth, and network latency.

They describe the same thing: the computing power is already in place; the question is whether the system can continuously feed data in and deliver the generated tokens as quickly as possible.

OpenAI, NVIDIA, and Google currently have different lines drawn, and what will likely continue to change is where this line should be drawn.

Training and inference can be separated, prefill and decode can be separated, and within decode, attention can be further separated from other computations. The finer the separation, the easier it becomes to improve individual efficiency, but resource scheduling, KV Cache movement, and cross-hardware communication also become more complex.

Therefore, the challenge in the next phase of AI chip competition may no longer be simply creating a more powerful chip.

More challenging is deciding: which tasks warrant a dedicated chip, and which should remain on the same hardware, to achieve the lowest overall system token cost.