OpenAI Launches Decisions API to Compete with TypeSafe's Jev

OpenAI made Decisions API available to all customers on Thursday after announcing the product at the company's developer day on September 29. OpenAI is entering a market for fast, inexpensive intelligence that recently brought TypeSafe to a valuation of $7.6 billion. A former researcher at OpenAI founded TypeSafe.

TypeSafe's Jev established a category of deep learning models that rapidly select among supplied options and assign confidence ratings to those choices. Developers found that these decision models, also called System One models, could substitute for large language models in various workflows with greater speed and substantially lower costs. Nikunj Handa, OpenAI's API product lead, said colleagues immediately identified many potential uses after seeing Jev. Handa's team adapted ChatGPT Luna for faster inference and lower costs instead of creating a specially calibrated decision model. Handa said Decisions API reduced processing times from one to three seconds to 100, 120 or 180 milliseconds. Handa said OpenAI might introduce specialized training to improve these capabilities in the future, but the current product uses Luna without that training.

OpenAI added voice and visual capabilities to the publicly available version. OpenAI also allocated inference resources geographically to reduce latency across regions worldwide. Engineers at early customers, including Box, Datadog, Ramp and Speak, replaced OpenAI's large language models with the API and achieved higher throughput. Andrew Hsu, CTO of AI language tutor developer Speak, said Decisions API reduced the interval between actions from 1.4 seconds to 0.13 seconds. OpenAI also improved the responsiveness of the company's Dots voice assistant using the product.

The criteria for a good decision model remain uncertain, with trade-offs among intelligence, cost, calibration and speed apparently varying by workload. TypeSafe CEO Diogo Almeida distrusts benchmarks and recommends testing models on users' own tasks. However, TypeSafe published rankings last week alongside the company's newest Jev release. The AIM System One Index and Artificial Analysis Decision Index generally favor Liquid's D1 Ultra and Base Lab's Elephant-1. Liquid released D1 Ultra last week. The latest Jev ranks seventh on the AIM System One Index and third on the Artificial Analysis Decision Index. OpenAI's Decision Model places eighth and fourteenth on those respective indices. However, TypeSafe's models currently receive the most queries on OpenRouter.

The newest Jev leads JevMan, a benchmark that tests decision models by having the models play PacMan. Opper AI developed JevMan. Felix Wunderlich, Opper AI's chief growth officer, said PacMan tests a continuous series of fast, consequential decisions. Wunderlich said models must choose a direction at each junction within a few hundred milliseconds.

OpenAI's arguably more capable model may gain an advantage from multimodality. Handa said Luna performs strongly on image classification and has demonstrated better calibration. Handa said OpenAI customers also want greater flexibility to use images as outputs from Decisions API. Handa suggested that further speed improvements could enable additional applications. Handa raised the possibility of reaching around 60 milliseconds, which could allow an API call for every frame of a video.

Faster and cheaper decision models can be expected to become more widely used, consistent with the prediction of the economist whose name inspired Jev. The economist predicted that a commodity becomes more ubiquitous as the commodity becomes cheaper. Handa said decision models had revealed many previously overlooked applications.