OpenAI Launches New Benchmarks for AI Programming in Low-Resource Languages

OpenAI has released two new benchmarks targeting "low-resource programming languages": MultiLangBench requires models to implement a SQLite-compatible database engine from scratch in 20 different languages; another benchmark requires models to migrate COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving original behavior. Official results show that GPT-6 Astra achieved an average pass rate of 87.4% on MultiLangBench, and GPT-6.1 Sol reached 97.9% accuracy in COBOL migration under high-reasoning conditions. This indicates that AI programming capabilities are extending beyond mainstream languages like Python and JavaScript to legacy technology stacks still in use by banks, governments, and research institutions. However, these results come from benchmarks designed and run by OpenAI; an 87.4% pass rate reflects the proportion of test cases passed, not that the model has produced a production-ready SQLite implementation, and a 97.9% accuracy rate does not mean banks can directly migrate core systems without human review.

Article author, source: OpenAI

In the field of AI programming, low-resource languages refer to those with relatively limited publicly available code, documentation, and Q&A resources for models to learn from—not because they lack real-world value.

Python and JavaScript have vast open-source ecosystems, allowing models to access large amounts of code, error reports, and fix records during training. In contrast, publicly available materials for COBOL, Fortran, Visual Basic, and OCaml are far more limited, with much of their code still confined within enterprise systems.

Meanwhile, these languages continue to perform critical tasks. According to IBM data cited by OpenAI, COBOL supports over 40% of online banking systems, 80% of offline credit card transactions, and 95% of ATM transactions; Fortran is still used for weather forecasting and nuclear reactor safety analysis.

Maintaining these systems is challenging not only because the syntax is outdated. The original developers may have left, documentation may be incomplete, and the actual business rules may exist only within the code. Reasons for truncating decimals or handling non-existent account records might be undocumented, yet they directly impact financial calculations.

Therefore, what AI truly needs to prove is not whether it can translate COBOL into Python that looks more modern, but whether the migrated code still executes the exact same business rules.

OpenAI released MultiLangBench for this purpose. It requires models to build a SQLite-compatible database engine using a specified language during an extended programming task.

The candidate program receives an initial database file and an SQL query; after execution, it produces a modified database. The evaluation system then invokes the official SQLite implementation to verify whether the database content matches the expected result. By using the database file as a common interface, the same set of functionalities can be compared across languages.

Tests cover 20 languages, including C, C++, COBOL, Fortran, Java, JavaScript, Lean, OCaml, Python, Rust, Visual Basic, and Zig, evaluating the model’s ability to organize codebases, manage state, handle errors, add new features, and avoid breaking existing functionality. Test cases are adapted from SQLite’s existing test suite.

At maximum reasoning intensity, GPT-6 Astra achieved an average test pass rate of 87.4%, compared to 26.8% for GPT-5.6 Sol. OpenAI stated that Astra showed improvements across all 20 languages:

COBOL increased from 10.9% to 72.1%, OCaml from 25.6% to 85.8%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. Nineteen out of twenty languages exceeded 83%, with COBOL remaining the most difficult.

(Original text contains an interactive module with 20 language options, allowing you to view each model's performance and cost for each language.)

These numbers must be understood correctly. The 87.4% represents the average pass rate across test cases and does not mean the model has an 87.4% probability of generating a database that is complete, stable, and fully compliant with all SQLite specifications. The remaining approximately 12.6% of test failures may still correspond to issues such as transaction handling, error management, or data consistency—problems sufficient to prevent production deployment.

It demonstrates that the model can continuously build more complex software in unfamiliar languages, not that SQLite can already be rewritten by AI in a single attempt.

OpenAI also tested the lower-cost GPT-6 Luna, which achieved an average pass rate of 40.3% across 20 languages, higher than GPT-5.6 Sol’s 26.8%.

The average API cost for Luna is approximately $0.77 per sample, compared to $3.48 for GPT-5.6 Sol, representing a reduction of about 78%. However, its performance improvements have not reached the level of Astra: Luna achieves only 17.1% on COBOL, 31.4% on Fortran, and 41.6% on Visual Basic.

This set of results presents two different approaches: Astra uses more reasoning resources to maximize the completion of complex tasks, while Luna processes more samples at lower costs, making it ideal for batch experimentation, preliminary analysis, and low-risk auxiliary tasks.

The so-called cost is merely an estimated API fee from the evaluation and does not include the manpower required for engineer reviews, testing environments, business acceptance, or bug fixes. For legacy systems, model invocation fees are often not the largest cost in migration projects.

The second evaluation requires the model to migrate a COBOL program to C++, Java, Kotlin, Python, or Rust.

The focus of the test is not whether the translated code is clean, but whether the observable outputs of the new and old programs are byte-for-byte identical. Each migration result must pass all hidden tests to be considered successful.

OpenAI specifically cited the example of fixed-point decimals: COBOL may by default simply truncate extra decimal places unless the program explicitly requests rounding. If the model arbitrarily changes this to rounding based on modern programming practices, the program may still run correctly, but the financial results will have changed.

At low inference intensity, GPT-6.1 Sol achieves an 86.6% transfer accuracy, compared to just 5.4% for GPT-5.6 Sol; the former also estimates API costs 39% lower. After increasing inference intensity, GPT-6.1 Sol reaches 97.9%, bringing this evaluation close to saturation.

The 97.9% here refers to "the proportion of programs that fully pass the hidden test cases in OpenAI's evaluation," not "that 97.9% of code in any COBOL system can be safely and automatically migrated."

Real banking systems may connect to mainframes, proprietary databases, batch schedulers, and internal communication protocols, and may include edge cases not covered by test scenarios. OpenAI has also explicitly stated that production migration still requires expert review and retesting against actual business rules and dependencies.

OpenAI has uploaded the evaluation environment, test suite, and language-specific results for MultiLangBench to GitHub, allowing developers to review the test content and rerun them with other models or agent frameworks.

This is more valuable than simply displaying a score sheet: outsiders can observe exactly which SQL functionalities the model fails on and compare whether scores are affected by prompts, tool permissions, reasoning budget, and runtime.

However, the official article only announced the public release of MultiLangBench, without disclosing that the complete program samples for COBOL migration evaluation and the hidden tests are also now available. The 97.9% result for the latter cannot yet be fully inspected or reproduced by external parties, unlike MultiLangBench.

Both evaluations were also designed by OpenAI and are primarily intended to compare OpenAI’s own models. They have not yet covered large real-world enterprise codebases, long-term maintenance costs, security vulnerabilities, performance degradation, or the amount of time required for human review. Therefore, the results primarily demonstrate progress on controlled tasks, rather than serving as definitive validation that enterprise software modernization has been solved.

Over the past two years, demonstrations of AI programming products have mostly focused on creating websites, mobile apps, and Python tools. These scenarios are intuitive but not the most costly software challenges for large enterprises.

What banks, insurance companies, governments, and research institutions truly struggle with are decades-old codes that no one dares to modify lightly. These systems may not be cutting-edge, but they contain the rules essential for account processing, tax calculations, supply chain management, and infrastructure operations.

The significance of MultiLangBench lies precisely here: whether a model’s capabilities can transcend the most common training languages and apply software engineering methods to technology stacks with scarce resources, outdated tools, and higher costs of errors.

If this capability continues to improve, the earliest change AI may bring is not “delete all COBOL with one click,” but rather assist engineers in reading unfamiliar code, supplementing tests, restoring missing documentation, identifying business rules, and then systematically completing verified migrations module by module. Human experts will still be responsible for deciding which behaviors must be retained and when it is truly appropriate to replace legacy systems.