OpenAI's MultiLangBench has GPT-6 Astra build SQLite in 20 languages, COBOL included

What OpenAI published

OpenAI on 19 October 2026 published results from two new coding evaluations aimed at what it calls low-resource programming languages, such as COBOL, Fortran, OCaml and Visual Basic, which have far less public code and documentation for models to learn from than Python or JavaScript. In its blog post, OpenAI says its latest models are much better at building and migrating software in these languages, at lower cost.

The first evaluation, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in each of 20 programming languages. The second asks a model to migrate COBOL programs to a modern language without changing what they do. OpenAI has released MultiLangBench's test suite and setup on GitHub under the Apache 2.0 licence; it has not said it will release the COBOL test set.

The headline numbers: GPT-6 Astra passes 87.4 percent of MultiLangBench tests on average across the 20 languages, against 26.8 percent for the older GPT-5.6 Sol, and GPT-6.1 Sol migrates COBOL with 86.6 percent accuracy at low reasoning effort and 97.9 percent at higher effort, against 5.4 percent for GPT-5.6 Sol.

How MultiLangBench works

Building a database engine is a long job. OpenAI says the model must organise a codebase, manage state, handle errors and add features without breaking what already works, over an extended coding session. The score is reported as a pair: the share of tests passed and the API cost of getting there.

The trick that makes 20 languages comparable is to use database files as the interface. According to the repository's README, the target is SQLite version 3.40.1 and the model gets SQLite's documentation but no internet access. Each test has three parts: SQL that creates an initial database, a query the model's program must run against it to produce a modified database file, and a validation query that the real SQLite runs on the result. A test passes only if that output matches the expected string exactly; a crash or a 60-second timeout counts as a failure.

The tests were converted from SQLite's own TCL test suite. Of about 51,000 tests, 24,000 could be converted into this format. Because running all of them is expensive, the README says evaluations should use only the 272 tests whose hashed names start with 00, 01 or 02, and note it if they run more. Any trace of SQLite must be removed from the environment first, and a solution written in the wrong language, or a mix of languages, scores zero.

The scores by language

OpenAI gives four languages in detail, all at maximum reasoning effort. GPT-6 Astra went from GPT-5.6 Sol's 10.9 percent to 72.1 percent in COBOL, from 25.6 to 85.8 percent in OCaml, from 21.4 to 83.5 percent in Fortran and from 27.5 to 89.4 percent in Visual Basic. OpenAI says Astra scored above 83 percent in 19 of the 20 languages, with COBOL the hardest.

GPT-6 Luna, the cheaper model, also improved on GPT-5.6 Sol in every language, but by much less: 17.1 percent in COBOL, 34.1 in OCaml, 31.4 in Fortran, 41.6 in Visual Basic and 40.3 percent on average. OpenAI's selling point for Luna is cost, an estimated average of $0.77 per sample against $3.48 for GPT-5.6 Sol, about 78 percent lower. The post does not give a per-sample cost for Astra.

OpenAI did not list the other 16 languages in the text AI Lookout read, and it compares only its own models. There are no scores for Anthropic, Google or open-weight models, so MultiLangBench cannot yet say who leads, unlike outside boards such as Artificial Analysis's FrontierCode leaderboard.

The COBOL migration test

The second evaluation targets the job banks and insurers actually pay for: moving old COBOL programs to C++, Java, Kotlin, Python or Rust without changing behaviour. OpenAI says the hard part is that business rules hide in language details. Its example is COBOL's fixed-point arithmetic, which discards extra decimal digits unless rounding is requested; a rewrite that rounds instead can run without errors and still produce different financial results.

To catch that, each migration is checked against hidden tests that require byte-for-byte matching output, and a migration counts only if every test passes. On this evaluation GPT-6.1 Sol scored 86.6 percent at low reasoning effort, compared with 5.4 percent for GPT-5.6 Sol at the same setting, at 61 percent of the API cost; the post's summary describes that as 39 percent cheaper. At higher effort GPT-6.1 Sol reached 97.9 percent, which OpenAI says nearly saturates the test.

OpenAI does not say how many programs the COBOL set contains, how long they are or where they came from. AI Lookout has previously looked at GPT-6.1 Sol's pitch as a lower-cost model for serious work; these results are OpenAI's first detailed numbers on legacy migration.

Why these languages are hard for models

OpenAI says developers and enterprise customers told it that coding models often struggle in these languages. Documentation can be incomplete, the original developers may have left, and important business rules may exist only in the code. The stakes are high: the post cites figures that COBOL supports more than 40 percent of online banking systems, 80 percent of in-person card transactions and 95 percent of ATM transactions, and says Fortran is used in weather forecasting and nuclear safety analysis. Those figures come from outside sources OpenAI links to, which AI Lookout did not check.

Most popular coding benchmarks, such as SWE-bench, are dominated by Python repositories, so they say little about these languages. MultiLangBench uses the same task in every language, which lets a team see how far a model's skill drops outside the mainstream.

What is in the GitHub release

The MultiLangBench repository contains the converted SQLite tests, each in a folder named by a SHA-256 hash of its three queries along with an expected-output file, plus scripts to generate the golden outputs, check the hashes and remove SQLite from a test machine. It is licensed under Apache 2.0. Its commit history shows the eval was added on 29 September, the licence on 6 October and the project renamed to MultiLangBench on 19 October, the day of the announcement.

OpenAI's post says the release includes results for each language. When AI Lookout checked the repository on 20 October, its top level held only the licence, README, requirements file, scripts and tests, with no results file, and no harness for running a model end to end. Teams that want to reproduce OpenAI's numbers would have to build their own agent loop around the tests.

Caveats

Every number here comes from OpenAI testing its own models on evaluations it designed, which is the pattern AI Lookout's guide to reading benchmark claims warns about. OpenAI does not give confidence intervals, the number of runs, or how much time and compute each attempt used, and GPT-6 Astra's cost per sample is missing.

Passing SQLite's tests is also not the same as producing a database engine anyone should ship. The evaluation measures functional behaviour against 272 sampled tests by default, not performance, security or maintainability. OpenAI itself says production migrations still need expert review and testing against the system's business rules and dependencies.

Large volumes of AI-written code can also strain the people who review it, as the PostgreSQL 19 delay showed this month. A near-perfect score on a hidden COBOL test says nothing about the specific programs a given bank runs.

Why it matters

Legacy code is one of the clearest business cases for AI coding tools: large, old, poorly documented systems that few people can still maintain. A jump from 5.4 to 97.9 percent on behaviour-preserving COBOL migration, if it holds on real codebases, would change the economics of modernisation projects that can take years.

Publishing the SQLite suite under an open licence also gives competitors and researchers a common yardstick across 20 languages. The next tests are whether other labs report MultiLangBench scores, whether OpenAI releases the per-language results and COBOL set it describes, and whether customers see similar gains on their own code.