OpenAI Says GPT-6.1 Sol Went From 5.4% to 86.6% on COBOL Migration, but That Test Isn’t Public and Its Open Benchmark Grades Just 272 Tests

OpenAI says its newest models have gone from nearly useless to nearly perfect at moving old COBOL programs into modern languages. In a post published on 19 October 2026, the company reported that GPT-6.1 Sol passed 86.6% of its COBOL migration tasks at low reasoning effort, against 5.4% for GPT-5.6 Sol, and reached 97.9% at higher effort.

It also published a second benchmark, MultiLangBench, where models build a SQLite-compatible database engine from scratch in 20 languages. GPT-6 Astra averaged 87.4% there, up from 26.8% for GPT-5.6 Sol. OpenAI’s announcement on X had passed 740,000 views by Tuesday morning IST.

The jump is real on OpenAI’s own numbers. But the COBOL test that produced the headline figure has not been released, and the open benchmark grades models on a small slice of its test suite. We read the OpenAI post, the MultiLangBench repository and the current API price list to see what holds up.

What OpenAI actually measured

OpenAI built two evaluations for what it calls low-resource languages: COBOL, Fortran, Visual Basic, OCaml and others with far less public code than Python or JavaScript. The company says the work came from developers and enterprise customers telling it that coding models struggle with these languages.

The first test, MultiLangBench, asks a model to build a working SQLite clone in a named language. The second asks a model to migrate existing COBOL programs to C++, Java, Kotlin, Python or Rust without changing what they do.

Evaluation	Model	Score	Baseline (GPT-5.6 Sol)	Cost note from OpenAI
COBOL migration, low effort	GPT-6.1 Sol	86.6%	5.4%	61% of GPT-5.6 Sol’s API cost
COBOL migration, higher effort	GPT-6.1 Sol	97.9%	Not given	Not given
MultiLangBench, 20-language average, max effort	GPT-6 Astra	87.4%	26.8%	Not given
MultiLangBench, 20-language average, max effort	GPT-6 Luna	40.3%	26.8%	$0.77 a sample vs $3.48

On MultiLangBench, Astra’s per-language gains include COBOL rising from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4% and OCaml from 25.6% to 85.8%. OpenAI says Astra cleared 83% in 19 of the 20 languages. COBOL was the one it did not clear.

Why COBOL migration is a hard test to pass

The migration test is stricter than “does the new code run”. OpenAI says hidden tests require byte-for-byte parity on observable output, and a migration only counts if every test passes.

Its own example explains why that matters. COBOL’s fixed-point arithmetic drops extra decimal digits unless rounding is explicitly requested. A Java or Python rewrite that rounds instead can run cleanly and still produce different financial results. Rules like that, and how a program treats a missing record, are often business requirements that nobody wrote down anywhere except the code.

OpenAI cites figures that COBOL supports more than 40% of online banking systems, 80% of in-person card transactions and 95% of ATM transactions. Those are third-party estimates OpenAI links to, not new OpenAI data.

What we found in the MultiLangBench repo

The MultiLangBench repository is public under Apache 2.0. Its commit history shows the eval was first added on 29 September and renamed to MultiLangBench on 19 October, the day of the announcement. It had only a handful of stars when we checked.

The README is more specific than the blog post. The model must implement SQLite 3.40.1 using only its documentation, with no internet access and every trace of SQLite removed from the machine. Tests come from SQLite’s TCL test suite. Of about 51,000 tests, roughly 24,000 could be converted to the eval’s format.

Here is the detail the headlines skipped. Because running all 24,000 tests is expensive, the README says evals “should only use the 272 tests” whose names start with 00, 01 or 02. Running the full suite is allowed for high scorers but should be noted. OpenAI’s post does not say which set its published scores used, so the safest reading is the 272-test default.

That is still a meaningful sample of real SQLite behaviour. But 272 tests per language is small enough that a few passes either way can move a percentage noticeably, so treat close per-language gaps with care.

Two other gaps. OpenAI’s post says it is releasing results for each language, but when we checked the repo it held tests and scripts, not per-model results. And the README does not list the 20 languages, so outsiders cannot yet rerun the exact set behind the 87.4% average.

The COBOL migration test is not public

The more dramatic number, 5.4% to 86.6%, comes from the evaluation OpenAI did not release. The post describes the target languages and the byte-for-byte rule, but not how many programs it contains, how long they are, or where they came from.

That matters because a near-saturated score of 97.9% on an unknown set tells you the test is now too easy for this model. It does not tell you how the model handles a 30-year-old bank batch system with copybooks, JCL scripts and database calls around it. OpenAI says as much: production use “still requires expert review and testing against the system’s business rules and dependencies.”

The post also compares only against OpenAI’s previous model. There are no numbers for Anthropic’s Claude Opus 5.5, Google’s Gemini 4 Argon or open-weight models, so this is not evidence that OpenAI leads the field on legacy code. We saw the same pattern with Datadog’s own-benchmark claims for Scout 1.0, where the public test told a different story.

The cost math OpenAI did not spell out

OpenAI gave relative costs, not token counts. We compared them with the current OpenAI API price list, where standard short-context rates are:

Model	Input per 1M tokens	Output per 1M tokens
GPT-6 Astra	$10.00	$50.00
GPT-6.1 Sol	$2.00	$10.00
GPT-5.6 Sol	$4.00	$20.00
GPT-6 Luna	$0.10	$0.50

GPT-6.1 Sol’s input and output rates are exactly half of GPT-5.6 Sol’s. Yet OpenAI says the migration run cost 61% as much, not 50%. If the mix of input and output was similar, that suggests GPT-6.1 Sol used roughly 20% more tokens to get its far better result. That is our estimate, not an OpenAI figure, and caching could shift it.

Luna is more striking. Its rates are one fortieth of GPT-5.6 Sol’s, but OpenAI puts its MultiLangBench cost at $0.77 a sample against $3.48, about 22% of the cost. On the same assumption, Luna burned close to nine times the tokens. Cheap per token is not the same as cheap per finished task.

OpenAI gave no cost figure for Astra on MultiLangBench. At $10 input and $50 output per million tokens, it is the priciest of the four, so the 87.4% headline comes from the most expensive option. OpenAI’s price page also notes that GPT-5.6 Sol’s current rate is promotional through at least 21 November 2026, so the baseline could change.

This is not the first COBOL scare

In February, Anthropic said Claude Code could automate much of the analysis that makes COBOL modernization slow. Reuters reported IBM shares fell 13.2% that day, the steepest drop since October 2000.

IBM pushed back. Rob Thomas, IBM’s senior vice president for software, wrote that “translation captures almost none of the actual complexity,” according to CIO. His point is that mainframe performance comes from hardware, I/O and decades of tuning, not just the language. OpenAI’s test measures exactly the translation step he was talking about.

Confirmed vs not yet shown

Confirmed by OpenAI’s post, repo or price list	Not yet shown
GPT-6.1 Sol scored 86.6% (low effort) and 97.9% (higher effort) on OpenAI’s COBOL migration test	The migration test itself, its size and its program sources
Migration passes only if every hidden test matches byte for byte	Results from any outside team
MultiLangBench is public under Apache 2.0, with about 24,000 converted SQLite tests	Which test set OpenAI’s scores used (README default is 272)
GPT-6 Astra averaged 87.4% across 20 languages	The list of 20 languages and per-model results in the repo
GPT-6 Luna cost $0.77 a sample vs $3.48 for GPT-5.6 Sol	Token counts and Astra’s cost
All new models are available in the API now	Comparisons with Claude, Gemini or open-weight models

What it means for Indian developers

India’s large IT services firms run big mainframe and legacy modernization practices for overseas banks and insurers. If translation work gets much cheaper, the billable hours shift toward the parts OpenAI admits it cannot do: mapping business rules, building regression suites and signing off on parity.

For a developer, that makes testing skills worth more than raw COBOL syntax. Knowing how to capture a legacy program’s exact outputs, including rounding and edge cases, is what turns a model’s draft into something a bank will run.

If you want to try it, MultiLangBench is a cheap way to see how models handle your own language. Run the 272-test default first, budget for token use rather than per-token price, and remember that GPT-4o and GPT-4.1 are leaving the API on 14 January 2027, so older pipelines need a new model anyway. For a sense of how vendor benchmarks can flatter results, see our read of Google’s Harness2 paper.

FAQ

What did OpenAI announce about COBOL?

On 19 October 2026, OpenAI published two coding evaluations for low-resource languages. On its COBOL migration test, GPT-6.1 Sol scored 86.6% at low reasoning effort, up from 5.4% for GPT-5.6 Sol, and 97.9% at higher effort.

Can I run OpenAI’s COBOL migration test myself?

No. OpenAI released MultiLangBench, the SQLite-building benchmark, on GitHub under Apache 2.0. The COBOL migration evaluation was described but not released.

What is MultiLangBench?

It asks a model to implement SQLite 3.40.1 from its documentation in a chosen language, with no internet. Results are checked against the real SQLite. The repo has about 24,000 converted tests, but its README says evals should use a 272-test subset by default.

Is GPT-6.1 Sol cheaper for COBOL work?

OpenAI says the migration run cost 61% of GPT-5.6 Sol’s. GPT-6.1 Sol costs $2 input and $10 output per million tokens, half of GPT-5.6 Sol’s listed rate, which suggests it used somewhat more tokens.

Does this mean banks can drop COBOL now?

Not on this evidence. OpenAI itself says production migrations still need expert review and testing against business rules and dependencies, and no outside team has reproduced the result yet.