Google’s Open-Source Harness2 Lifts GLM-5 Past Gemini 3.5 Flash Without Touching Its Weights, but Every Task Runs at Least Four Times and AI Models Do the Grading

Google Research has open-sourced Harness², a framework that improves an AI agent by rewriting everything around the model (its prompts, tools, memory and skills) while leaving the model weights untouched. The code is on GitHub under Apache 2.0, and it has picked up more than 12,000 stars since the repository went up on 6 October 2026.

The headline claim from the accompanying arXiv paper is that a single improvement step lifted the open GLM-5 model by 11.5 points on WorkBuddy-Bench, enough to pass Gemini 3.5 Flash running on its stock setup. That is a real result. It also costs extra runs on every task, and the scores come from benchmarks where AI models do much of the grading.

We read the paper, the README and the experiment guide to separate what Google actually showed from what is circulating on X.

What Harness² actually does

In agent jargon, the “harness” is the system around a model that lets it act: the system prompt, guardrails, memory, tool instructions, subagents, skills, scripts and plugins. Harness² exposes those eight parts as editable files and lets the agent rewrite them while it works.

It runs in two loops. The first, which the paper calls task-level recursion, works on the current task in three moves: propose, probe and compose.

• Propose: the agent runs the task once with its current setup, then drafts two candidate edits. One targets general habits such as planning or tool use. The other targets knowledge for that domain.
• Probe: each candidate runs on the same task, and the system compares the new trajectory with the original.
• Compose: a composer step merges the edits the evidence supports, drops the rest, and runs the final version.

The second loop saves what worked into a library, so the next task starts with lessons from earlier ones. The authors, a team from Google Cloud AI Research with a Northwestern University co-author, say this happens “without ground-truth feedback”, meaning no answer key is used while the agent is improving itself.

The benchmark numbers, checked against the paper

Google tested two executor models, GLM-5 and Gemini 3.5 Flash, inside two existing coding agent harnesses, OpenCode and OpenAI’s Codex. Here are the WorkBuddy-Bench averages from Table 1 of the paper.

The last column adds the gain the paper reports for parallel recursion at a budget of five to the one-step score. On the other two benchmarks, one step added up to 4.5 points on JobBench and 11.8 percentage points on the partial pass rate of Harvey’s Legal Agent Benchmark (LAB).

The viral line about lighter models beating stronger ones comes straight from the paper: “A weaker model with an improved harness can thus outperform a stronger one with its original harness.” The evidence is GLM-5 at 75.7 after one step against Gemini 3.5 Flash at 72.4 on its base OpenCode setup. It is a fair reading, but it compares a tuned agent with an untuned one.

A few other results are worth knowing:

• When the same model acted as both worker and editor, all five models tested improved, from GLM-5 up to Claude Opus 5. GLM-5 gained 13.7 points on LAB that way.
• A harness tuned for Gemini 3.5 Flash and handed unchanged to GLM-5 raised its coding score from 58.5 to 75.8.
• Tool errors on JobBench fell from 14.4% to 9.1% after one step.
• Against stock Claude Code, Codex and CodeBuddy running Gemini 3.5 Flash, Harness² starting from OpenCode at a budget of five beat the strongest fixed harness by 4.6, 5.8 and 4.9 points on code, web and office tasks.

The catches the X posts leave out

First, it is not free. The paper’s own limitations section says Harness² “requires additional executions and editor sessions” and that a bigger budget “adds inference cost”. By our count from the method description, one step means at least four runs of each task: one reference run, two candidate probes and the composed final. At a budget of five in parallel mode, that is five reference runs, ten candidate probes and five composed runs, or 20 executions before editor sessions. The authors argue the trade still lands on the Pareto frontier for compute against score, and say rival method Meta-Harness used more compute for less.

Second, the judges are mostly models. WorkBuddy-Bench mixes deterministic checks with LLM and vision judge items, and the judge model there is Gemini 3.5 Flash, the same model as one of the two executors. JobBench is graded by Claude Opus 4.8 against a rubric, and LAB by Claude Sonnet 4.6. The paper says rubrics and expected answers are never shown to the agent, which helps, but self-grading by a sibling model is still a weakness.

Third, the test sets are small: 100 WorkBuddy tasks, 34 JobBench tasks and 100 LAB tasks. Several reported gains sit within a few points, and the paper’s own failure analysis lists nine regressions. In one case the selector picked an output scoring 55 over one scoring 95.

Fourth, Harness² does not win everywhere. Meta-Harness leads on LAB for Gemini 3.5 Flash under Codex, and on GLM-5 coding tasks under Codex it scored 76.8 to Harness²’s 74.9.

Finally, the README says plainly: “This is not an officially supported Google product.” It is a research release, not a Vertex AI feature.

Confirmed vs unconfirmed

Should you try it?

The repository ships a local demo that, according to the README, “requires no model credentials” and runs both recursion modes with deterministic editors. That is the safe first step. Model-backed runs need Linux, Python 3.12, Node.js 20 or later and the Google Cloud CLI configured for Vertex AI, and the README warns that they “incur API costs”.

Read the web access note before you point it at anything real. The README says executor network access is unrestricted and that Codex runs with its sandbox disabled. That is fine for a benchmark container, and a bad idea on a laptop holding production keys. Our recent coverage of the OpenCode console breach showed how fast leaked API keys turn into bills.

The bigger idea matters more than this release. Vendors keep shipping agents tuned on their own benchmarks, as we saw with Datadog Scout 1.0. Harness² is evidence that the scaffolding can move scores as much as a model upgrade, which means buyers should ask which harness produced any agent number they are shown.

What it means for Indian developers

The most useful finding for cost-conscious teams is that the open GLM-5 model gained the most. If a better harness closes much of the gap to a pricier hosted model, a startup in Bengaluru or Pune can test that before paying for a bigger model tier. Compare that with the pricing on Google’s Gemini 4 Argon before deciding.

The catch is that each improvement step multiplies runs, so a cheaper model can still cost more per task if you run a high budget. Measure total tokens per finished task, not price per million tokens.

The eight-component layout is also a practical checklist even if you never run Harness². Many teams in India building agents on OpenCode, Codex or Claude Code keep everything in one long system prompt. Splitting it into guardrails, memory, tool guidance and on-demand skills, as the paper does, is a cheap change you can make today.

FAQ

What is Google Harness²?

It is an open-source research framework from Google Research that improves an AI agent by editing its prompts, tools, memory, skills and other scaffolding during a task, while keeping the underlying model unchanged.

Does Harness² retrain or fine-tune the model?

No. The paper and README both say model weights stay fixed. Only the harness files around the model are edited.

Which models and tools does it support?

The release supports OpenCode and Codex as executors and ships setups for JobBench, LAB and WorkBuddy-Bench. The paper tested GLM-5 and Gemini 3.5 Flash as main executors, with extra tests on Claude Opus models.

Is Harness² free to use?

The code is free under Apache 2.0, and a local demo runs without model credentials. Real runs call paid model APIs and each improvement step runs a task several times, so expect higher token use.

Is it an official Google product?

No. The README states it is not an officially supported Google product and is not covered by Google’s open-source vulnerability rewards program.