Skip to content

AWS Let AI Agents Write Its New Emulator With No Human Review, and Its Own Scores Pass Just 56 Percent of the Tests

AWS says AI agents wrote its new local AWS Emulator without human code review. The published fidelity scores show 56.5 percent of checks passing, and AWS itself says not to use it in production.

Diagram showing three teams of AI agents building AWS Emulator

AWS released AWS Emulator on 11 November 2026, a free tool that runs a stand-in for AWS APIs on a developer’s own machine. The company says autonomous AI agents wrote every line, and no AWS engineer reviewed the code.

Coding agents test code in tight loops, and every loop against real AWS costs money and time. AWS is selling a way to move that loop onto a laptop, with no AWS account and no cloud bill.

The catch is in the receipt. The release’s own scores show 24,243 of 42,937 checks passing, or 56.5 percent. This is not a finished emulator, but a fast-moving one with a public list of what breaks.

Quick answer: AWS Emulator is a free, CC0-licensed local emulator covering 438 AWS API models, written and maintained by AI agents. It is useful for local tests and CI, but AWS says not to use it for production or security testing.

What AWS actually shipped

AWS Emulator has no graphical interface. It serves an AWS-style endpoint on 127.0.0.1:9297 with a self-signed certificate, and you point the AWS CLI or any SDK at it. Requests go to the emulator, so you need no AWS account.

Linux gets RPM or Debian packages for x86_64 and aarch64. macOS and Windows run it in Docker from an AWS-supplied Dockerfile. Rust tests can embed it, though a full build can need around 100 GB of disk.

The code is on GitHub under CC0, alongside a separate fidelity test suite. Release v1.0.322 shipped on 11 November. AWS charges no extra fee; you pay only for real AWS resources when you deploy.

Why AWS built it

Senior principal engineer Matt Goldberg started with a weekend experiment. He gave Amazon’s Kiro agent three steps: write tests against the AWS CLI, iterate until they pass against real AWS, then build an in-memory version that passes the same tests. Kiro produced Amazon SQS in 15 minutes, and by Sunday it had a rough version of the rest of AWS.

Anthropic’s C compiler demo was the spark. Goldberg’s point is that agents do well with a tight spec and a reference implementation, and AWS had both: its API models and the real cloud.

The product follows the same logic. Agents test by running code repeatedly, and real AWS adds latency and cost. A local endpoint removes both, which is the loop behind OpenAI’s agent bug hunt without the bill.

The 56 percent number AWS publishes

AWS publishes the gap. Each release ships a scores file counting every fidelity check by service and outcome. The 11 November file lists 42,937 checks: 24,243 pass, 18,614 fail, and 80 skipped, across 349 service models.

Of the 291 services with 20 or more checks, the average pass rate is 64.6 percent, and only 12 have no failures. S3 passes 1,024 of 1,373 checks, or 74.6 percent. DynamoDB passes 1,015 of 1,320, or 76.9 percent. EC2 passes 864 of 2,070, or 41.7 percent. CloudFormation passes 743 of 3,936, or 18.9 percent.

Service Checks passed Checks failed Pass rate
Amazon S3 1,024 349 74.6%
DynamoDB 1,015 305 76.9%
Amazon EC2 864 1,206 41.7%
CloudFormation 743 3,193 18.9%
All checks in release 24,243 18,614 56.5%

Read it as a coverage map, not a verdict. One failed workflow can create several failing checks, and the suite includes invalid calls and exact error messages. A high S3 score does not prove every S3 feature works. A low CloudFormation score means test your templates on real AWS.

How three agent teams ship code

Three agent teams do the work. The first writes fidelity tests as real scenarios, such as creating a queue, sending a message, receiving it, and deleting it. It verifies each against real AWS, then stores tens of thousands in a shared repository.

The second replays those tests and files an issue for each failure. The third gives one issue to one agent, which finds the cause and edits until the test passes. A merge gate then checks compilation, standards, security, and fidelity.

AWS says hundreds of agents fix thousands of issues a day. That cannot be checked outside, since the public repo shows output, not run logs. What is visible is unusual: the code, the failing issues, and the fixed issues, including a folder called things-that-do-not-work.

The “entirely by agents” claim has a boundary. The repo contains a vendor/ directory of framework crates replaced on update, and AWS says an earlier agent version wrote its own Python interpreter before being told to reuse libraries. The agents write the emulator, not every dependency.

The limits AWS admits

AWS is direct about limits. The emulator is for building and testing, not production. It does not evaluate IAM or resource policies the way AWS does, never denies a request signed with its own key, and does not check KMS or S3 bucket policies. A call real AWS rejects can succeed locally.

It keeps one account and one Region. You can send any Region or account ID, but resources are not separated that way, so testing multiple Regions means separate emulators. Fidelity is verified in US West (Oregon), except global services in US East (N. Virginia).

Telemetry is on by default. AWS says it never includes request or response contents, code, credentials, or personal data, and it can be turned off with a flag, environment variable, or config file. Sensitive teams should switch it off.

Confirmed vs Not Confirmed

Claim Status
Free local emulator, CC0 licensed, covering 438 API models Confirmed by AWS blog, docs, GitHub and the release
No AWS engineer wrote or reviewed the code Not confirmed independently; it is AWS’s own account
Fidelity is 24,243 of 42,937 checks, about 56.5 percent Confirmed by the release scores file
Safe for production and security testing Not confirmed; AWS explicitly says no
Hundreds of agents fix thousands of issues daily Not confirmed; internal figures are not public
Request and resource data stay local Confirmed, with anonymous telemetry on by default

What it means for Indian developers

The appeal is strongest for small teams, students, and anyone paying cloud bills in dollars. A local emulator removes the AWS account from the inner test loop, cutting learning and CI costs. Requests and resource data stay on the machine, apart from telemetry, which helps teams wary of sending test payloads abroad.

That is not compliance. The emulator is not a data residency control, and AWS says not to test security with it. India’s DPDP Act duties attach to real personal data in real systems, not a local mock. Cut iteration cost locally, then run security and integration tests on AWS in the region you serve.

Fidelity is verified in Oregon, so behavior that differs in Mumbai, including newer features and quotas, may not be represented. Check the scores for the services you depend on. Cloud savings have a way of arriving with a later bill, as CoreWeave’s interest costs show, and a free local test loop is one of the few that needs no negotiation.

FAQ

Is AWS Emulator free?

Yes. AWS charges no additional service fee, and the source is CC0. You pay only for real AWS resources when you deploy.

Can I use it without an AWS account?

Yes. It runs locally, creates credentials for each run, and accepts requests signed with those keys. You can start it and run the published tests without an account.

Can I use it in production?

No. AWS says it is for building and testing only. It does not reproduce IAM enforcement, cross-account access, or multi-Region behavior, so a local pass is not proof of production safety.

How close is it to real AWS?

It varies by service. In the 11 November release, S3 passed 74.6 percent of its checks and DynamoDB 76.9 percent, while EC2 passed 41.7 percent and CloudFormation 18.9 percent.

Does it send my code or data to AWS?

AWS says requests, responses, and resource data stay local. Anonymous usage and error telemetry is on by default. Disable it with --telemetry off, EMULATOR_TELEMETRY, or the config file.

The same agents that make the emulator useful create new risks. The Claude Code secret leak showed how fast an agent’s convenience setting can expose credentials, and a local emulator does not change that.

Sources

Share this article

Leave a Reply

Your email address will not be published. Required fields are marked *

Loading the next article…

Continue reading