The Problem
With more and more AI (slop) code being pushed into production, testing at scale becomes a huge problem. More specifically, how do we ensure that coding agents: (a) understand the behaviour of the code and test appropriately, and (b) don't cheat on their test suites. Specifically, we've seen the following patterns in scholarly articles and in our own development process.
Testing full-stack changes
I'm probably not the only one sick of manual QA. Browser-use agents are amazing at doing things for us in a browser, but they SUCK at navigating through flows like a real user. Mainly because 1. They can't do certain things (and we probably don't want them to), and 2. There's too much context bloat.
Agents are limited
They can't go into our Stripe dashboard and press buttons. Nor create a new Github App. They also can't go on the YouTube app and click a number for Google OAuth. And that's probably good. However, this significantly reduces the things it can do for us in QA. For example, Github OAuth was our only method of signing in, so we couldn't let an agent test Arga end-to-end.
There's too much context bloat
Imagine your first time walking into a casino. That's probably what an agent feels visiting amazon.com for the first time. If you just ask it to roam freely in your web app and "look for bugs", chances are they won't do anything useful AND spend a shit load of tokens. Maybe it did in fact walk into the casino.
Edge-case behaviour
LLMs tend to give users the happy path. And only the happy path. Similarly, in testing, LLMs tend to test for success, rather than all of the edge cases or boundary behaviour.
These tests don't actually test anything aside from proper code logic, but in production systems, this often requires domain knowledge, specs, correct output, and business logic. And if you were to explain all this to the LLMs, you might as well just write the tests yourself. So it really isn't your AI being lazy, but rather a mix of: I don't know what's going on in the codebase (business-logic wise), I don't know what a user would do, and I don't care to generate all the possible cases someone would use this code.
Hitting real APIs
With the increased number of external services being used, how can we test that our changes work with their API/SDKs? If you prompt the LLMs to write tests for your pre-commit hook, there's a good chance they will run the actual code. And the actual API too. This causes flakiness when you hit the rate limit or when API schemas change, and tests become non-deterministic.
Overly-extensive mocks
On the contrary, LLM-generated tests frequently mock too much, meaning external integrations are never truly tested. The classical example is: if you're testing A that depends on B, your LLM might mock the entirety of B and maybe even parts of A. The reason behind this is simple: the agent's success criteria is to make the tests pass, regardless of what it takes. Sadly, this means extensive mocking.
For example, I told GPT to write tests for checking if a repo had the Arga Github app installed, except I intentionally messed up the query syntax for the function. The tests mocked the results and the DB with the correct logic, but it wouldn't have caught the malformed query made to Github.
LLM tests just miss many bugs
This is probably the ONE thing that every single paper I've read about AI testing has said. LLM TESTS MISS MANY BUGS. This is likely a combination of all the reasons I've mentioned above, but specifically, I've seen that LLMs have failed at testing: timeouts, rate limits, auth failures, and malformed JSON from API calls. These should be caught in staging, if not dev, but many times the systems just fail silently because these errors are not even acknowledged. Things like proper error handling, retries, and fallbacks are crucial for dealing with external services, but coding agents often skip the error handling and imagine that everything will be successful.
So what now?
We identified these issues and built three main "solutions" to help speedrun QA.
Digital Twins
We create fully-functioning, behavioural clones of external services so your agents can roam free and test however they like without you having to worry about your Stripe API key getting leaked. Or anything like that. The difference between our twins and simply mocking a service is that our twins are standalone services that don't lean into how your program will behave. They are fully compatible with the official API/SDKs, and respond the same way as the actual service to surface edge cases traditional mocks or assertions miss. Plus, you (or your agent) will never hit rate limits.
Red-team Agents
To combat edge cases, we built out a workflow to red-team your entire web app using a fleet of agents. First, Arga understands your business through integrating with your existing tools, and generates an attack plan by identifying every single path a user might take. Then, we release agents into either your staging or deploy your code in our sandbox to start looking for bugs. We surface the behaviour of every agent with full screenshots and flag specific states in your UI that are buggy.
You can either run it manually on your entire web app or set it up on a repo so it runs after every PR/commit.
Context Aggregator
We know that people don't like writing test cases. So, instead of asking you to "describe what you would like to test", Arga pulls from tools like Slack, Linear, Sentry, Posthog, and your logging platforms to understand your product. This allows us to automatically run tests that are tailored to your codebase, so you never have to describe to agents what you want.