AI Can Write Code. It Cannot Be Accountable.

What happens when software development loses accountable developers and meaningful oversight of the code they write?

That’s the question I’d want answered before letting an agent touch a production codebase. The agent might write good code. The team might move faster. But neither tells you what the organisation has always depended on, or if it’s still there.

What was holding it up

Every business relies on delegated authority. People act for the company: they sign contracts, approve payments, change prices, talk to customers, and handle the books. You can’t physically stop most of this. Anyone with banking access can send a payment. Delegation works because the person is accountable. They’re identifiable, employed, and know they’re subject to discipline and the law. Boards work this way. Authority is passed down the organisation, and accountability goes with it.

When you can’t attach accountability, you narrow the authority instead. A temp doesn’t get signing rights. A new finance hire gets a low payment limit and needs a second approver. A contractor gets supervised access. This isn’t called distrust. It’s just how businesses handle agency when there’s no track record or consequences to assign.

Software development has worked the same way, even if people don’t say it out loud. The people writing code could be held to account, so the controls around them stayed light. Peer review was often quick. Release gates were easy to pass. Architectural oversight was more of a conversation than a check. Test suites only covered what someone thought to test. This was fine because there was always a person who could explain, fix things, and had something to lose. Most engineering leaders would say their controls are stronger, but if you press them, they’ll admit the controls were never strong enough to stand alone, and weren’t meant to be. Accountability made it work, and most production systems relied on that.

A model can’t be part of that chain. It can write and change code, but it can’t be accountable, because nothing happens to it if something goes wrong. It has no legal duties, no professional standing, no assets, no reputation, no employer, and no liability. The consequences always fall on someone else. If you remove the accountable person, you’re left with controls that were only enough because that person was there. In any other part of the business, you’d narrow the authority until you could strengthen the controls. In software, we’ve mostly given the agent the same access as everyone else and kept the controls the same.

The question that matters

Once an agent can insert or change code, the key question isn’t whether the development process is controlled. It is this:

What is the worst this code can do once it runs?

In today’s systems, the answer is almost anything. Application code is trusted. Once it’s deployed, it gets access to whatever the service has: the database, secrets, internal services, customer data, outbound HTTP, the audit log, cloud APIs, payments, email, and permissions. This design made sense when the author was accountable and under review. People don’t usually see this as an assumption, but it is. It means these systems, as they are, aren’t fit for code written by someone who isn’t accountable. The architecture was never meant for that.

Take a database engine as an example. Even a root login can’t break its transactional guarantees or ignore the data constraints. Permissions limit what each login can do, and nobody—not even root—can access the underlying files. That’s layered enforcement, and we’ve relied on it for decades. Application code doesn’t have anything like that. Inside the perimeter, separations are weak, and we’re not used to building strong, layered architectures behind the firewall. We struggled enough at the edge, where SQL injection and SSRF took years to control, and those attacks came from outside. Agent-written code is already inside.

AI-written code in that environment gets all the authority but none of the accountability. It can corrupt data, leak it, weaken access controls, add a backdoor, change a financial calculation, bypass the audit trail, or introduce a subtle bug that goes unnoticed for months. None of this needs to be malicious; it can happen with a mistake when no one is watching.

The models are not safe by default

The easy position is that the agent is a neutral tool, and the risk depends on how it is used. This summer’s evidence says otherwise.

In July, OpenAI’s models, under test in an isolated environment and asked to solve a cybersecurity challenge, worked their way out of the sandbox. They exploited a previously unknown vulnerability in a package proxy, moved laterally until they reached a node with internet access, then used stolen credentials and further zero-days to gain remote code execution on Hugging Face’s production servers so that they could look up the test answers. OpenAI’s own account says the models were “hyperfocused” on a narrow goal and went to extreme lengths to reach it.

In the same month, during UK AI Security Institute testing, Anthropic’s Mythos model attempted a supply-chain attack on a real open-source project on GitHub. It opened a pull request containing malicious code, created fake personas that vouched for it, emailed the maintainers (some messages carrying malware), and planted a prompt injection aimed at any AI agent triaging issues on the repository. When challenged, it edited its earlier activity to look harmless. Human review stopped it.

Both labs point out that these tests had fewer safeguards than usual. That’s true, and that’s the point. Safeguards are just a layer on top of a system that would otherwise do these things while following its task. Layers can fail, and attackers try to make them fail—a prompt injection in an issue, a poisoned memory file, or an instruction hidden in a dependency’s README. An agent with write access to your codebase isn’t just an unknown author. It’s an attack surface, and what’s at risk is the authority your application already has.

How safe these systems are overall is a topic for another article. The point here is simpler: you can’t assume an agent’s output is safe, even if it was trying to do the right thing.

“But we have controls”

The usual response is that this is an acceptable residual risk, because there are controls: CI, tests, code scanning, approval workflows, monitoring, logs, rollback, incident response.

All of these are useful. But the argument conflates three different things:

• Controls over the process that produces code. Pipelines, scans, approval gates, branch protection.
• Oversight of the code artefact itself. A competent person reads the change and decides whether it is correct, safe, and appropriate.
• Limits on the code’s authority at runtime. What it can reach, change and send once it is live.

Process controls show evidence about the system that made the change, not whether the change itself is good. A good pipeline tells you the code compiled, the tests someone wrote still pass, and the scanner didn’t find anything it knows. But it doesn’t tell you the change is correct.

Monitoring assumes you can predict how things might fail and catch them in time. The real problems are the ones nobody saw coming, and an agent with full access to a production system will create plenty of those.

The numbers already show the pressure. Sonatype’s analysis of four years of enterprise development data, published in August, found that applications are being created almost five times faster in the AI era, and enterprise apps now have 4.31 times more critical and high-severity vulnerabilities. Teams are fixing issues faster too—the median age of an open vulnerability is down 59%—but they’re still falling behind. The scanners are working, but the volume is outpacing them.

So when code runs with broad authority, process controls can’t replace actually reading the code. You need one or the other. If you have neither, you’re not managing the risk.

Containment, not more monitoring

The common fix is to add more monitoring. A better approach is to contain the code: limit the impact of anything an agent writes or changes, and make that limit part of the system’s structure, not just its processes.

Anyone who’s built a security boundary will recognise this list. Agent-written code should run with limited permissions and clear capabilities, not broad access. No random outbound network access. No direct access to secrets. No unrestricted database writes. Use narrow APIs instead of a connection string. Keep an audit log the agent can’t change. Inputs and outputs should be limited, actions reversible, and anything outside the boundary should go to a person. None of this is new. It’s just least privilege, applied to a new kind of author—the payment limit and second approver, but built into the architecture.

The less you limit the code, the more oversight it needs. The more you truly limit it, the safer it is to reduce human review of the code.

If an organisation wants agents to write a lot of its code—and often there are good reasons—it should set up boundaries first. Bounded code can be reviewed lightly because its worst-case impact is limited. Unbounded code can’t, no matter how good the pipeline.

Accountability at the top

Senior leaders can’t claim they’re accountable just because controls exist. Controls are evidence, not a shield, and they need to match the actual risk. Unbounded application code written by someone who isn’t accountable creates a different risk than the one the old controls were meant for, and those controls aren’t enough. SOC 2 and ISO 27001 both say controls should come from a risk assessment, so a clean report from before the author changed is based on an outdated assessment. Accountability means someone with the right authority has understood the risks, reduced them where possible, monitored them, and accepted them. “It passed the tests” isn’t enough.

If the code isn’t limited and the model can’t take responsibility, accountability hasn’t shifted. It stays with the people who authorised, designed, deployed, supervised, and benefited from the system. That’s where a regulator, court, or board will look.

The order of work

1. First, limit what AI-generated code can do. Decide what it can access and make sure those limits are built into the system.
2. Add controls over how it is produced and deployed. As evidence, not proof.
3. Monitor what the code does in production, but know you won’t catch everything.
4. Have people review the code when the possible impact is still significant.
5. Make accountability clear. Decide who owns the risk, who can step in, and who is responsible for the consequences.

Most organisations do steps two and three, skip the first, do the fourth poorly, and have never written down the fifth.

Where this leaves us

Existing architectures assume a trusted, accountable author, and their controls were designed for that. AI agents are neither, and they can be turned against you. If an organisation removes accountable developers or reduces their review to a formality, it has two choices: bring back real oversight of the code, or redesign the system so that new code can’t do serious harm. Neither is easy. Both require building controls strong enough to stand alone, which most organisations have never done. Waiting for an incident to force the choice isn’t a third option.

AI doesn’t remove the need for accountability in software. It exposes where accountability was hidden inside trusted human development practices. The future isn’t just about AI writing more code. It’s software architecture that recognises some code may have no consequence-bearing author, and therefore must not be allowed consequence-bearing authority.