Software AI Can Use

The term “AI-native” is one of those phrases that appears to describe an architectural approach but actually says very little about the quality of that architecture.

It actually means two different things. Using AI throughout software development, from requirements to testing, has a lot of potential, and I’m excited about that. But if AI-native software means putting an AI model at the centre of a business system, we need to think much more carefully.

My advice is simple: don’t write AI-native software. Instead, write good software that AI can use.

Software and AI have different ways of providing assurance

Traditional software isn’t perfectly predictable. It can have bugs, concurrency issues, and unexpected interactions. But we can set rules, build tests, limit its interfaces, and make useful guarantees about how it behaves.

An LLM-based agent is a different challenge. Most of its logic is hidden in weights we can’t really inspect. Its behaviour depends on the context it keeps, the information it finds, and the instructions it gets. Sometimes those instructions could even be hidden in data that was supposed to be safe.

This isn’t just another function that returns a score. An agent can understand a request, make a plan, pick tools, and choose arguments. Its possible actions are much broader, and even if it passes some tests, we can’t be as sure about what it will do next.

This makes AI great for interpreting, exploring, and suggesting actions. But it shouldn’t be responsible for keeping a business system’s integrity.

An accounting system should still work like an accounting system

Imagine an AI helping someone look into issues in an accounting system. It could check transactions, explain oddities, read invoices, and suggest corrections. These are great uses for AI.

But should it write directly to the general ledger? Absolutely not.

The accounting system still has to enforce double-entry rules, valid account codes, open periods, authorisation limits, separation of duties, and full audit trails. If the AI suggests a change, it should go through the same process, controls, and audits as a suggestion from a person or another system. Depending on the risk and authority, this might need independent checks, human approval, or a tightly controlled automatic process.

AI can help figure out what needs to be done. The software is still responsible for what is allowed and what actually happens.

If an AI can bypass those controls, the problem isn’t the prompt. It means the security and business architecture is flawed.

Treat an AI agent like a user, but trust it even less

We already know how to build systems for people who can make mistakes, misunderstand instructions, or act maliciously. We don’t give users full database access and just hope they follow the rules. Instead, we give them an identity, permissions, controlled actions, and accountability.

AI agents should get the same treatment, but with even more caution:

• Give each agent its own identity, instead of letting it pretend to be a person or share someone else’s credentials.
• Limit its permissions based on its role and the authority given by the person or process it represents.
• Only allow access to specific business operations, not full access to internal data or how things work inside.
• Make sure validation and authorisation happen outside the model, for every operation.
• Keep records of requests, decisions, approvals, and changes so actions can be traced and checked.
• Require independent approval for sensitive or permanent actions. Never let the agent approve its own escalation.

Read-only access isn’t always safe. An AI agent with wide access to financial or personal data could share it with someone who shouldn’t see it. This could happen through a normal response, an external tool, or a hidden prompt in what looks like safe material.

There’s also a risk in letting sensitive information into the AI’s context at all. Data pulled for one task might stay in chat history, memory, logs, or caches, and could end up being seen by the wrong people or used for the wrong purpose.

An AI agent should only get the information it needs for its task, and never more than the person asking is allowed to access. Enforce these limits when the data is retrieved, not by telling the model what it may share. The same rules should apply when information is reused across sessions or users.

The AI’s ability to read or change information should be limited by security boundaries, just like a user’s.

Let AI access business capabilities, but never make business rules optional

APIs and protocols like the Model Context Protocol (MCP) let AI agents discover and use business capabilities from other systems. But exposing these capabilities doesn’t mean business rules can be ignored. The software behind the interface must still handle authentication, authorisation, validation, and rule enforcement.

The real design work happens behind the interface. For example, an accounting system can let an AI retrieve authorised transactions, suggest journals, or submit them for approval. These are clear actions with set permissions. It should never allow a general-purpose ledger update.

MCP connects AI to software features, but it doesn’t guarantee those features are safe. The software underneath must enforce the rules every time.

Keeping interface and enforcement separate pays off. Teams can try new AI models or agents without risking the core system’s integrity. Business operations stay stable, and the system remains trustworthy, no matter how the AI changes.

What about products that are built specifically for AI?

Some products are made to deliver AI features, like generating images, transcribing speech, or reading complex documents. Their main value really does depend on AI, so it makes sense to call them AI-native.

But that doesn’t mean we should let model-generated reasoning handle authorisation, transactional integrity, or other guarantees. Even an AI-focused product needs reliable software around the model.

None of this is against AI-native development. Definitely use AI to speed up specification, coding, testing, and maintenance. How we build software can change a lot, as long as we keep the principles that make systems reliable.

A simple test for your architecture

When I review an AI integration, I ask one question:

If the AI acts incorrectly, unpredictably, or even maliciously, can the software around it still keep its essential security and business rules?

If the answer is no, that’s an architectural problem, not a sign of being especially AI-native. We don’t need to reinvent software engineering basics to use AI. We need clear business features, strong boundaries, least-privilege access, independent checks, and reliable audit trails.

Don’t write AI-native software. Write software that AI can use, and that stays trustworthy even when AI makes mistakes.