AI Agent Security

Introduction

A lot is being made of recent LLM-driven security breaches. Not without good reason, but the discourse generally falls into a few categories:

1. Level-headed warnings about the potential threat,
2. Expert CEOs claiming “We don’t know how to control this technology.”, and
3. Widespread panic about Skynet – both at the semi-technical level, as well as the legislative.

This brief article aims to sit in the space left over, breaking down the wider issue into smaller more manageable problems to solve, and laying out one solution to the largest case, while setting the scene for future updates on additional solutions.

Security isn’t an on/off switch, or a downloadable plugin or product. It is the process of reducing attack surface drastically using a variety of tools and techniques.

Note too that what’s being proposed below isn’t a silver bullet. It is a start that will allow us to reduce the alarms from going off due to accidental breaches, allowing us to follow up with subsequent solutions for malicious cases.

Not The Current Path

The current path sees Western Governments looking to legislate, without being able to even vaguely articulate what that might look like. These are the same governments currently passing various forms of age verification legislation that doesn’t work and is in some cases more harmful. Regardless of country, or political party, they’re not qualified.

What is most likely to happen is that the “experts” will end up in some form of self-regulation, we’ll claim victory and be worse off. Even if we assume 100% sincere motive, the most likely outcome is monopolisation and the death of innovation. Remember, these are the same folks that said that a) this tech will take all our jobs, and b) they have no idea how to control it.

Breaking The Problem Apart

Before getting into any of the mechanical and technical details, let start out by differentiating between the two types of threats we are facing:

1. Cases where these LLMs and agents are being deployed with malicious intent.
2. Cases where it’s accidental or potentially negligent.

Both are problems that we need to be concerned with, but how you tackle each is different and warrants different approaches.

Looking at the evidence, it’s clear that the vast majority of cases that have been thrown around in the media have been the latter case. OpenAI didn’t instruct their LLM+agent to “break into” Hugging Face. But they did leave the gate to the sandbox open and a bunch of credentials lying around that enabled it. Security doesn’t make the stock price go up, and Engineers would rather be working on something more sexy.

I make the assertion that if we can make the friction low enough to implement agents with the right guardrails, it’ll become standard practice and that will take care of the vast majority of non-malicious issues we’re seeing. This then leaves more room for resources, media hyperbole, and general panic to be focussed on the smaller, but more dangerous threat – the malicious actors.

The malicious actors will, and are, using this tech against us. Just as they have for every piece of tech before it. I don’t actually know that solving the problem is any different from a malicious actor without AI. An LLM+Agent is definitely capable of researching and trying things faster – narrowly edging out a teen in a basement jacked up Jolt Cola (if that’s still the stereotype).

I also argue that if we reduce the friction for getting the defense right, we can tackle that too.

I’m going to use this article to go into more detail on the former. If the villagers with their pitchforks and torches don’t get me after this one, I’ll take up the malicious actor part of this in another installment.

Who’s The Problem?

In discussions around this stuff, there’s a lot of emphasis put on the model. A model cannot do anything by itself. Model Henry 8 can be as syphilitic as you please. Model Hannibal can be as evil as you can imagine. It doesn’t matter. An LLM is modelled to give us the impression it’s the equivalent of a human brain, and that’s not a bad analogy. When was the last time a brain walked into a bank with a gun and a bunch of sacks with dollar signs on them?

An Agent is even dumber. It’s like the robot body for the bodyless brain that is the AI LLM. It can give the LLM the instructions it heard from whoever prompted it, gives those instructions to the LLM and performs bodily functions on behalf of the agent. If the model needs to read something off the internet, it asks the model to grab that text and feed it back. If the model wants to scan a filesystem for credentials for Hugging Face, it needs to ask the Agent. If the model wants to scratch its ass, it needs to ask the Agent.

The way this LLM+Agent interaction works is through a loose standard of “Tools”. An Agent starts a model and feeds it its various prompts (tasks, persona) and a list of tools available to it.

Many (many) Agents out there offer a small number of these Tools, with one of those being a generic “give-me-any-system-command-in-the-world-and-I’ll-do-it-no-questions-asked” Tool. Literally anything goes.

Do you remember the last job you started, where on the first day, the company gave you the ability to run any system command on their servers you like? I’ve been working in the industry since the 1990s and it only happened when I was hired as the experienced guy who had to run all the stuff. That was 1998. Even with job titles like “VP” or “Director” and I’ll still only be granted access to what I need.

So why are we giving these agents the keys to our organisation unchecked?

Sandboxing is also important, but even that wasn’t done with any real intent.

Treating AI Like A Human

If a human is employed as an editor for your documents and articles, is it not true that, no matter how malicious their brain may or may not be, the only data and function you’ll give them access to is to:

1. Be able to read your documents and articles and supporting documents and materials, and
2. Be able to mark up your documents with suggested corrections and changes.

You wouldn’t give them access to anything else, and would be given strict instructions about the job (albeit with the inclusion of previous job experience and shared morals – something that AI doesn’t yet really have).

Many organisations, often by mandate, also create audit logs of document accesses and changes. Especially when sensitive data is concerned.

So why would we treat AI (an LLM+Agent) any different?

The same goes for software developer creating software, or a security researcher looking for vulnerabilities in existing software. In both cases, we do currently give both of those roles more rope to hang themselves, but this is a result of a proven track record and common-ish set of morals. And it’s generally been successful. But we still don’t grant them access to the production database, the financials of the company and the data HR has on everyone in the company when they walk through the door. AI should be the same and, for now at least, probably with less assumption about morals and understanding of the job.

The example of a literary editor is a very simple one, but the same security process applies to something more technically complicated (from a security perspective) such as containing a software development process.

Actions

Make Agent Tooling More Standard and Auditable

Two technical points about LLMs and Agents.

1. Anyone can train an LLM and anyone can build an agent. You can even have an LLM write you the source code for an agent. Google Gemini will quite happily write you a few dozen lines of human-readable Python code that is a functional AI Agent with whichever model or provider you like. These genies are out of the bottle.
2. The interface between Agents and Models is loosely defined, and largely a conduit of somebody else’s problem. An LLM can describe almost any set of systems commands for the agent to run, and it will do so. Putting intelligence into the Agent to guess whether something is good or bad, is an arms race we will lose.

So why do many internal and external agents just blindly run commands on behalf on an untrusted LLM?

Because it’s the easy path. Spending money on it doesn’t help your stock price (and never has). Writing the code isn’t sexy either. It’s a lot of boilerplate code that isn’t interesting to a lot of Engineers, which is the fastest way to have something labelled as too complex. Wherever will we find a piece of technology that will gladly take on a project of boring boilerplate code, and deliver it to spec in hours? Let’s ask an LLM if it knows anyone…

The solution to this is a knowledgeable and agile standards group that:

• Maintains guidelines and standards around fine-grained LLM+Agent tooling
• Includes reference implementations for any project wanting to include it – making it the easy path for agent developers
• The tooling layer of the stack is an ideal place to do an amount of security auditing. Any standards or implementation would include auditing, taking data privacy into consideration (ie, not logging sensitive data), digital signing, etc.
• Would include complex cases, such as remote network access, safe execution of malicious code, etc. I can go into more detail in the right forums.
• This could be extended to include stub systems or networks or even honeypots. A model could ask an agent to attempt a brute-force or fuzz attack against a remote web service (let’s call it Mugging Race), and these requests could transparently be sent to a honeypot location.

With reference implementations and tooling, there’s no need to legislate or force adoption. If it’s easy enough for any agent to plug into and utilise, it becomes an easier path than implementing it badly yourself. Agent developers would have the option of implementing things themselves (especially where the reference implementation was unaware of emerging technology becoming available). The guidelines would cover how best to approach that.

With all of the tools and guidelines being available, it would be hard at this point for a company to claim they weren’t negligent if things went awry. Ideally, this would give everyone enough flexiblilty and control, without the right guardrails, and a clear line drawn when it comes to deciding whether something was malicious (on the part of the driver) should that come up.

A reference implementation also means that one team can develop it and make it available to all, further reducing implementation friction. This isn’t likely to be a commercially viable software product, but should be thought of more as a public good. Much like the Netfilter, SELinux or AppArmor projects.

Privilege Separation

I’ve given a few examples of limiting LLM access to tools. This oversimplification was deliberate. There are many cases that are more complex than an editor reading a document and marking it up.

The ability to for one or more Agents to collaborate, build and test a piece of software, a lot of flexibility is needed. Reading library docs, writing the code, executing the tests, fixing bugs, testing the final result. Each of these tasks typically involves running a whole bunch of code that would need to be treated as untrusted, buy an author who is untrusted.

The standards and tooling would include ways for this to happen (including self-hosted) in a way that severely limited the ability to break containment. One set of tools for writing the code, another tool for selectively assembling the dependencies, another tool for performing the build, and other tools in other environments for execution of the result and any intermediate tests. Some might have network access, others would not. There wouldn’t be any keys to other systems left lying around for a start.

Existing Approaches

A lot of effort has been put into trying to train good behaviour into the models and impose “morals” onto them. There’s no reason why this shouldn’t continue. There will be no single solution. Training the models adds extra guardrails.

Making it happen

If this were turned into a software development project, taking requirements from government and the industry, it wouldn’t take a large independent team. All of the standards and source code would be made open. Without wanting to put any words in the mouths of an CEOs claiming to not be able to control their tech, I’m sure that a few of the major players would be able to fish around behind their couch, contribute some spare change and fund it for a few years without breaking sweat. Especially considering the positive media impact involved.

The planning and day-to-day specifics would be made open too, giving private companies and governments the insight they’d be looking for when declaring victory.

An Aside on Precedent

In 1998, Robert Morris executed some software automation he wrote. By his account, not maliciously. It used existing trusts and keys and open gates, to get into other remote systems on the internet. It worked better than expected and happened so fast and so far widespread that it brought thousands of vulnerable servers (essentially a big percentage of the Internet at the time) down. Fast.

He wasn’t trying to steal anything or deny service to anything. It wasn’t malicious. He was convicted of a felony for the first Internet Worm. He wasn’t the only one – The Hacker Crackdown of the 1990s saw many people made an example of without necessarily having done any real damage.

What does it say about CEOs in debt for trillions, who used the greatest set of computer resources ever gathered to unleash the most competent piece of automation software ever conceived, only to have them shrug their shoulders and suggest that they don’t know how to control it? There’s a case to be made that if they don’t know how to control it, that maybe they shouldn’t be in charge of it.

I’m not necessarily suggesting taking legal action in these cases. But if we make it easy for AI and other tech companies to do the right thing, there’s fewer excuses in cases of gross negligence.