Controlling AI: Closing the gap between theory and reality

The gap between what AI can do and our ability to control it has become critical to the success of AI initiatives across the organization. I recently debated this topic with my colleagues Kev Breen and Matt Parven. We agree that AI adoption isn’t slowing down and organizations must figure out how to govern it.

Employees are experimenting with new tools, developers are embedding AI into applications, and teams are building agents to automate complex tasks. At the same time, organizations are under pressure to prove that this investment is delivering value while governance and security controls try to catch up.

Regulation will always be chasing the technology

AI models and capabilities are evolving in weeks and months, while regulations can take years to develop. Organizations can’t wait for regulations to tell them exactly what good AI governance looks like. At the same time, governing AI isn’t as simple as restricting one application. AI is increasingly embedded across enterprise technology. A better place to start is with something security teams already understand: data and access. Where is sensitive data stored? Who or what can access it? Where can it flow?

What has changed is how AI systems interact with data. Agentic systems, RAG, and technologies such as MCP can expose information and trigger actions in new ways. Understanding the architecture and the resulting threat model is critical.

Guardrails need to extend beyond the model

“Guardrails” has quickly become one of the most common terms in AI security. But organizations first need to ask: guardrails around what?

The safeguards built into a frontier model are only one layer. If an organization deploys an AI agent to interact with customers or access internal systems, it also needs controls around what that agent can do: what tools it can call, what information it can retrieve, what data it can return, and what actions it can take.

That requires a clearly defined purpose, testing, and monitoring. Traditional threat modeling, evaluations, and observability remain valuable. Organizations need to recognize when an agent begins behaving outside its intended use case.

Open-weight AI changes the equation

Open-weight models add another wrinkle. They can offer organizations more control over data and model behavior by being hosted internally, helping address concerns around sovereignty, regulations, and dependence on external providers.

But that freedom also creates a security challenge. Powerful models can be run locally with safeguards altered or removed. Defenders therefore can’t build security strategies around the assumption that attackers will operate under the same restrictions as legitimate users.

Don’t focus on the “AI attack.” Focus on the attack.

From a defender’s perspective, does it ultimately matter whether an attack was launched by AI, a script, a penetration testing tool or a human attacker? Not as much as we sometimes assume.

Don’t focus on "we’re being attacked by AI." Focus on "we’re being attacked." The fundamentals still apply: defense in depth, assuming compromise, and detecting and responding to attackers quickly.

AI may accelerate attacks and lower barriers for less-skilled attackers. That makes speed more important for defenders, but it doesn’t invalidate existing security playbooks.

Keep humans in the loop

The same principle applies when organizations use AI themselves. Despite the push toward autonomous agents, we aren’t at a point where organizations should simply hand over complex processes and walk away.

Human oversight remains essential, whether through direct involvement, supervision or strong review, and escalation mechanisms. AI systems remain unpredictable, and controls can fail. Humans can also answer an important question that is critical to security strategy: do we actually need to be an AI agent?

Every agent introduces potential overhead: permissions, data access, monitoring, testing, guardrails, cost, and risk. If a conventional application, automation or script can solve the problem effectively, adding an AI agent may create complexity without equivalent value.

The same discipline applies to model selection. The largest model isn’t automatically the right answer; choose the appropriate model and cost for the task.

We’ve all asked whether a meeting could have been an email. Now it may be time to ask whether an AI agent could have been a script.

Moving from AI adoption to AI readiness

Organizations don’t need to choose between innovation and control. But successful AI adoption requires moving beyond simply giving employees access to tools or adding AI wherever possible.

Start with the business problem and define what the AI system is supposed to accomplish. Make sure to understand the data it can access, threat-model the architecture, and test how it behaves when users don’t interact with it as intended. Then, put appropriate guardrails and monitoring around it and make sure the people using and building these systems understand both their capabilities and limitations.

AI is an extraordinarily powerful tool. But it is still a tool. The organizations that get the most from it won’t necessarily be those that deploy the most AI or move the fastest. They’ll be the ones that understand where AI creates real business value and can prove they are using it securely.

‍