How I Would Make AI Agents Safe in Production

A practical workflow for giving AI agents autonomy without losing control.

AI agents are getting closer to production systems every month. They can already investigate incidents, generate infrastructure changes, modify configuration, open pull requests, and trigger operational workflows on their own.

The interesting question at this point isn't really whether an agent can make a change. It's what the agent should be allowed to change, under which conditions, and how we recover when it inevitably gets something wrong.

If I were introducing AI agents into a production environment, I wouldn't start by handing them broader access. I'd start by designing the control layer around them first.

Step 1: Define exactly what the agent can do

Before picking tools or models, the permitted actions need to be nailed down clearly.

Something like:

            Agent can:
→ Read logs
→ Query metrics
→ Analyse traces
→ Open Jira incidents
→ Create pull requests
→ Suggest Terraform changes

Agent cannot:
→ Apply infrastructure directly
→ Delete production resources
→ Change IAM policies
→ Access secrets
→ Modify production databases

          

This can't just live in a doc somewhere. It has to be enforced technically, through tools like:

• AWS IAM
• Kubernetes RBAC
• GitHub permissions
• HashiCorp Vault
• OPA / Gatekeeper

The underlying principle is pretty simple: give the agent the minimum authority it actually needs to get the job done, nothing more.

Step 2: Put Git between the agent and production

For most infrastructure changes, I'd avoid giving the agent direct access to production at all. Instead, the flow would look more like this:

            AI Agent
   ↓
Proposed change
   ↓
Git branch
   ↓
Pull Request
   ↓
Automated validation
   ↓
Human / policy approval
   ↓
CI/CD
   ↓
Production

          

That setup buys us a lot: version history, peer review, auditability, automated testing, rollback, and clear ownership over every change.

Tools that fit well here include GitHub or GitLab for change management, Terraform for infrastructure, GitHub Actions, GitLab CI or Azure DevOps for validation and deployment, and Argo CD for GitOps delivery into Kubernetes.

The agent proposes the change. The platform decides whether it's actually allowed to move forward.

Step 3: Add policy-as-code before deployment

Every change should go through automated policy checks before it ever gets close to production.

            Terraform Plan
      ↓
Security Scan
      ↓
Policy Validation
      ↓
Cost Check
      ↓
Approval
      ↓
Apply

          

Tools worth considering here include Open Policy Agent, Gatekeeper, Checkov, tfsec, HashiCorp Sentinel, and AWS Config. These policies could block things like:

• public S3 buckets
• unencrypted databases
• open security groups
• privileged Kubernetes containers
• unapproved regions
• resources that cross a defined cost threshold

The agent doesn't need to memorise every single rule here. The platform just enforces them regardless.

Step 4: Make every action observable

If an agent is doing operational work, we need a clear trail of exactly what happened.

            Agent request
↓
Context received
↓
Decision made
↓
Tool invoked
↓
Resource changed
↓
Result
↓
Rollback status

          

A typical observability stack might include OpenTelemetry, Prometheus, Grafana, CloudWatch, Elastic, or Datadog. I'd also give agent actions their own distinct identity rather than lumping them under something generic like admin-user. Something like ai-incident-agent-prod works much better and makes auditing far easier.

The question you always want to be able to answer is simple: what did the agent do, why did it do that, and what happened afterwards?

Step 5: Design rollback before autonomy

Before letting an agent make autonomous changes, recovery needs to actually be proven to work, not just assumed.

For application deployment:

            New version
↓
Health checks
↓
Error rate monitored
↓
SLO violation?
   ↓ YES
Automatic rollback

          

For infrastructure:

            Terraform change
↓
Validation
↓
Apply
↓
Monitoring
↓
Unexpected behaviour
↓
Revert commit / previous state

          

Useful tools here include Argo Rollouts, Kubernetes health probes, Terraform state, feature flags, AWS deployment services, and Grafana or Prometheus alerts. If rollback isn't reliable, autonomous execution really should stay limited until it is.

Step 6: Increase autonomy gradually

Don't jump straight from "AI assistant" to "AI operator" in one leap. Autonomy should come in stages instead.

Stage	Agent capability	Approval
1. Observe	Logs, metrics, traces	None
2. Recommend	Suggest remediation	Human
3. Prepare	Create PR / change plan	Human
4. Execute low-risk	Restart, scale, known runbooks	Policy controlled
5. Autonomous remediation	Production changes	Only within strict guardrails

An agent might eventually be trusted to restart a failed Kubernetes pod on its own, for instance. But touching networking, deleting infrastructure, or modifying IAM should stay far more restricted for a lot longer.

Autonomy should grow based on proven reliability, the actual risk of the action, blast radius, rollback confidence, and observability, not simply because the underlying model got more impressive.

What the final workflow looks like

Put together, a controlled agent workflow could look something like this:

            Incident detected
      ↓
AI agent investigates
      ↓
Logs + metrics + traces
      ↓
Root cause hypothesis
      ↓
Change proposed
      ↓
GitHub Pull Request
      ↓
Terraform / Kubernetes validation
      ↓
OPA security policies
      ↓
CI tests
      ↓
Approval based on risk
      ↓
Deployment
      ↓
Prometheus / Grafana monitoring
      ↓
Healthy?
   ↙       ↘
 YES       NO
 ↓          ↓
Keep     Rollback

          
          

The AI agent is really just one piece of this. What actually makes the whole thing safe is the platform wrapped around it.

The real objective

The goal here shouldn't be "give AI agents production access." It should be closer to "let AI agents act through the exact same engineering controls we already expect from every other production change."

That's a much safer architecture, and honestly it scales better too. Once policy, identity, observability, and rollback are baked into the platform itself, they protect every change the same way, whether it comes from a developer, a CI/CD pipeline, an automation script, or an AI agent.

So the interesting question for engineering leaders probably isn't how much autonomy an agent can technically handle. It's how much autonomy your platform can actually support safely.