Stateless vs Stateful Incident-Response AI Agents with OpenAI API

Why does a single alert feel like a dead end?

Imagine a monitoring system fires an alarm about a CPU spike, and the on‑call engineer spends ten minutes copying logs, opening a ticket, and hunting for the root cause. The same steps repeat day after day, and the manual effort adds up. What if an AI assistant could take the alert, pull the right metrics, suggest a probable cause, and even draft a remediation plan without waiting for a human to type?

Why the timing is right for AI‑driven incident response

Observability stacks have matured to expose rich, queryable APIs. Ticketing platforms now accept webhook payloads. At the same time, OpenAI released the Agents API, which lets you define a chain of tool calls that an LLM can orchestrate on the fly. Combining these pieces means you can hand over the triage loop to a software agent that talks to Prometheus, Grafana, Jira, and even your code repository. Senior developers who already own the CI/CD pipeline and security policies are in a sweet spot to embed such agents safely.

Technical deep dive: stateless vs stateful agents

Both approaches start with the same basic request payload – an alert identifier, a short description, and optionally a reference to a monitoring query. The difference lies in how the agent remembers context across multiple tool calls.

Stateless workflow treats each tool invocation as an independent HTTP request. The LLM receives the current tool output, decides the next tool, and discards everything after the response is sent back to the caller. This model is easy to scale because any compute node can pick up the next step, and you don’t need a database to keep track of conversation state.

Stateful workflow persists a conversation object – usually a JSON blob – that contains the full list of tool calls, their inputs, and outputs. When the LLM needs to decide the next action, it receives the entire history, allowing it to reason about earlier steps (e.g., “we already queried the error logs, now check the recent deployments”). Persistence can be in Redis, PostgreSQL, or a managed key‑value store.

Choosing between them depends on three factors: latency tolerance, complexity of the incident, and operational overhead.

import openai, os

openai.api_key = os.getenv("OPENAI_API_KEY")

agent = openai.Agent.create(
    name="incident_triage",
    model="gpt-4o-mini",
    tools=[
        {"type": "function", "function": {"name": "query_prometheus", "description": "Run a PromQL query", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}}}},
        {"type": "function", "function": {"name": "create_jira_ticket", "description": "Open a ticket in Jira", "parameters": {"type": "object", "properties": {"summary": {"type": "string"}, "description": {"type": "string"}}}}}
    ]
)

response = openai.Agent.run(
    agent_id=agent.id,
    input={"alert_id": "cpu_spike_123", "message": "CPU usage > 90% on node-7"},
    # For a stateful run you would also pass a `session_id`
)
print(response)

In the snippet above, the run call is stateless by default. Adding a session_id argument tells the API to load the persisted conversation and continue from where it left off.

Common pitfalls and what tends to go wrong

• Missing authentication headers: When the agent calls your internal Grafana endpoint, the request often fails because the service expects a Bearer token or a client certificate. Make sure the tool definition includes a secure way to inject the token at runtime, not hard‑coded in the code.
• State bloat: A stateful session can grow quickly if you keep raw log dumps in the history. Trim large payloads or store them externally and reference them with IDs.
• Privacy leakage: Sending full stack traces or configuration files to a hosted LLM may violate compliance rules. Filter sensitive fields before handing data to the LLM, or run the agent on‑prem where the data never leaves your network.
• Observability gaps: If the agent itself fails to call a tool, you might not see the error because the failure is swallowed by the LLM response. Wrap each tool call in a try/catch block and emit a metric (e.g., agent_tool_failure_total) to Prometheus.
• Race conditions: When multiple alerts trigger the same stateful session, concurrent updates can overwrite each other. Use optimistic locking or a per‑alert session ID to avoid clashes.

Practical implementation guide: step‑by‑step

1. Define the tool set. List every external system you need – Prometheus query, Grafana snapshot, Jira ticket, GitHub issue, Kubernetes exec. Write a small wrapper function for each that validates inputs and returns a concise JSON payload.
2. Choose persistence. For a pilot, Redis with a 24‑hour TTL is fast and cheap. In production, consider PostgreSQL with a JSONB column to allow ad‑hoc queries on the conversation history.
3. Set up authentication. Store API keys in a secret manager (e.g., HashiCorp Vault). In each wrapper, fetch the secret at call time and add it to the request headers. Never log the raw secret.
4. Implement the agent runner. Create an HTTP endpoint (e.g., /run‑agent) that receives the alert payload, generates a session_id if you want stateful behavior, and forwards the request to openai.Agent.run. Log the request ID and the LLM's chosen tool for traceability.
5. Integrate with your observability stack. Export a metric like agent_run_duration_seconds and a log entry that includes the alert ID, selected tool, and outcome. This makes it easy to set alerts if the AI starts failing silently.
6. Test end‑to‑end. Simulate a real alert, verify that the agent can query metrics, create a ticket, and optionally comment on a pull request. Use a staging environment where the LLM is forced to a deterministic temperature (e.g., temperature=0) to make tests reproducible.
7. Decide on orchestration. If you want full control, spin up your own Docker container that runs the runner and talks to the OpenAI API – this is the on‑prem option. For a quicker start, use a managed function platform (AWS Lambda, GCP Cloud Functions) and let the provider handle scaling – the hosted option.

Closing thoughts: balancing flexibility and control

Stateless agents are cheap to run at scale and fit well with bursty alert traffic. Stateful agents, on the other hand, give the LLM a memory that can reduce the number of tool calls and produce richer remediation plans. The trade‑off is a bit more operational work and the need to protect persisted data.

When it comes to orchestration, on‑prem gives you full visibility into network traffic, audit logs, and the ability to enforce strict data residency. Hosted orchestration reduces the operational burden but means you trust the provider with the payloads. A hybrid approach – run the runner on‑prem while pointing to the hosted OpenAI endpoint – often hits the sweet spot.

Finally, remember that AI agents are assistants, not replacements. Keep a human in the loop for high‑impact actions, and make sure the hand‑off is clear in your incident‑response playbook.

Ready to try it out?

Grab the starter repo at akkistech.com, spin up the Docker compose stack, and fire a test alert. You’ll see how a few lines of code can turn a noisy alarm into a guided remediation flow.