DigitalOcean Signals: Observability for AI Agents and Inference

You ship an agent. It runs in production, sessions start and finish, and the dashboards stay green. But then a user mentions that they spent ten minutes retrying the same failing command with slightly different arguments before giving up and rephrasing the request.

The problem? Nothing in your metrics caught this poor user experience. From the infrastructure’s point of view, nothing was wrong. The run started, the run finished, and the tokens were billed.

Metrics can report a finished run, but they don’t tell you whether it went well. Agents are non-deterministic, so the same prompt can go well on Monday and sideways on Tuesday. The evidence is all there in the conversation, but a single conversation can be thousands of events long, and your production AI offering can facilitate hundreds of them a day. Clearly, reading transcripts by hand does not scale. And asking another model to grade every conversation works, right up until the bill for grading rivals the bill for running the agent.

DigitalOcean Signals, now in Public Preview, watches production conversations on DigitalOcean Managed Agents (Harness Runtime) and DigitalOcean Inference Engine, with your consent, and turns them into structured behavioral signals. These include where a user got frustrated, where an agent looped, where a tool kept failing, where a session hit a rate limit, or where it ran out of context. Every signal points to the exact step in the conversation that produced it, so you go from “something is off with this agent” to “here is the turn where it went wrong” in a few clicks.

Not every signal is a problem. Satisfaction signals mark the sessions where the user confirmed the answer, said thanks, or reached their goal. Use DigitalOcean Signals to also determine how often your agent gets it right and what those conversations look like. Because flagged sessions can be exported as datasets, the same production cases become the material you use to evaluate and improve your agents.

For the full announcement, read the Signals launch post.

Signals, at a glance:

• Consent first. Off by default, enabled per environment, recorded and reversible, with deletion on request.
• Two sources today. Managed Agents on Harness Runtime, and Inference Engine conversations (API and doctl in this preview).
• Three layers of signals. Interaction, execution, and environment: 25 signal types, each pointing to the exact step that produced it.
• No instrumentation. Works on the conversation stream the platform already produces. Nothing to add to your agent.
• Model-free detection. Deterministic signals whose cost does not grow with your traffic.
• Evidence, not opinions. Every signal carries the run, the step, a confidence score, and a snippet.
• Datasets, not just dashboards. Exports feed Evaluations and Simulations directly, with personal information removed.

What Signals looks for

Signals detects 25 signal types in seven categories, grouped into three layers. The taxonomy comes from research the DigitalOcean team published earlier this year on arXiv about triaging agent trajectories.

These Signals layers and types include:

• Interaction signals describe how the user responds to the conversation. Misalignment covers clarifying questions, corrections, and rephrased requests. Stagnation covers conversations that drag without progress or responses that repeat information already given. Disengagement covers escalation requests, negative stance, and users quitting. Satisfaction covers confirmations, gratitude, and goals achieved.

• Execution signals show whether tool calls and actions succeed. Failures cover invalid arguments, bad queries, tools that don’t exist, actions invalid in the current state, and credential misuse. Loops cover plain retries, parameter drift (the same action retried with slightly different inputs), and oscillation between two actions.

• Environment signals show whether the infrastructure around the agent held up: external API errors, context overflow, malformed responses, network failures, rate limits, and timeouts.

Signals are computed from the interaction itself, with no model call in the loop. Detection is deterministic and inexpensive, so the cost of watching your agents does not climb with your traffic, and the same conversation produces the same signals every time.

Signals is a first-pass triage: its job is to decide which conversations deserve a reviewer’s time or a more expensive LLM judge, so you never have to pay for either across your entire traffic.

In a controlled study on tau-bench, a standard benchmark for tool-using agents, annotators found that 82% of the sessions Signals flagged were informative, meaning they showed a real failure or a good example worth learning from. For randomly chosen sessions, that figure was 54%. Put simply, reviewers reading flagged sessions find about 1.5 times as many useful ones in the same amount of time. In production terms, that is the difference between a review queue your team can clear and one it ignores.

It’s important to understand that signals are not grades - a signal marks a session as worth a closer look. Sometimes that is a failure you need to fix. Sometimes it is an exemplar you want more of. Signals surfaces both and leaves the judgment to you and the tools downstream.

How Signals works

Signals organizes every conversation into three levels:

• A dialogue is one completed run: the user’s message and the steps taken to respond to it, including every tool call and result and any human approvals, in order.
• A session is one conversation, made up of one or more dialogues.
• A signal is one behavioral pattern detected in that conversation.

For each session, Signals writes a signal report. The report rates the conversation’s overall quality from Excellent to Severe, lists the signals it detected along with their layer and category, and flags the session as concerning when it warrants a closer look. Each signal carries the run it fired in, the step index, a confidence score, and a short snippet, so you can land on the exact step in the reconstructed dialogue. Sessions are analyzed when they go idle for a period of inactivity, and again when the session is closed or destroyed. In other words, a long-running coding session that a developer returns to over several hours keeps receiving an updated report rather than a single final grade. Note that metrics in the Signals view can take up to an hour to reflect recent activity.

Signals is off by default; per environment. Turn it on and it only analyzes conversations that begin after you enable it. Conversation content stays within your DigitalOcean team boundary and is never used for cross-customer model training. Exports remove personal information from free text, such as user messages, step text, and snippets. Scenarios created from exporting traces utilize the non-redacted version in order to maintain scenario fidelity, but confidential content will be redacted when scenarios are displayed to users. Turning Signals off stops collection for new conversations, and a deletion request permanently removes what was collected. For more details, review the Signals data privacy documentation.

Getting started with Signals

Signals is in Public Preview. You’ll need doctl 1.181.0 or later and a token with the signals:query and signals:update scopes (a full-access token includes them). During the preview, accounts in a free trial can’t use Signals.

1. Enable Signals for an environment. In the Control Panel, open the Signals tab on the Harness Runtime page, select the environment, and switch the toggle on. This is the consent step: it is recorded, and it is reversible.

Here’s the same action from the command line:

doctl signals consent set --agent-id <agent-id> --enabled=true

The agent ID is the ID of the Environment Config the agent runs from (doctl harness-runtime config list shows them). Signals also works with Inference Engine conversations. In this preview, enabling and reading Inference signals is available through the API and doctl only, with the Console experience to follow.

2. Use your agents as usual. Signals consumes the same conversation stream that Harness Runtime and Inference Engine produce. There is nothing to instrument and nothing to add to your agent’s code. Signals for a session typically appear within minutes of the session going idle.

3. Read the signals. The Signals view lists recent sessions with their quality rating, signal count, and the concerning flag. Most rows will be unremarkable, because you triage a whole agent at a glance and open only what is flagged. From a session, open its dialogues and jump to the step where a signal fired.

From the command line, list sessions and narrow them to the pattern you care about:

doctl signals session list --agent-id <agent-id> --signal-type ExecutionLoopsRetry
doctl signals session dialogues --session-id <session-id> --output json

The JSON output has one object per dialogue, with the user message, the steps, the run status, and the signals detected in that run:

{
  "run_id": "5b4a3c2d-1e0f-4a9b-8c7d-6e5f4a3b2c1d",
  "sequence": 2,
  "user_message": "Deploy the latest build to staging.",
  "steps": [
    { "seq": 1, "type": "agent_reply", "text": "I couldn't find a deployment tool." }
  ],
  "run_status": "completed",
  "signals": [
    {
      "signal_type": "ExecutionFailureToolNotFound",
      "label": "Tool not found",
      "layer": "execution",
      "category": "failure",
      "step_index": 1,
      "confidence": 0.92
    }
  ]
}

Everything in this view is also available through the Signals API.

4. Export sessions as datasets. Select the sessions you want, or filter by signal type and time window, and create an export. Signals packages the sessions, their signal reports, and their full dialogues into one compressed JSON file, with personal information removed from free text, plus a presigned download link. When you export, Signals also creates Simulation scenarios from those conversations, so the export is ready for Evaluations without any manual test-data preparation.

Here is the same flow from doctl, filtered to sessions where users corrected the agent:

doctl signals export create --agent-id <agent-id> --signal-type MisalignmentCorrection
doctl signals export get <export-id>
doctl signals export download <export-id>

Exports stay available for seven days. That file is the handoff to the rest of your workflow.

From signals to better agents

Flagged production sessions make better evaluation cases than those that are synthetic, because they are the situations your agent actually struggled with. Export them as datasets into DigitalOcean Evaluations to score a new prompt, tool configuration, or model against the exact conversations that went wrong. Run them as Simulation scenarios to check that a fix holds across the whole conversation, not just one response. And because exemplars are flagged alongside failures, you can build datasets of what good looks like, not only what broke. The same exports can serve as fine-tuning data, or as reward and preference data for reinforcement learning, for teams training their own models.

We are building toward closing that loop further, so that findings from Signals can drive proposed changes to an agent and those changes can be validated against your own evaluations before they ship. Signals is the part of that loop that watches production.

What’s next for Signals

Harness Runtime and Inference Engine are the first two sources for Signals, but not the only ones planned. The pipeline behind the consent gate does not care where a conversation came from. Tool-call telemetry from Action Gateway is next, so that governed tool calls carry their failure cause and policy outcome into the signals. The Console experience for Inference signals follows the API. Scheduled exports are coming, so datasets can refresh automatically rather than on request. The signal taxonomy will keep growing as we learn from Public Preview usage.

DigitalOcean Signals is available today in Public Preview. Enable it for an agent you’re already running, and use the Signals documentation and the doctl signals reference to help with setup.