AI Agent Monitoring

If your app calls an LLM or runs an AI agent, Rustrak can track it — runs, tool calls, token usage, model breakdowns, and a per-trace waterfall. There’s nothing extra to install: turn on your Sentry SDK’s AI instrumentation and it shows up automatically.

Where to find it

Open a project and click Agents in the sidebar. Until your app sends any AI activity, you’ll see an empty state with a setup pointer. Once data arrives, the page shows:

• Agent Runs — how many agent runs happened per time interval
• Duration — average and p95 run time
• LLM Calls by Model — which models get called the most
• Tokens Used by Model — which models use the most tokens
• Tool Calls by Tool — which tools your agents call the most
• Traces — one row per run: agent name(s), duration, total tokens, tool calls

Click a trace to open its waterfall: every step of that run (LLM calls, tool calls, handoffs) laid out in order, with duration, tokens, model, and tool name for each.

There’s deliberately no cost/spend widget — per-model pricing changes too often to estimate reliably, so Rustrak shows exact token counts instead of a guess.

Handoffs between agents

If one agent hands off to another mid-run, the Traces table lists every agent involved, not just the first — so handoffs are visible instead of hidden. A run with no identifiable agent name shows as (unnamed agent).

Sending data from your SDK

Enable your Sentry SDK’s AI integration — Rustrak reads the same data Sentry’s own AI Agent Monitoring does, so there’s no server-side setup.

Node.js — Vercel AI SDK

import * as Sentry from '@sentry/node';
 
Sentry.init({
  dsn: 'http://<sentry_key>@localhost:8080/<project_id>',
  integrations: [Sentry.vercelAIIntegration()],
});

import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';
 
await Sentry.startSpan({ name: 'research_agent', op: 'gen_ai.invoke_agent' }, async () => {
  const result = await generateText({
    model: openai('gpt-4o'),
    tools: { webSearch /* ... */ },
    prompt: 'Research the latest Rust release',
  });
});

vercelAIIntegration() automatically instruments every LLM call and tool call — you don’t need to tag anything by hand beyond wrapping the run in a span, as above.

Other SDKs

Any Sentry SDK with AI/LLM auto-instrumentation enabled works the same way. Check your SDK’s docs for its equivalent integration (e.g. Python’s OpenAI/LangChain instrumentation).

API access

Agents data is read-only over the API:

# Agent runs over time (default: all time, 1h buckets)
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/runs?period_hours=168&interval_hours=6"
 
# Avg/p95 duration over time
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/duration"
 
# Top models by call count / token usage (default limit: 3)
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/models/calls?limit=5"
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/models/tokens?limit=5"
 
# Top tools by call count
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/tools"
 
# Paginated traces table
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "http://localhost:8080/api/projects/1/agents/traces?page=1&per_page=20"

A trace in the response looks like:

{
  "trace_id": "6c1f...b2",
  "agent_names": ["research_agent"],
  "duration_ms": 842.3,
  "total_tokens": 642,
  "tool_call_count": 1,
  "started_at": "2026-07-18T10:15:00Z"
}

To see the individual steps of one run, list its spans with GET /api/projects/1/spans?trace_id=<trace_id> — this is what powers the waterfall view.

You can also pull this data from an AI assistant via the MCP server: get_agent_runs, get_agent_duration, get_agent_models_by_calls, get_agent_models_by_tokens, get_agent_tools, list_agent_traces.

Managing storage

Agent run data is made of spans, which count toward your instance storage. Unlike other data types, standalone AI spans aren’t removed by the retention cleanup on the Storage page yet — they’re counted in the totals shown there, but a cleanup run won’t reclaim their space.

Behavior reference