Monitoring Agentic AI: Cost, Latency, and Quality Drift
# Monitoring AI agents requires different metrics than standard APIs. Track cost per task, tool failure rates, quality scores, and context utilization — not just request rate and latency.
Standard APM dashboards show request rate, error rate, and latency. For AI agents, these metrics are incomplete. An agent can have a 0% HTTP error rate while failing 40% of tasks, burning 10× expected budget, and degrading in quality week over week. You need agent-native metrics.
The Four Metric Categories
1. Cost metrics → Are we within budget?
2. Quality metrics → Are tasks completing successfully?
3. Reliability metrics → Are tools and models stable?
4. Efficiency metrics → Are we using resources wisely?
Cost Metrics
Track cost at multiple granularities:
interface CostMetrics {
// Per request
tokensCostUSD: number;
totalCostUSD: number;
// Per task type (aggregate)
costPerTaskType: Record<string, number>;
costTrend: "stable" | "increasing" | "decreasing";
// Budget tracking
dailySpendUSD: number;
projectedMonthlyUSD: number;
budgetUtilizationPct: number;
budgetForecastedExceedanceDate?: Date;
}
// Track in your DB
async function recordCostMetrics(traceId: string, metrics: {
model: string;
inputTokens: number;
outputTokens: number;
cacheReadTokens: number;
taskType: string;
}) {
const cost = calculateCost(metrics);
await db.execute(`
INSERT INTO cost_events (trace_id, model, input_tokens, output_tokens,
cache_read_tokens, cost_usd, task_type, recorded_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW())
`, [traceId, metrics.model, metrics.inputTokens, metrics.outputTokens,
metrics.cacheReadTokens, cost, metrics.taskType]);
}
// Alert when daily spend exceeds threshold
async function checkDailyBudgetAlert() {
const { daily_cost } = await db.queryOne<{ daily_cost: number }>(`
SELECT SUM(cost_usd) as daily_cost FROM cost_events
WHERE recorded_at > NOW() - INTERVAL '24 hours'
`);
const DAILY_BUDGET = parseFloat(process.env.DAILY_AI_BUDGET_USD ?? "100");
if (daily_cost > DAILY_BUDGET * 0.8) {
await sendAlert(`AI budget at ${(daily_cost / DAILY_BUDGET * 100).toFixed(0)}% — $${daily_cost.toFixed(2)} of $${DAILY_BUDGET}`);
}
}
Quality Metrics
Define what "task completed successfully" means and measure it:
interface QualityMetrics {
taskSuccessRate: number; // % tasks completed successfully
taskPartialRate: number; // % tasks partially completed
avgQualityScore: number; // LLM judge score 0–1
qualityTrend: number[]; // rolling 7-day scores
regressionDetected: boolean; // significant drop in quality
}
// Async quality evaluation (don't block user response)
async function evaluateTaskQuality(
taskType: string,
input: string,
output: string,
traceId: string
): Promise<void> {
const { score } = await llmJudge(input, output, getQualityCriteria(taskType));
await db.execute(`
INSERT INTO quality_scores (trace_id, task_type, score, evaluated_at)
VALUES ($1, $2, $3, NOW())
`, [traceId, taskType, score]);
// Check for quality regression
await checkQualityRegression(taskType);
}
async function checkQualityRegression(taskType: string) {
const { current_avg, previous_avg } = await db.queryOne<{
current_avg: number;
previous_avg: number;
}>(`
SELECT
AVG(score) FILTER (WHERE evaluated_at > NOW() - INTERVAL '24 hours') as current_avg,
AVG(score) FILTER (WHERE evaluated_at BETWEEN NOW() - INTERVAL '7 days' AND NOW() - INTERVAL '24 hours') as previous_avg
FROM quality_scores
WHERE task_type = $1
`, [taskType]);
const drop = previous_avg - current_avg;
if (drop > 0.1) { // >10% quality drop
await sendAlert(`Quality regression: ${taskType} dropped ${(drop * 100).toFixed(1)}% in 24h`);
}
}
Tool Reliability Metrics
For each tool in your agent's toolkit, track independently:
interface ToolMetrics {
toolName: string;
callCount: number;
errorRate: number;
timeoutRate: number;
avgLatencyMs: number;
p99LatencyMs: number;
avgOutputSizeTokens: number;
}
// Record every tool call
async function recordToolCall(event: {
toolName: string;
success: boolean;
timedOut: boolean;
latencyMs: number;
outputTokens: number;
traceId: string;
}) {
await db.execute(`
INSERT INTO tool_calls (tool_name, success, timed_out, latency_ms, output_tokens, trace_id, called_at)
VALUES ($1, $2, $3, $4, $5, $6, NOW())
`, [event.toolName, event.success, event.timedOut, event.latencyMs, event.outputTokens, event.traceId]);
}
// Tool health dashboard query
async function getToolHealthReport(lookbackHours = 24): Promise<ToolMetrics[]> {
return db.query<ToolMetrics>(`
SELECT
tool_name,
COUNT(*) as call_count,
1.0 - AVG(CASE WHEN success THEN 1 ELSE 0 END) as error_rate,
AVG(CASE WHEN timed_out THEN 1 ELSE 0 END) as timeout_rate,
AVG(latency_ms) as avg_latency_ms,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY latency_ms) as p99_latency_ms,
AVG(output_tokens) as avg_output_size_tokens
FROM tool_calls
WHERE called_at > NOW() - $1 * INTERVAL '1 hour'
GROUP BY tool_name
ORDER BY error_rate DESC
`, [lookbackHours]);
}
Efficiency Metrics
Are agents doing more work than necessary?
interface EfficiencyMetrics {
avgToolCallsPerTask: number; // fewer = more efficient
avgRetryRate: number; // retries = wasted tokens
contextUtilizationPct: number; // tokens used / context limit
cacheHitRate: number; // higher = better
parallelizationRate: number; // parallel tool calls / total calls
}
// Detect inefficient agent runs
async function flagInefficientRuns(threshold: number = 3.0) {
const inefficient = await db.query(`
SELECT trace_id, tool_calls_count, retry_count, task_type
FROM agent_runs
WHERE created_at > NOW() - INTERVAL '1 day'
AND (tool_calls_count / NULLIF(expected_tool_calls, 0)) > $1
`, [threshold]);
for (const run of inefficient) {
console.warn(`Inefficient run: ${run.trace_id} used ${run.tool_calls_count} tools (${threshold}× expected)`);
}
}
Dashboard Queries
Key queries for a monitoring dashboard:
-- Cost anomaly: runs that cost 5× the median
SELECT trace_id, task_type, cost_usd, created_at
FROM agent_runs
WHERE cost_usd > 5 * (SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY cost_usd) FROM agent_runs WHERE created_at > NOW() - INTERVAL '7 days')
AND created_at > NOW() - INTERVAL '24 hours'
ORDER BY cost_usd DESC;
-- Quality trend: 7-day rolling average
SELECT DATE_TRUNC('day', evaluated_at) as day, task_type, AVG(score) as avg_quality
FROM quality_scores
WHERE evaluated_at > NOW() - INTERVAL '7 days'
GROUP BY day, task_type
ORDER BY day, task_type;
-- Top failing tools
SELECT tool_name, COUNT(*) as failures, AVG(latency_ms) as avg_latency
FROM tool_calls
WHERE success = false AND called_at > NOW() - INTERVAL '24 hours'
GROUP BY tool_name
ORDER BY failures DESC
LIMIT 10;
Alerting Thresholds
const ALERT_THRESHOLDS = {
costDailyUSD: 100, // alert at $100/day
taskSuccessRateMin: 0.85, // alert if <85% success
toolErrorRateMax: 0.05, // alert if >5% tool errors
qualityDropMin: 0.1, // alert on >10% quality drop
p99LatencyMs: 30_000, // alert if p99 > 30s
cacheHitRateMin: 0.50, // alert if caching efficiency drops
};
See also AI Observability: Trace-Driven Debugging for the per-request tracing layer that feeds these aggregate metrics.
FAQ
How do I monitor quality without LLM-as-judge overhead? Sample 5–10% of requests for quality evaluation. For the rest, track proxy metrics: task completion time, retry count, user feedback signals (thumbs up/down, regenerate clicks).
What's the most important metric to start with? Task success rate. If you can only instrument one thing, make it "did the agent complete the task?" Everything else is secondary.
How do I detect model degradation? Track quality scores weekly. A >5% drop vs. the 30-day average signals a potential model change from the provider. Re-run your eval suite to confirm.
Should I alert on individual failures or trends? Both. Individual critical failures (tool down, cost anomaly) need immediate alerts. Quality and efficiency trends need weekly review — daily noise would be overwhelming.
How do I handle alert fatigue? Tier your alerts: P1 (page someone now), P2 (Slack notification), P3 (weekly digest). Most monitoring alerts should be P3 — only page on things that require immediate action.