Field Notes from the AI Wizard: Observability Debt, Gaslighting Metrics, and Why Your Agents Are Already Lying to You

Dispatches from Observability Summit North America, Minneapolis — May 2026 

By Rael — Principal Intelligent Operations Architect @ Evolving Solutions & the woman whose arm literally says “AI Wizard”

I rolled into Minneapolis last month with my staff, my tattoos, and a healthy amount of skepticism. Two days later I left with three things: a stronger appreciation for good beer, a renewed hatred of static SLOs, and a very clear picture of how badly we’re currently screwing up observability in the age of agentic AI.

This is not a polite conference recap. This is the unfiltered field report from someone who spends her days building the systems that are supposed to keep the chaos at bay.

Observability Debt: Your Dashboards Are Lying and They Know It

Let’s start with the cleanest, most brutal definition I heard all week (courtesy of Spoorthi Palakshaiah):

Observability debt is not missing telemetry. 

It is telemetry whose meaning has drifted because the system changed but nobody updated the spells we cast on it.

This is the difference between “we forgot to instrument something” and “we have perfect instrumentation for a system that no longer exists in production.”

Here’s how it actually goes down:

You ship a clean synchronous service. You define `http_request_duration`, wire up beautiful dashboards, set your SLOs, and feel like a responsible adult. Then the architecture evolves — because of course it does. You add async workers, a queue, caching, and split the service. The old metric is still firing. It now only measures the synchronous sliver. The queue backlog? Invisible. The silent failures in Worker B? Never appear in your error rate.

Your dashboard proudly shows p99 latency dropped 64%. Meanwhile real users are waiting 4–8 seconds and 15% of the actual work is failing without ever surfacing.

The SLO didn’t break. It just started measuring the API’s feelings instead of the user’s or the job’s completion.

This is what observability debt looks like in the wild. It doesn’t scream. It quietly gaslights you with green dashboards while the real problems fester underneath.

Technical implication: We broke the causal chain between the metric and the user experience, then acted surprised when the metric stopped being useful. We treat dashboards and SLOs like sacred runes instead of living code that must evolve with the system.

MCP Servers and the Context Window Tax: We’re Doing This to Ourselves

Thomas Johnson laid out the current state of MCP servers with painful clarity.

Right now, many of the popular observability MCP servers are shipping 40+ tools. That’s roughly 20,000 tokens of tool definitions loaded before the agent even gets to ask a single question. In one real measurement, tool definitions alone were eating 144k+ tokens — over 70% of the context window gone before any actual reasoning happens.

This is the context window tax, and it is stupidly expensive.

Even worse, most of these tools are still low-level and data-centric: “get this dashboard,” “query that source,” “list all the alerts.” What agents actually need are goal-centric capabilities: “investigate this error,” “trace this user journey,” “tell me what this deployment actually broke.”

We are repeating the same mistake we made with early microservices — exposing every internal function as a public endpoint and calling it “flexibility.” The winning agent tooling of the next 18 months won’t be the one with the most tools. It will be the one with the smallest, highest-signal, goal-oriented surface.

Wizard’s technical note: Agents perform dramatically better with fewer, semantically distinct tools. We have the research. We’re just choosing to ignore it because building another “get_all_the_things” endpoint feels productive.

The Missing Layer Nobody Wants to Talk About: Decision-Level Telemetry

This was the most technically important thread of the entire conference.

Current OpenTelemetry + MCP gives us four layers:

• Infrastructure (CPU, memory, network)

• Application (latency, errors)

• LLM (tokens, model id, cost)

• MCP Protocol (`gen_ai.tool.name`, conversation.id)

What we don’t have is visibility into whether the decision the agent made was actually reasonable.

We can see that an agent called `DeleteNamespace`. We cannot see whether that decision deviated from expected behavior, violated the scope it was granted, or carried high risk.

Several sessions proposed adding a third observability layer on the same spans using new attribute groups:

• Decision Quality attributes (`gen_ai.agent.tool.confidence`, `gen_ai.agent.tool.risk_severity`, category)

• Permission Scope attributes (what was requested vs granted vs denied)

• Behavioral Baseline attributes (was this tool the expected one for this task? Did it match the baseline pattern?)

This is how we start detecting agent drift before it deletes production.

Without this layer, we are flying blind the moment agents move beyond simple retrieval into actual decision-making and action.

This isn’t a “nice to have.” This is the difference between “our agent used a tool” and “our agent made a bad decision and we have the receipts.”

Real Talk from the People Actually Running at Scale

The Nubank team was refreshingly honest: they evaluated multiple vendors, none of them cleared the bar on cost predictability, compliance, data residency, and integration with their actual stack, so two engineers built their own AI SRE agent in two months. It now runs real investigations in production for under $0.20 each.

Sometimes the build path really is faster and safer than waiting for the perfect vendor solution. Especially when your scale and constraints are non-standard.

The AI Wizard’s Actual Recommendations

Stop treating observability artifacts as immutable. They are code. They must evolve when the architecture evolves.

Stop building MCP servers that look like they were designed by someone who just discovered REST in 2012. Goal-centric, narrow, high-signal tools win.

Start instrumenting decisions, not just actions. The next wave of production incidents will come from agents confidently doing the wrong thing. We need to see that coming.

And for the love of all that is holy, stop celebrating green dashboards when you haven’t validated that the underlying causal model still holds.

The future belongs to the teams who treat observability as a living, decision-aware, closed-loop discipline instead of a collection of pretty lies we tell ourselves at 3 a.m.

I’ll be over here with my staff and my increasingly sarcastic agent monitoring dashboards, waiting to see who figures it out first.

- Your Self Proclaimed AI Wizard (Rael Rodning)

Avatar Note: The red-haired, tattooed, staff-wielding woman in the red pants and glitter sneakers? That’s me. Or at least the version of me that shows up when the dashboards start lying.