AI is changing how enterprise applications behave. As CIOs, we need to think about whether the observability architecture we have built over the years is ready for this change.

Over the years, we have made significant progress in monitoring enterprise IT environments.

We started with infrastructure monitoring—servers, networks, databases, storage, and their availability. As applications became more distributed, our focus expanded to Application Performance Monitoring (APM), distributed tracing, logs, and understanding the dependencies between services.

Today, a reasonably mature observability implementation can help us understand why an application is slow, where a transaction failed, which service is creating a bottleneck, and how infrastructure issues are affecting business applications.

But with AI, and particularly AI agents, I believe we are entering an interesting new phase.

What happens when an application is no longer simply executing predefined business logic, but can decide what to do next, select tools, access information, and initiate actions?

Our existing observability architecture remains relevant. However, the questions we need it to answer are becoming quite different.

AI will appear in more places than we might initially expect

When we discuss enterprise AI, the conversation often starts with chatbots, copilots, or employees accessing LLMs.

But AI adoption is likely to become much more deeply embedded in enterprise architecture.

Consider some fairly practical examples:

These are quite different implementations.

Some applications will simply make calls to LLMs. Others will incorporate AI into predefined workflows. More advanced agents will determine parts of their own execution path, decide which tools to invoke, and potentially interact with other agents.

And some of these capabilities will arrive through the SaaS platforms enterprises already use.

This means AI will not necessarily appear as a separate infrastructure stack that the IT team has explicitly deployed.

AI capabilities and agents may be distributed across enterprise applications, cloud services, integration platforms, and business processes.

Understanding where they operate—and what systems and data they can access—will itself become an important part of enterprise observability.

An application can be healthy, but an agent can still get things wrong

Let me take a simple example from banking.

Imagine a customer disputing a credit card fee.

In a conventional application, the workflow might be quite predictable. The application retrieves transaction details, checks the relevant fee code, applies predefined rules, and presents the outcome.

Now consider an AI agent performing the same task.

The agent might retrieve the customer’s transactions, search the bank’s fee policies, interpret applicable conditions, call a fee-calculation service, and determine whether a refund is appropriate.

Depending on its design and permissions, it may then propose a refund or initiate an approved transaction.

From an APM perspective, everything might look perfectly healthy.

The APIs respond successfully. The database queries complete. There are no application exceptions. Response times are within acceptable limits.

Yet several important questions remain unanswered.

Did the agent refer to the correct policy? Why did it call the fee-calculation service multiple times? Was the refund amount correct? Did the agent have permission to initiate the transaction? Was the refund actually posted correctly?

And, ultimately, was the customer’s problem resolved?

These questions are not easily answered through traditional infrastructure metrics or application traces alone.

This is an important distinction: technical transaction success and successful business outcomes are not necessarily the same thing.

With AI agents, that distinction becomes much more significant.

We may need to look beyond application performance

I see at least six dimensions that will become increasingly relevant when monitoring AI-enabled enterprise applications.

Operational reliability: Is the AI service available? How are LLM response times, tool failures, retries, and agent execution times affecting the application?

Output quality: Are the responses useful, accurate, and grounded in appropriate enterprise information?

Agent behavior: Which tools did an agent invoke? Did it repeat actions unnecessarily? Did it complete its task or get stuck in a loop?

Security and authorization: Which information did the agent access? What actions did it attempt? Were the required permissions and approvals applied?

Consumption and economics: How many model calls and tokens were consumed? What did a completed business task cost, including retries and failed attempts?

Business effectiveness: Did the agent actually achieve the intended outcome? How frequently did people need to intervene or correct its work?

These dimensions are connected, but they do not all come from the same monitoring source.

An APM tool may identify a slow model call. An agent trace may reveal repeated tool invocations. An evaluation process may determine whether an answer was correct. A policy engine may establish whether an action was authorized. The business application must confirm whether the intended transaction actually occurred.

The opportunity is to bring this evidence together.

What would I want to see on an AI observability dashboard?

I don’t think a single dashboard showing LLM response times and token consumption would be sufficient.

Different stakeholders will need different perspectives.

For a CIO, I would want to understand how many AI applications and agents are operating across the enterprise, what they cost, which business processes they support, how effectively they are completing tasks, and where important risks or exceptions are emerging.

For IT Operations, the questions would be more technical. Which model is causing increased response times? Which agent has started making excessive tool calls? Where are failures occurring—in the LLM service, retrieval system, agent orchestration, or enterprise API?

For Security and Governance, the focus would be on the actions agents are attempting, which permissions they use, whether policy controls are working, and whether consequential actions can be independently audited.

All three views should ideally relate to the same underlying execution and business task.

For example, if an agent starts consuming three times more tokens, IT Operations should be able to investigate the execution path. The CIO should be able to see whether the increased cost is affecting business efficiency. Governance teams should be able to establish whether there has also been an unusual change in agent behavior.

That is the kind of connected visibility I believe we should aim for.

Do we need an entirely new observability platform?

My inclination is that we should first examine how far our existing observability investments can take us.

Many of the fundamental capabilities are already available.

We understand infrastructure monitoring, log analytics, distributed tracing, correlation, alerting, and service dependencies.

An LLM invocation is still a service call. A tool execution may still involve an API. An agent may run inside a container or virtual machine. Its dependencies may include databases, queues, and other applications.

These can be monitored using familiar techniques.

What changes is the additional context we need to capture.

This is where OpenTelemetry becomes particularly interesting.

The emerging OpenTelemetry conventions for Generative AI and agent operations provide a way to describe model invocations, agent execution, tool calls, token usage, and related events within distributed traces.

Imagine following a single business request from the application through an agent, into an LLM, across several tools, and ultimately to the enterprise transaction that resulted.

That would provide a much more useful operational picture than looking at separate infrastructure, application, and LLM dashboards.

Of course, the instrumentation needs to be implemented, and the emerging conventions are not yet uniformly supported across all frameworks and platforms.

There is also an important limitation: traces can tell us what an agent observably did, but they cannot independently establish whether its decision was correct or its action appropriate.

For that, we need to connect observability with evaluation mechanisms, authorization records, and business outcomes.

So I see the direction as extending our current observability architecture with AI-specific telemetry and business context, rather than automatically creating an entirely separate monitoring environment for AI.

Where should CIOs begin?

I would start with a simple exercise.

Identify a few AI-enabled applications or agents that are already being used—or are likely to enter production soon.

Map the path from the business request through the AI components, models, tools, and enterprise systems.

Then ask a practical question:

If something goes wrong, can our current monitoring environment explain what happened?

Not just whether the model responded or an API failed, but which actions were taken, which decisions or controls were recorded, what information was used, and what ultimately happened in the business system.

That exercise can reveal the gaps in our current observability architecture much more clearly than starting with a product comparison.

It also helps distinguish what we can already achieve using APM, logs, and infrastructure monitoring from what genuinely requires additional instrumentation, analytics, or governance integration.

A discussion worth having now

I believe enterprise AI observability will evolve much as application observability evolved from infrastructure monitoring.

We did not stop monitoring servers when we introduced APM. We added visibility into application transactions and dependencies because infrastructure health alone was no longer enough.

Similarly, AI agents are adding another dimension: understanding the behavior and outcomes of software that can make choices and initiate actions.

The underlying observability disciplines remain valuable. We now need to connect them to a richer understanding of how intelligent systems operate.

I’ve recently consolidated these observations into a more detailed practitioner-oriented paper, “Observing the AI-Enabled and Agentic Enterprise.”

The paper explores where AI agents fit within enterprise architecture, practical agent-execution scenarios, example dashboards for CIOs and operations teams, the role of OpenTelemetry, the capabilities of existing observability tools, and a reference architecture for extending enterprise monitoring.

In my next post, I’ll share the paper and walk through its key architectural considerations.

For now, I think there is one question worth discussing among CIOs and enterprise technology leaders:

As AI agents become part of our critical business processes, how confident are we that our current observability architecture can explain not just whether they are running, but what they are actually doing?