Site icon Spherical Cow Consulting

The AI System That Never Was

Heather Flanagan

concept of cloud technologies. artificial intelligence. Blockchain technologies. Big data. Cube with binary numbers in the cloud. 3d render.

“I have an embarrassing backlog of reading about what’s been happening with AI over the last few years. Going through it chronologically turned out to be unexpectedly useful.”

One pattern stood out more than any single policy shift or technical breakthrough: the use and then decline of the phrase “AI system.”

A quick caveat before I go further. I didn’t set out to research the term “AI system.” This pattern emerged while I was reading broadly across policy, standards, and implementation discussions over several months. My source set is diverse, but not exhaustive, which means what follows is an observation from the field, not a claim of statistical completeness.

With that said, if you’re responsible for identity architecture, platform risk, or standards participation inside a real organisation, you’ve probably encountered this already.

You’re being asked to inventory “AI systems,” assign owners, and document risk — but what you actually operate are chains of models, tools, APIs, agents, and delegated workflows that cross teams, vendors, and sometimes jurisdictions.

This post is for the people trying to make that mismatch workable. It’s about why accountability, identity, and risk management are getting harder even as AI tooling improves.

A Digital Identity Digest
The AI System That Never Was

You can Subscribe and Listen to the Podcast on Apple Podcasts, or wherever you listen to Podcasts.

And be sure to leave me a Rating and Review!

Defining AI Systems

“AI system” never really originated from engineering culture. It came from governance. It emerged in the late 2010s as policymakers searched for a neutral abstraction that could hold people, processes, software, data, and organizational responsibility without naming specific technologies. Model was too narrow. Algorithm was too technical. Application was too product-specific. So they reached for system.

You can see that move clearly in early policy work:

In that world, the abstraction made sense. An AI system was something you could point to. It had a name, a vendor, possibly even a deployment diagram. You could imagine building a compliance checklist around it.

But engineers never really adopted the term. They talked about models, pipelines, agents, tools, workflows, and platforms. They shipped stacks, not “systems.” As often happens, how people focused on governance and regulation talk about technology did not align with how people focused on standardizing and operationalizing technology.

That mismatch between the conceptual unit of governance and the operational reality of deployment is now landing on the desks of identity architects, security leads, and standards contributors who are expected to make it enforceable.

From bounded products to fluid systems

The OECD’s earlier work on defining “AI systems” still provides a useful baseline, even if it now feels historically distant:

You can see that framing in the original definitional updates here.

That model worked when AI was mostly a feature embedded inside a product, which still happens in some cases today, but in more cases it has gone well beyond that.

A single user interaction may trigger:

At no point is there a single object that cleanly qualifies as the AI system. 

I expect engineers and technology architects are finding this entirely unsurprising, something of a “well, yeah, of course.” But it also feels like people involved in governance and regulation are missing that complexity and are experiencing misaligned expectations. 

The global divergence nobody talks about

Governments are no longer trying to define “AI system.” They’re drawing boundaries around behaviour, capability, and acceptable use, often without ever naming what, exactly, they are regulating. My suspicion is that many are still assuming the definitions from that early governance work, even as their policies drift far beyond it.

Recent examples from CSIS’s PacTech Pulse newsletter make this visible in practice. A few that stood out:

Each of these policies redefines the scope of AI accountability, but they do it without a shared object to anchor that responsibility. None of them tries to answer the question “what is an AI system?”

And yet all of them answer it in practice.

They define what is in scope, what is out of bounds, and who is responsible through consequences that land directly on consumer platforms, enterprise software, and the identity systems that glue them together, rather than defining terms.

That’s why the language still matters even when it isn’t used. The term is fading, but the expectations attached to it are being rewritten into law, trade policy, education, and economic development, and are being done so differently in every jurisdiction.

The missing chapter: standards are catching up 

The standards community is only now beginning to articulate the problem clearly and to design around it. This is very much a work in progress, and discussions occasionally become heated.

For years, standards work inherited the same language as policy: AI systems, high-risk systems, system lifecycle management. Those terms were good enough while models were embedded in products and objectives were explicit. Agentic workflows break that illusion.

A concrete example is the IETF WebBotAuth working group’s current use-case draft, which explores how bots and agents authenticate and act across domains, explicitly modelling delegation, identity, and authority boundaries rather than assuming a single, bounded system.

This work tries to create language that bridges technology with policy so that we’re all building towards the same thing. Standards language tries for nothing if not being precise in what it’s attempting to accomplish. 

Where governance language starts to wobble

So, this is where the story stops being about terminology and starts being about accountability.

When one country draws AI boundaries through copyright law, another through readiness metrics, and another through patent ethics, “AI system” stops behaving like a technical term. It becomes a policy placeholder, a word that points to responsibility without clearly naming the thing that carries it.

The term still appears in frameworks, white papers, and standards drafts. But it increasingly describes something nobody in production can inventory, because engineers were never building “AI systems” in the first place; they were building chains of capability.

Most governance models still assume that:

In practice, teams are deploying chains of capability, not standalone systems. Accountability blurs not because people are careless, but because the mental model they’re working from no longer matches what’s being built.

Digital identity: when the system no longer has a single “user”

Nowhere does this mismatch between governance abstraction and engineering reality land harder than in digital identity.

Most identity frameworks still rely on a simple triangle:

Agentic AI breaks that geometry almost immediately. We now see:

At that point, asking “who is the user?” stops being helpful. The better question becomes: who is accountable for this action? (Yay for the delegation challenge. It’s a thing.)

This isn’t just a security problem. It’s a language problem that turns into a governance problem.

When an AI agent books a flight, approves a workflow, or triggers a financial transaction, existing identity models struggle to express:

Identity was designed to answer “who are you?” Agentic systems require us to answer “on whose behalf are you acting, under what conditions, and for how long?”

As AI dissolves into workflows, identity stops being a bolt-on control and becomes the connective tissue that makes accountability possible at all.

What this makes possible

The collapse of a single, tidy definition of “AI system” is something we all could see as an invitation to govern the world as it actually exists. Crazy talk, I know. 

When we stop trying to draw boxes around systems, we can start asking better questions:

This opens space for:

In other words, the end of the neat “AI system” may be the beginning of governance that finally matches production reality.

A constructive call to action

Working through that pile of articles, policy drafts, standards notes, and newsletters wasn’t about keeping up. OK, it was a little bit. But it was fun to do because consuming all that material in one big chunk helped me think through patterns that only become visible when I step back: the abstraction we’ve been building regulations and governance around doesn’t match reality enough.

So maybe the question is not:

What is an AI system?

It’s:

What kinds of relationships are we willing to govern, and how?

If you work anywhere near digital identity, platform risk, or standards development, this is where your attention is most valuable:

Let’s line up the pieces so that regulation, governance, and standards support each other the way they need to, especially for the world of AI.

📩 If you’d rather track the blog than the podcast, I have an option for you! Subscribe to get a notification when new blog posts go live. No spam, just announcements of new posts. [Subscribe here] 

Transcript

From Headlines to Patterns [00:00:30]

After spending months immersed in AI governance materials, something shifts.

At first, the focus is on absorbing information. However, once the headlines fade, patterns begin to emerge. That moment—when repetition reveals structure—is what led to this episode’s central insight.

Importantly, this is not a statistically complete survey. Instead, it reflects sustained engagement across:

The value lies not in completeness, but in pattern recognition.

The Quiet Disappearance of the “AI System” [00:01:35]

At the center of this episode is a deceptively simple phrase: AI system.

Once foundational in AI governance conversations, the term is now quietly fading—moving out of governance language and into standards debates.

Most people never noticed the term in the first place. Fewer will notice it disappearing.

Yet for anyone tasked with:

…the problem has already become tangible.

Why “AI System” Never Matched Reality [00:02:30]

In practice, organizations don’t operate singular AI systems.

Instead, they manage interconnected chains of:

These components often span teams, vendors, and even national borders.

And yet, the governance question remains oddly simplistic:
“What AI systems do you run?”

There is no single object to point to—only choreography.

A Governance Term, Not an Engineering One [00:03:20]

The phrase AI system did not originate in engineering culture.

Instead, it emerged from governance efforts in the late 2010s, when policymakers needed a technology-agnostic way to talk about AI.

Other terms felt insufficient:

“System” was broad enough to include people, processes, software, data, and accountability.

This framing worked well for early policy initiatives such as:

At the time, it seemed possible to build compliance checklists around this abstraction.

When Production Reality Broke the Abstraction [00:04:45]

Engineering teams never truly adopted the term AI system.

They built and shipped:

For a while, this disconnect didn’t matter. However, modern production environments have exposed the gap.

Today, a single interaction may involve:

At no point does a single, governable “system” exist.

The Governance Failure Beneath the Paperwork [00:05:45]

This mismatch isn’t just inconvenient—it’s a governance failure.

Why? Because if you can’t clearly name what you’re governing, you can’t:

Identity architects, security leads, and standards contributors are being asked to enforce controls over something that does not exist as a discrete object.

That tension is now unavoidable.

How Governments Are Working Around the Problem [00:06:40]

Rather than defining AI system directly, governments are increasingly governing by consequence.

Examples include:

Each approach avoids defining the system itself. Instead, they decide:

As a result, the concept of an AI system becomes implicit—an uncomfortable and risky place for governance.

Why Standards Bodies Can’t Avoid the Definition [00:07:45]

Standards organizations don’t have the same luxury.

For interoperability to exist, terms must be defined.

Historically, standards inherited policy language, relying on phrases like:

Agentic AI workflows have shattered those assumptions.

A notable example is the IETF WebBotAuth Working Group, which is explicitly defining agents, bots, and authentication use cases to establish shared meaning.

Without definition, interoperability is impossible.

The Coming Identity Crisis in Agentic AI [00:08:30]

Digital identity frameworks are particularly vulnerable.

They were built around a model of:

Agentic AI disrupts this entirely.

Now we see:

The core question is no longer “Who clicked the button?”
It is “Who is accountable, and under what mandate?”

Language as the Foundation of Governance [00:09:00]

The issue is not that AI system is a foolish term.

The real problem is that it never became a shared object of understanding between governance and engineering.

Now, different communities are defining scope, responsibility, and accountability in incompatible ways.

That fragmentation will make governance, standards, and deployment significantly harder to align.

If we cannot clearly name what we are governing, we cannot govern it well.

Closing Thoughts and Next Steps [00:09:23]

As always, this episode is ultimately about language—and why precision matters.

If we want better AI governance, we must bring more rigor to how we describe:

Even when those “systems” are distributed, fluid, and uncomfortable to define.

Thank you for listening. Stay curious, stay engaged, and join the conversation next week.

Exit mobile version