Not perfectly, and not in the fantasy version where software quietly runs the company end-to-end. But well enough to matter. Give a strong model the right tools, permissions, files, browser access, business applications, evaluations, and a decent harness, and it can do useful work: draft customer follow-ups, compare contracts, assemble diligence notes, reconcile numbers across systems, update a CRM, prepare a first pass on a market map.

That shifts the question. The interesting debate is no longer just “can an agent act?” It is what an agent needs in order for its work to compound.

I keep coming back to three primitives: memory, identity, and compute. Identity determines who the agent is acting for and what it is allowed to touch. Compute, or inference, determines how much reasoning and tool use can happen economically. Memory determines whether the agent starts every task from zero or carries forward what the organization has already learned.

This piece is about memory because it is the least settled of the three, and maybe the most strategically exposed. A sales agent that forgets last week’s objection is not a sales agent; it is a very expensive autocomplete. A finance agent that forgets how the company defines ARR will keep re-litigating the same metric. A research agent that cannot remember which sources were rejected, which assumptions changed, and what the team already knows is not building institutional knowledge. It is producing one-off work product.

The first wave of agent tooling was about action. Function calling, browser use, app connectors, orchestration, traces, evals. The plumbing that lets a model do something instead of just say something. Most of that is now built. What’s missing is the layer underneath, the thing that makes the doing accumulate, and it isn’t a longer context window or a vector database bolted onto chat. It’s actual memory: something an agent can write to, revise, retrieve, and be trusted with.

That’s why the memory market got noisy over the last twelve months. Memory-native startups raising real Series A rounds. Frameworks treating memory as a first-class runtime primitive. Vector databases arguing they always were the memory layer. Note-taking apps and meeting recorders quietly hoarding what is probably the most useful context in the company. And every model provider would, naturally, prefer to keep all of it inside their own platform.

The question I haven’t seen anybody answer well is which of those layers actually ends up owning it.

Memory is not chat history

The cheapest move in this market is to treat “memory” as a synonym for “the previous messages.” Save the chat, retrieve from it, call it memory. A lot of products look the same because they all stop there.

Conversation history is the most visible piece, but it’s only the first one. It helps an assistant answer a follow-up. It does almost nothing to make an agent durable.

When I’m trying to think clearly about a memory system, I break it into five layers, because each one fails in a different way.

Session memory is the current thread. Working memory is the task plan, the tool outputs, the files the agent has touched, the approvals it’s waiting on — closer to an OS process than a chat log. Semantic memory is the small handful of stable facts about a user or account, the kind of thing a good account manager keeps in their head (”prefers concise email,” “production database is Postgres,” “security review still open”). Episodic memory is what happened before, and crucially how it happened: past tasks, corrections, failures, what was learned the hard way. And then there’s the layer that actually matters in enterprises but that almost nobody has built well yet, which is governed organizational memory — the shared context with permissions, provenance, and an audit trail, the version a security team will let you ship.

Most products today are still at layer one. The companies I think have a chance of mattering are climbing the stack.

A lot of this is just catching up to the academic work. MemGPT borrowed memory hierarchies from operating systems back in 2023. LangGraph splits thread-scoped state from cross-thread stores. Mem0 explicitly carves memory into conversation, session, user, and organization scopes. Zep’s Graphiti puts time on facts so they can change instead of pretending they’re static. Different framings, same underlying observation: passive storage doesn’t get you there.

Subscribe now

Why a bigger context window isn’t the answer

There’s a tempting argument that all of this just disappears as context windows grow. Put more in the prompt. Let the model read everything. I don’t buy it, and I don’t think anyone shipping into production really buys it either.

A context window is a room. Memory is a life. The room can hold more papers as it grows, sure. It still doesn’t decide which of those papers are true, which are old, which belong to a different customer, or which got slipped in by something pretending to be a paper.

Cost is the obvious objection — stuffing fifty thousand tokens into every answer is a benchmark stunt, not a runtime. Attention is the less obvious one: even when the model can read the whole thing, stale context pulls it sideways in ways you only notice when you’re already in production. Revision is where it gets really uncomfortable, because facts change, preferences flip, policies move, and a plain vector store happily preserves the contradictions until somebody notices. Then there’s governance, which is the one that actually closes deals. Who wrote this? What evidence is behind it? Which agent can read it? When does it expire? Can a user delete it? Can compliance audit it?

The companies taking this seriously are not selling RAG over chat history. They’re trying to own the lifecycle, which is a much harder and much more interesting business.

The enterprise version is shared memory

The consumer version of memory is easy to picture. An assistant that remembers your name, your projects, your tone, the people you usually email. Useful. Almost certainly bundled into ChatGPT, Claude, and Copilot before any standalone company can defend it.

The enterprise version is the harder one, and the more interesting one.

In a company you don’t actually want one shared brain that every agent can read. That’s a breach waiting to happen. What you want is scoped memory: user, agent, project, account, organization, each with its own permissions and retention and a clean answer to “where did this come from.”

The hard part isn’t storing it. It’s the write path. What from a meeting becomes a durable fact? Which exception in a support case turns into reusable process knowledge versus a one-off? Which user correction is a preference, and which one is a long-term lesson the org should learn? Most teams I’ve talked to have no honest answer, because they’ve never had to think about it before. Wikis went stale because nobody updated them. Memory is going to fail in a more interesting way: it’ll get updated too aggressively, by the wrong things, with no obvious way to roll back.

When the enterprise version actually works, the asset isn’t user preferences. It’s the workaround that never made it into the wiki. The customer-specific constraint buried in a sales call three quarters ago. The policy exception from last summer that everyone forgot about until somebody got burned. The objection that keeps coming up on demos. That’s the gap agents fall into the moment they leave the demo and try to do real work.

The market map

With that frame, the market stops looking like a database category and starts looking like a fight over entry points.

This is not a clean category. It’s a stack with overlapping claims, and the most useful question to ask about any company in it is where they enter the workflow. Where context gets created. Where memory gets written. Where it’s stored, retrieved, used. I find the picture below easier to argue from than a list.

Share

A company’s placement here is best read as its wedge, not its destiny. The important pattern is that venture-backed companies are attacking memory from every direction: some sell the memory abstraction directly, some own the agent runtime, some own the knowledge graph underneath, and some capture the context before anyone else sees it.

1 · Memory-native startups

The smallest companies, and the boldest claim: agents need a dedicated memory layer, and that’s the whole product.

Mem0 is the most visible right now — $25M at an $89M post in July 2025, with a deliberately broad pitch: add memories, search them, organize them by scope, plug into whatever framework you’re already using. Zep is the one I find most architecturally interesting. Its open-source project, Graphiti, turns conversation and business data into a temporal graph, and time matters more in enterprise memory than people admit. “True” in a company is almost always “true as of,” and most systems don’t model that at all.

Letta comes out of the MemGPT lineage (Felicis led the $10M seed in July 2024). Their bet is that memory belongs inside the agent process itself, with the agent managing its own memory blocks across sessions, more like an OS than a chatbot. Cognee (seed in February 2026, Pebblebed) is structure-first: entities, relationships, feedback loops, on the argument that retrieval alone won’t get you there. Interloom (seed in March 2026, DN Capital) is the one I’d actually watch for enterprise process memory specifically — it’s closer to “what the company knows” than “what the user prefers.”

The narrower wedges are useful reminders. Supermemory is going after personal knowledge. ByteRover is going after coding context across IDEs, projects, and organizations. Neither is general-purpose enterprise memory, but both show that “memory” inside one high-frequency workflow can be a real product on its own.

The honest question for this whole group is whether a memory-native company can stay independent once OpenAI and Microsoft ship native memory at scale. Their answer has to be depth, not just being early.

2 · Shared enterprise context

Most of the companies in this orbit do not sell themselves as memory companies. They sell enterprise search, AI work assistants, knowledge platforms. The underlying asset is the same: organizational context.

Glean is the obvious growth-stage example, $150M at a $7.2B post in November 2025, with the enterprise data plumbing any memory layer eventually needs. Dust raised $40M from Sequoia in May 2026 and uses the phrase “shared compounding organizational capability,” which is a long way of saying enterprise memory. Writer, Hebbia, Contextual AI, and Sana (Workday picked it up in 2025) all sit nearby. The strategic question for this tier is whether they become the memory layer themselves, or stay as the systems memory-native infrastructure plugs into. I’d guess different answers for different companies, but the ones with the strongest data graphs and the deepest workflow integration have an obvious head start.

3 · Agent frameworks

Frameworks aren’t memory-first, but they own the place where memory ends up being used.

LangChain and LangGraph already treat memory as part of agent state ($125M Series B at a $1.25B post, October 2025). LlamaIndex sits closer to structured access over enterprise data. CrewAI is the multi-agent angle, and multi-agent systems need more memory infrastructure, not less, because state has to survive handoffs. Then Semantic Kernel and AutoGen, which matter mostly because Microsoft can put memory in front of every Office and Azure user on the planet without anyone having to choose it.

The framework advantage isn’t depth. It’s developer surface area. Memory tends to get adopted wherever the agent already runs, which is why this tier is dangerous for memory-native vendors that don’t have a serious framework story.

4 · Retrieval and data substrate

The substrate fight underneath the memory fight. Pinecone, Weaviate, Qdrant, Chroma, LanceDB, Zilliz on the database side. Unstructured on ingest, which is the unglamorous and absolutely essential layer that turns documents, tickets, PDFs, and logs into something a memory system can work with.

These companies don’t have to call themselves memory companies to win a piece of the budget. But a vector DB answers “what’s similar to this.” It doesn’t decide what should be remembered, how a fact should be revised, who can read it, or when it should expire. That gap is precisely where the layer above gets built — and where most of the differentiation eventually goes.

5 · Application wedges

The case I find most underrated. A lot of the most valuable memory in a company is going to be captured at the application edge, not inside an infrastructure product.

Granola — AI notes for back-to-back meetings — closed $125M at a $1.5B post in March 2026. It isn’t pitched as memory infrastructure. That’s exactly why it might end up being some of the most important memory infrastructure in the industry. Meetings are where preferences, decisions, customer objections, and project history actually get created. Read AI, Fireflies, Otter, Limitless — same raw material, different surface.

The awkward implication for investors is that the strongest memory layer might not look like infrastructure at all. It might look like an app the team already loves, that almost as a side effect ends up holding the most useful context in the company.

6 · Model and cloud incumbents

And then the incumbents. OpenAI already ships sessions and a personal memory primitive. Microsoft has Semantic Kernel, Copilot, Azure, and enterprise identity. Google has Workspace plus Gemini. AWS will eventually package memory as a managed service. The platform risk for everyone above is real — if memory becomes a default checkbox inside the model provider, a lot of standalone valuations are going to look quite expensive in retrospect.

What protects the independent vendors is that enterprises mostly don’t want one model vendor holding all of their context. They want it portable across models, auditable across systems, and governed by their own policies. That’s a real opening, but it requires a vendor willing to actually earn the trust, which takes longer than most startups budget for.

What the funding actually says

The funding pattern is messier than the press-release version. Yes, memory-native startups are getting checks. Letta, Cognee, Interloom, Supermemory, Zep, Mem0 — the rounds aren’t trivial anymore.

But the bigger checks are mostly going to companies that already own one of the surrounding control points. LangChain at the framework layer. Granola at the application edge. Glean and Writer at the enterprise data layer. Pinecone, Qdrant, and Weaviate at the substrate. The honest read is that investors aren’t picking a memory winner. They’re spreading bets across every plausible chokepoint near memory, which tells you something about how unresolved this still is.

If I had to write the implicit thesis behind that funding pattern in one sentence, it would be this: the winning memory company will own at least one trusted control point around the write path. It may own the source of context, the policy layer that decides what gets written, or the runtime where memory is used. Ideally it owns two. The uncomfortable middle — thin memory wrapper, no workflow lock, no governance story — is where the squeeze hits first.

What I think matters next

Two things will, I think, separate the durable companies from the ones that look durable for another six months.

Evaluation is going to get serious. Right now most memory benchmarks measure recall on a synthetic set of facts, which is useful but isn’t what buyers actually care about. Buyers care about latency, token cost, freshness, contradictions, audit, deletion, and how the whole thing holds up at ten thousand stored facts instead of ten. As soon as procurement starts asking those questions in RFPs, and they will, memory selection stops being a vibes-based decision.

Security is the other one, and it’s the one I’m most quietly worried about. Prompt injection was bad when it only affected a single answer. The version of prompt injection that writes to persistent memory, and then influences every future answer, is a category worse, and most of the systems shipping today don’t have a credible story for it. The architectures that take this seriously will separate observation from memory writes, require explicit evidence, log everything, and let humans or policy systems review sensitive updates before they land. The companies that do this well will sell to security teams. The companies that don’t will have to explain themselves after the first incident, which usually doesn’t go well.

The rest of the trend list is easier and more obvious. Memory will be multi-layered by default. The write path will start mattering more than the read path. Temporal modeling will become table stakes. Personal memory will be bundled. Enterprise memory will be sold separately, governed carefully, and probably charged for at infrastructure prices.

What separates a real memory company from a wrapper

If you’re building in this market, the easy failure mode is to wrap a vector database, call it memory, and ship the demo. That gets you a seed round. It does not get you to a Series B.

The harder bar, the one I’d actually look for as a buyer or investor, looks roughly like this:

• The write path is real, and the team can talk about it in detail. What gets remembered, why, by what process, and what the system refuses to write.

• Scopes are clean. Session, user, agent, project, and organizational memory aren’t all dumped into one undifferentiated bucket.

• Revision is a primitive, not an afterthought. Contradictions and policy changes are first-class events, not bugs to be patched later.

• Provenance is visible. Every memory has a clear answer to “why is this here.”

• Permissioning is built in from day one. Retrofitting access control onto a shared context graph is a nightmare I have personally watched kill more than one internal project.

• Distribution exists. Memory alone is not a workflow; it has to live where the agents and users already are.

For investors, my heuristic has gotten simpler over time: where does context originate, and does this company have a credible path to it? Infrastructure sits downstream of the apps that create knowledge. Apps own the raw material but rarely build the infrastructure. Frameworks own the developer. Incumbents own the identity and the runtime. The strongest position is having a believable path to the source — not just a place to store what somebody else collected.

The end state, roughly

I don’t think agent memory settles into one product category. It ends up spread across the stack. Model providers ship native session memory and personalization. Frameworks expose stores and APIs. Databases optimize for memory-shaped workloads. Apps keep hoarding the context users create every day. Memory-native startups try to stitch a governed, model-neutral layer across all of it.

The unsettled question is which of those layers gets to decide what the agent knows. Whoever earns that decision ends up sitting in a strange and powerful spot — close to identity, close to workflow, close to the data, neutral enough that nobody minds.

My bias, for what it’s worth, is that the biggest independent opportunity is governed memory for enterprise agents. Personal memory is going to be useful, but it’s already being bundled by the assistants people open every day. Enterprise memory is harder. It needs permissions, audit, revision, cross-system context, and the kind of trust that takes years to build. It has to work across models. It has to survive procurement. It has to be legible to a security team whose job is to find reasons to say no. That’s a lot of barriers, and that’s exactly why somebody is going to build a real company out of it.

For the last two years, the agent conversation has been about action. Can the model call the tool. Browse the web. Write the code. Send the email. Update the CRM. The next two years are going to be about memory. Does the agent remember what it learned last time. Does it know what’s still true. Does it know what it isn’t allowed to use.

The companies with good answers to those questions are going to be worth a lot. The ones still saving every chat into a vector store and calling it memory are going to look quaint by 2027.

That’s the market I’m watching, anyway.

Share

The numbers come from very different companies in very different sub-segments. They are all describing the same thing: the layer of the AI stack that turns a model’s output into a token in a customer’s response. Inference serving. That is the layer this memo is about.

Inference is now where the money in AI flows. By the end of this year, somewhere around two-thirds of AI compute spend will be inference, up from roughly one-third in 2023. Training gets the headlines; inference is the recurring revenue. The companies that get this layer right will compound for the next decade.

What follows is my best attempt at a current map. About eighty named companies organized into four horizontal sub-layers plus a column of modality verticals. Funding figures and ARR through late May. A take, layer by layer, on where margin actually accrues.

The stack

There is no agreed-on map of how inference serving is organized. Here is mine.

The Chips at the bottom are what hardware actually runs the inference. The Engines are the open-source runtimes (vLLM, SGLang, NIM) that turn a model file into served tokens. The Clusters are the companies that operate the fleets at scale — AWS Bedrock at one extreme, Fireworks and Cerebras Inference at the other. The Gateway is the access layer developers actually call through — OpenRouter, Vercel AI Gateway, lab SDKs.

Running alongside all of that, the modality verticals: companies that own the chip choice, engine, cluster, and API surface within one slice of the world. ElevenLabs in voice. Black Forest Labs in image. Runway in video. Voyage in embeddings (now part of MongoDB). Poolside in code. Each of these is its own stack, vertically integrated, capturing the full margin within its modality.

The next figure is the map of who actually sits where.

The map

The Chips

This layer is the most obvious story in AI infrastructure, so I’ll be brief. NVIDIA cleared $75.2 billion in datacenter revenue last quarter at 74.9% overall gross margin. AMD has become a credible second source with the MI355X and MI450, helped by the OpenAI six-gigawatt commitment. Hyperscaler custom silicon (Trainium, TPU, Maia, MTIA) is a margin lever inside each hyperscaler, not an external market. Cerebras just IPO’d at $56 billion. Groq took $17 billion from NVIDIA in a licensing deal. SambaNova reset to $2.2 billion after a tough year.

The interesting bets on the silicon board are less about NVIDIA vs. challengers and more about whether two specific outliers work. Transformer-specific ASICs — Etched ($500M) and MatX ($500M, led by Jane Street and Situational Awareness) — are the cleanest binary in the stack. If transformers stay the dominant architecture through 2028, dedicated silicon outperforms general-purpose by an order of magnitude. If they don’t, both go to zero. The South Korean edge-inference cluster (Rebellions at $400M pre-IPO, FuriosaAI at $125M, Hailo going public via SPAC) is the quieter story: real money chasing inference-specific designs at the edge, where NVIDIA’s networking lock-in doesn’t apply the same way.

The take: chips is settled territory at the top and interesting at the edges. NVIDIA wins the obvious moat. The asymmetric bets sit in transformer ASICs and edge inference silicon, not in the wafer-scale plays competing with NVIDIA at the datacenter scale.

The Engines

The engine is the layer between a model file and a stream of tokens. It is where research papers like PagedAttention, continuous batching, speculative decoding, and FP8 quantization turn into running code. And it is where the open-source layer has won so completely that there is almost no margin left for pure-play vendors.

vLLM, originally out of UC Berkeley, is the default. SGLang is the faster alternative for certain configurations — xAI uses SGLang to serve Grok. NVIDIA TensorRT-LLM was the previous answer; NVIDIA is transitioning to Dynamo, a newer disaggregated serving framework that separates prefill from decode. NIM, NVIDIA’s containerized inference microservices, has become the enterprise standard way to ship an LLM in 2026. TGI from Hugging Face. llama.cpp for CPU and edge. DeepSpeed-Inference for the more research-oriented workloads. All open source.

The commercial layer above OSS is small and absorbing fast. Modular MAX is Chris Lattner’s commercial play. Friendli Engine is Korean and currently ranked the fastest GPU-based serving provider by Artificial Analysis. Predibase specializes in serving LoRA fine-tunes. Each occupies a defensible niche, but none of them is a venture-scale standalone business.

The cleanest signal that engines don’t capture margin alone is that the most successful proprietary engines aren’t sold as products. Fireworks’ FireAttention is internal. Together’s TGI fork is internal. Baseten’s optimization stack is internal. Each is a competitive moat for the cluster it powers, not a standalone product line.

The take: this layer has already settled. OSS dominates. Real margin gets captured one layer down (NVIDIA with NIM) or one layer up (clusters with their internal optimizations). I don’t see a pure-play engine vendor reaching $100M of ARR.

Share

The Clusters

This is where the dollars are. And where the four most recent up-rounds happened.

Together at $150M+ ARR. Fireworks at $315M ARR and reportedly in talks at $15 billion, four times its $4 billion Series C valuation from seven months earlier. Baseten in talks at $11 billion in May, up from $5 billion in January. fal.ai at $400M ARR and in talks at $8 billion in March. DeepInfra closed $107 million Series B in May with NVIDIA and Samsung Next participating, and is now processing close to five trillion tokens per week. All of this in five months.

The cluster layer breaks into four very different business models, and the difference matters more than the funding numbers do.

Hyperscaler-managed (AWS Bedrock, Azure AI Foundry, Google Vertex AI, Databricks Mosaic AI) wins where customer data already lives. AWS Bedrock is growing 180% year over year in adoption. Over 100,000 enterprises run Claude on Bedrock as of April 2026. The hyperscalers don’t need to win on price or speed; they win on the customer not wanting to move their data out of the perimeter they already trust.

Independent neoinference (Together, Fireworks, Baseten, Modal, DeepInfra, fal.ai, Anyscale, RunPod, Friendli) is the most contested sub-layer. Open-weight serving on commodity GPUs at competitive cost-per-token. Margin depends on whether open-weight share of inference keeps growing. It is. Whether $15B for Fireworks is the right price is a separate question.

Hardware-native vertical (Cerebras Inference, Groq Cloud, SambaNova Cloud) is the only credible NVIDIA bypass at the cluster level. Own the chip, own the cluster, capture the full vertical margin. Limited to what your silicon can serve well — Cerebras for long-context flagship models, Groq for low-latency open-weight serving.

Edge serving (Cloudflare Workers AI, Apple Intelligence on-device, Fastly Compute@Edge) is small but growing fast. Cloudflare’s Q1 2026 revenue hit $640 million on +34% growth, with Workers AI billing driving the upside. The December 2025 acquisition of Replicate added a model marketplace on top of the edge cluster — the only player in this layer attempting all four horizontal positions at once.

The headline numbers belong to the top of the cluster layer. The more interesting question is what’s happening underneath them, because that’s where the next category-creating outcomes will come from. Featherless.ai closed a $20 million Series A from AMD Ventures and Airbus Ventures in April for serverless hosting of 30,000+ open-weight models — an order of magnitude more breadth than Together or Fireworks. TensorWave is the AMD-exclusive specialist, with the largest AMD GPU training cluster in North America (8,192 MI325X accelerators) and a clear thesis that AMD second-source becomes a real category. Vast.ai runs a decentralized GPU marketplace with 42 GPU model families. Simplismart in India is in talks for $20M led by NVIDIA at a $100M valuation. Below Featherless: Beam Cloud, Cerebrium, Inferless, Novita, Salad Cloud serving the long tail at $5–30M of disclosed funding each.

The take: clusters is where the VC dollars are landing right now, and where the next valuation reset will start. The top of the layer (Fireworks, Baseten, fal.ai) is priced for compounding; the bottom is where the asymmetric returns sit if you can pick which thesis-specific play (long-tail breadth, AMD specialization, regional focus, GPU marketplace) actually has structural advantage. Hardware-native vertical and hyperscaler-managed are the most durable sub-models. Independent neoinference will consolidate aggressively over the next 18 months — my best guess is two or three names at the top survive at scale, three or four emerging names break out, and the rest get acqui-hired or quietly fold.

The Gateway

The thinnest layer in the stack. Marketplaces (OpenRouter, Hugging Face Inference), gateways (Portkey, LiteLLM, Vercel AI Gateway, Cloudflare AI Gateway), smart routers (Not Diamond, Martian), observability-plus-routing (Helicone). Each one sits between a developer’s code and the inference endpoint of their choice.

The category exists because enterprises don’t standardize on one model. The current shape: most enterprises run AI across five or more surfaces — Anthropic direct, OpenAI direct, AWS Bedrock, Azure AI Foundry, Google Vertex, plus AI features in their data platform, dev tooling, CRM, and support stack. Someone has to manage that, log it, govern it, fall back when a provider has an outage. Gateway tools do.

But the take rate per call is thin and the moat is mostly switching cost. And here is the structural risk that decides the layer: the frontier labs are absorbing the function with their own SDKs. Claude SDK plus MCP. OpenAI Agents SDK. Google AI SDK. Each one increasingly assumes you call its parent lab’s models, with routing and orchestration logic embedded in the SDK itself.

Portkey’s response in March 2026 was to open-source the entire gateway under Apache 2.0. LiteLLM was OSS to begin with. The category is racing to commodify itself before the lab SDKs eat it.

The take: gateway is the most likely layer to consolidate or get absorbed in the next 18 months. The winners are the ones that become infrastructure rather than products. OpenRouter has the best position because it operates as a true marketplace with credit-based billing — the lab SDKs structurally cannot aggregate competitors’ models the way OpenRouter can. The OSS gateways (LiteLLM, Portkey) win as embedded primitives. Everyone else gets compressed.

Modality verticals

The other half of the inference serving story. Some companies don’t sit on a horizontal layer at all — they own all four (chip choice, engine, cluster, API) within one modality, and capture the full margin within their slice.

Voice has gone from $315 million of VC in 2022 to $2.1 billion in 2024 to multi-billion-dollar valuations in 2026. ElevenLabs raised $500 million Series D at $11 billion in February. Vapi raised $50 million Series B at $500 million in May after winning Amazon Ring over 40 competitors. Deepgram raised $130 million Series C at $1.3 billion in January. Cartesia is the latency leader. Hume is the empathy and emotion play. Retell hit $40M+ ARR with 300% quarter-over-quarter user growth. AssemblyAI, PlayHT, and Bland round out the field. This is the hottest sub-category in inference right now, and it is at the early stage of consolidation. ElevenLabs is priced as the winner; the rest disagree.

Image has been a category for two years now and has stabilized. Black Forest Labs ($3.25 billion in December 2025) is the open-weight standard. Ideogram and Recraft are the major paid challengers. Midjourney is profitable, private, and notably has never taken venture funding. The category is more stable than voice because the open-weight base (Flux) sets a price floor that everyone else respects.

Video is the fastest-moving. Runway, Luma ($900 million round), Pika, Kling (ByteDance). The cost curve is dropping faster than text — what cost roughly $1 per second to generate in 2024 is closer to $0.10 in 2026. Hollywood-grade output is the differentiator that’s actually worth paying for.

Embeddings consolidated first. MongoDB acquired Voyage AI in February 2025 for $220 million. Cohere has embed-v4 inside its $7 billion model business. Nomic ships open-source. Jina has multilingual long-context. The category was small enough that a horizontal infrastructure player (MongoDB) could roll up the leader. That pattern will repeat in other modalities.

Code is the most contested vertical. Poolside is in talks at $12 billion with $1 billion already committed, including $500 million to $1 billion from NVIDIA. Magic is targeting $1.5 billion-plus. Reflection AI is earlier-stage. But the more important move is that the application-layer leaders — Cursor at $50 billion in talks, Cognition at $25 billion — are increasingly running their own inference internally rather than buying from cluster providers. That is a sign the vertical-by-modality thesis extends up the stack into applications themselves.

The take: vertical integration is real in modalities where the demand profile is distinct (voice latency, video frame coherence, embedding semantics). Whether each modality consolidates to a single winner or stays a multi-player market is open. Voice and code look like they consolidate. Image and video look like they stay multi-player. Embeddings already consolidated — into an adjacent infrastructure vendor, which is a model I expect to repeat.

Underexplored corners

If you spent twenty minutes looking at the map above, you noticed that the obvious bets are now priced for compounding. Cerebras is public at $56 billion. Fireworks is in talks at $15 billion. ElevenLabs at $11 billion. Cursor at $50 billion. The headlines are fully claimed.

The interesting question, especially for venture investors entering today, is which underexplored corners of the inference stack become the next category-defining names. Four worth watching closely.

Confidential and private inference. Tinfoil is the cleanest pure-play: multi-GPU confidential computing on NVIDIA Hopper and Blackwell, hardware-enforced privacy, Red Hat partnership. Apple Private Cloud Compute exists at hyperscaler scale but isn’t available to other developers. Edgeless Systems and Phala Network occupy adjacent slices. The category exists because every regulated industry (healthcare, finance, defense, legal) needs a way to call frontier inference without sending data outside its trust boundary. Today there is essentially no neutral commercial answer. Whoever builds it first becomes infrastructure.

Decentralized / verifiable inference. Akash Network already runs a permissionless GPU marketplace with instant Llama and Stable Diffusion deployment. Gensyn launched Delphi on an Ethereum L2 in April with a Reproducible Execution Environment that produces cryptographic proofs of correct compute. io.net, Render, Aethir round out the DePIN side. The thesis is contested — crypto-infrastructure for AI compute has been promised for three years and mostly underdelivered — but the primitive (verifiable compute the user doesn’t have to trust) is genuinely useful, especially for agentic workloads that move money. Worth watching whether 2026 is the year it actually ships.

Real-time generative video. Decart closed $300 million in May from Radical Ventures, NVIDIA, Sequoia, Benchmark, Adobe Ventures, and Toyota Ventures for what it calls real-time world models — sub-30-millisecond generative video built on a proprietary optimization stack (DOS). Runway and Luma can’t reach this latency profile with batch generation. If real-time video opens a category (gaming, robotics simulation, AR), Decart sits alone in it. The bet is binary and big.

Long-tail open-weight serving. Featherless hosts 30,000-plus open models, an order of magnitude beyond Together or Fireworks. The economic question is whether depth-of-catalog matters more than speed-on-Llama for the next $100 million of cluster ARR. If you believe the open-weight long tail keeps fragmenting (as Hugging Face suggests it does), Featherless is a structurally different bet from the leaders. AMD Ventures led the Series A, which is its own signal about where AMD thinks the open-weight serving market goes.

Adjacent to all four: voice frameworks (Pipecat from Daily.co, Vocode, Ultravox) running as open-source plumbing under most production voice agents, and AMD-specialized clusters like TensorWave, where the bet is that AMD second-source becomes a real category over the next 24 months.

None of these are guaranteed. All of them are priced one or two orders of magnitude below the category leaders — which is exactly why they belong on a market map for investors, not just an industry roundup for incumbents.

Share

Where margin lives

The scorecard above is what it actually feels like to be a company at each layer. The pattern that emerges is the one that holds across most large infrastructure markets: margin lives at the edges, and the middle is where the squeeze happens.

Chips at the bottom: high pricing power, high capex, low substitution risk for NVIDIA, multi-year time-to-resolution. Modality verticals at the top of their slices: meaningful pricing power, moderate capex, moderate substitution from open weights, two-to-three-year window before the leader is visible by name.

The middle layers each have their own story. Engines are the cleanest negative: OSS-dominant, hard to charge, already settled. Clusters are the most interesting because the recent up-rounds suggest investors believe the category compounds — but substitution risk is high and the time-to-resolution is short. Gateway is racing the lab SDKs and the next 18 months decide whether it survives as a venture category at all.

If I had to allocate venture dollars into this map today, I’d bias roughly 35% to chips and silicon-native vertical serving, 30% to modality verticals (especially voice and code), 20% to a small number of horizontal cluster names (Fireworks and Together, possibly Baseten if the premium-quality thesis holds), 10% to gateway plays biased toward OSS-distributable infrastructure (OpenRouter, LiteLLM), and 5% to engines, which mostly means betting on the people behind the open-source consortia rather than the consortia themselves.

Speed vs. cost

The chart above shows what each major serving endpoint actually delivers. Two patterns jump out.

First, the upper-left quadrant — fast for cheap — is mostly owned by hardware-native plays. Cerebras at 3,000 tokens per second on gpt-oss is in a class by itself. Groq at 750-plus tokens per second on Llama 70B is the closest competitor. SambaNova rounds out the trio. These speeds are inaccessible to GPU-based serving without paying premium prices.

Second, the lower-right — slower and more expensive — is where the frontier labs sell direct. GPT-5.5 and Claude Opus 4.7 deliver around 80 tokens per second at $25 to $30 per million output tokens. That is the price of brand, trust, and managed support, not raw performance. The same Sonnet 4.6 served on AWS Bedrock isn’t dramatically faster than calling Anthropic direct, because it’s the same model on the same hardware mediated through enterprise contracts.

The middle is the bidding war. Fireworks, Together, DeepInfra, Cloudflare Workers AI, Gemini 2.5 Flash all sit within roughly a 3× envelope on both axes. This is where price competition is most intense, where investors are paying the highest revenue multiples, and where the next eighteen months of consolidation will play out.

What I think happens

ree years and roughly $700 billion of committed compute. $109 billion of VC into AI infrastructure in 2025 alone. The headline names on the map are priced for compounding. By any normal standard, the obvious bets are made.

What’s actually interesting about inference serving is not which obvious bet wins. It’s that this is a market where capital is uncapped, demand is uncapped, and the bottleneck moves between layers faster than in any prior infrastructure cycle. Six months ago the binding constraint was H100 supply. Now it’s HBM and advanced packaging. In 2025 it was datacenter power. The layer that captures margin this quarter isn’t necessarily the layer that captures it next quarter. Most of the layer-winner predictions you’ll read this year — including some of the ones above — will look wrong by 2027.

So the question worth answering is the one I keep coming back to. Which positions in the stack are structurally impossible for OpenAI, Anthropic, and Google to take themselves, even if they wanted to?

I count three.

Confidential inference the customer can cryptographically verify, which the labs can’t credibly offer because they are the trust boundary. Edge and on-device serving at sub-30-millisecond latency, which they can’t reach without hardware they don’t own. And specialty silicon designed for inference workloads the labs are too generalist to optimize.

Those are the three positions where asymmetric upside actually lives. Not because they’re the biggest categories today — two of them barely exist as categories — but because the labs can’t replicate them by next quarter’s release, the way they can absorb gateway, agent SDK, or generic horizontal serving.

Sources

• Cerebras IPO: TechCrunch; CNBC.

• Fireworks AI: Bloomberg, $15B talks; Sacra for ARR.

• Baseten: $11B talks; Sacra.

• fal.ai: Sacra ($400M ARR).

• Together AI: Tracxn.

• DeepInfra: Series B announcement.

• Modal: $87M Series B Sep 2025; PitchBook.

• Groq + NVIDIA license: IntuitionLabs writeup.

• SambaNova: Sacra.

• MatX: TechCrunch.

• Etched: Bloomberg.

• ElevenLabs: Series D announcement.

• Vapi: TechCrunch.

• Deepgram: Deepgram.

• Black Forest Labs: TechCrunch.

• Luma Labs: $900M round (per industry reporting).

• MongoDB / Voyage AI: MongoDB IR.

• Cloudflare / Replicate acquisition: Cloudflare press, December 2025.

• Poolside: Sacra.

• NVIDIA Q1 FY27: NVIDIA IR.

• Anthropic ARR May 2026: Bitget aggregating Anthropic disclosures.

• Salesforce Agentforce ARR: Futurum Group / Salesforce.

• Databricks: Databricks IR.

• Performance benchmarks: Artificial Analysis.

• VC market sizing: OECD AI VC report through 2025.

That is a more serious claim than “Gemini got better.” It is also a more dangerous one. A model competes for attention. An agent competes for delegated authority.

The phrase on stage was “the agentic Gemini era.”

I think the strategy is plain and simple: Google wants Gemini to become the execution layer across Search, Chrome, Android, Workspace, YouTube, Shopping, Cloud, and eventually glasses. It wants the assistant to move from a product you open to a system that works through the products you already use.

For investors, that makes I/O a capital allocation story as much as a product story. Does agentic Google defend Alphabet’s profit pool? Does it expand it through Cloud, subscriptions, and commerce? Or does it raise the cost of protecting Search while dragging the company deeper into an infrastructure arms race?

My view: I/O strengthened the bull case that Google has the best distribution in AI. It did not settle the harder question of return on invested capital. The product story is credible. The capex story is now too large to treat as background noise.

Share

Search Is The Hinge

The most important I/O announcement in my opinion was Search.

Google said AI Mode has surpassed one billion monthly users one year after launch, while AI Overviews now has more than 2.5 billion monthly active users. Gemini 3.5 Flash becomes the default model in AI Mode globally. The Search box itself is being redesigned for longer, multimodal questions using text, images, files, videos, and Chrome tabs.

The product direction is clear: Search is becoming a workspace for intent. You ask, follow up, monitor, compare, generate, and act without always leaving the surface.

For investors, that cuts both ways. Bulls can point to query expansion: Google says AI-powered Search features make people use Search more, and Q1 Search and other revenue grew 19%. Bears can point to value migration: if Google gives users the synthesized answer, the publisher, merchant, or app behind that answer loses leverage.

The old Search bargain was imperfect but legible: publishers created content, Google organized it, users clicked out, and ads funded the system. Agentic Search changes the bargain. If agents monitor the web, summarize changes, create dashboards, and trigger actions, the click becomes optional. That may be good for users. It is not automatically good for the open web.

Time will tell whether AI Mode increases commercial query volume without degrading ad pricing, merchant relationships, or regulatory tolerance. The top-line Search number can look healthy even while the ecosystem underneath gets angrier.

Spark Is Where Google’s Distribution Advantage Becomes Obvious

Gemini Spark is the product that best explains the keynote. Google describes it as a 24/7 personal AI agent inside the Gemini app. It runs on dedicated virtual machines on Google Cloud, uses Gemini 3.5 and the Antigravity harness, works in the background, and is planned to connect first to Google’s own tools and later to third-party tools through MCP.

The comparison here is not another chatbot. The comparison is the new class of open and semi-open agents: OpenClaw, Hermes Agent, browser-use stacks, local computer-use agents, hosted coding agents, and the growing ecosystem of tools that can browse, click, run code, manage files, use APIs, and remember context across sessions.

OpenClaw and Hermes are important because they prove the appetite. People do want agents that operate software instead of merely talking about it. They want agents that can use a browser, carry memory, spawn subagents, connect to messaging tools, run in the background, and do multi-step work without being babysat minute by minute.

But they also expose the tax. A serious self-hosted agent asks the user to think about models, API keys, browser profiles, local versus cloud browser execution, tool permissions, operating system access, memory, storage, gateways, and security posture. That is fine for developers, researchers, and power users. It is not how a billion people adopt a new behavior.

This is where Google’s advantage is not subtle. Spark does not have to persuade a user to install an agent framework, configure a browser sidecar, pick a model provider, connect a calendar, wire up email, and decide how much of the file system to expose. Google already has the account, the inbox, the calendar, the docs, the browser, the phone, the payment surface, the search history, and the cloud runtime.

That does not make Spark automatically better than OpenClaw or Hermes. It makes the adoption problem different. Open agents are agent-as-project. Spark is agent-as-default. That distinction matters.

For a technical user, OpenClaw or Hermes may remain more flexible, more inspectable, more portable, and easier to bend into odd workflows. For the mainstream market, setup friction is destiny. The average user will not choose an agent because it has the most elegant tool registry. They will use the agent that appears inside the surfaces where their life already happens.

The danger is that distribution can hide complexity without removing it. When an agent reads Gmail, edits Docs, checks Drive, browses with Chrome, and acts through connected tools, the permission problem becomes larger, not smaller. Google says Spark will ask before high-stakes actions like sending emails and adding calendar events. Good. But confirmation prompts are not enough. Persistent agents need visible progress logs, scoped permissions, audit trails, reversible actions, and one-click revocation. “Under your direction” is not a trust model. It is a slogan until the interface proves otherwise.

Cloud Is Where The Story Gets Expensive

I/O sounded like a product keynote. Underneath it was an infrastructure keynote.

Pichai said Google is processing more than 3.2 quadrillion tokens per month across its surfaces. Model APIs process roughly 19 billion tokens per minute. More than 8.5 million developers are building with Google’s models monthly. Over 375 Google Cloud customers each processed more than one trillion tokens in the past 12 months.

Those numbers matter because they connect the consumer AI story to Cloud. Agents need persistent environments, tool execution, inference, storage, security boundaries, and observability. OpenClaw and Hermes show the primitive version of this: an agent needs a place to run, a browser to control, tools to call, memory to carry forward, and logs that explain what happened. Google is trying to industrialize that stack.

If Google can turn Gemini into both a consumer interface and a developer platform, Cloud becomes more than a place to rent GPUs. It becomes the operating venue for agent workloads. Antigravity 2.0 matters for the same reason. It is not just a coding environment. It is Google building the control plane for managed agents.

The Q1 numbers support the bull case: Google Cloud revenue grew 63% to $20.0 billion, operating income reached $6.6 billion, and backlog nearly doubled quarter over quarter to more than $460 billion. That is no longer a hobby business.

But the cost side is just as important. Alphabet’s Q1 capital expenditures were $35.7 billion, up 107% year over year. Free cash flow was $10.1 billion for the quarter, down 47% year over year in the company slides. Pichai told I/O that annual capex was $31 billion in 2022 and is expected to be about $180 billion to $190 billion this year.

This is the investor tension in one sentence: Alphabet has the distribution to make AI ubiquitous, but ubiquity is capital hungry.

The open-source agent market makes this tension easier to see. When a power user runs an agent locally, the compute bill is explicit. When Google embeds agents into Search, Gemini, Workspace, and Android, the user experience feels free or bundled, but the cost does not disappear. It moves onto Alphabet’s income statement.

Omni Is A Creative Weapon, Not Yet A Moat

Gemini Omni was the keynote’s obvious showpiece. It can combine text, image, video, and audio inputs, then generate and edit video through conversation. Google says characters remain consistent, physics hold up, and each edit builds on the last.

If it works in the wild, Omni compresses a creative workflow that currently spans ideation, generation, editing, compositing, voice, and revision. That is valuable. It also enters a brutal market. OpenAI, Adobe, Runway, Midjourney, Meta, and a long tail of video-model labs are all fighting for the same creative surface.

Google’s advantage is not merely model quality. It is placement. Omni can live in Gemini, Flow, YouTube Shorts, Workspace, Android, and eventually Search. OpenAI can create a better standalone creative model and still have to fight for workflow distribution. Adobe has the professional creative surface. Meta has social distribution. Google has YouTube, Android, Workspace, and Search. The model race matters, but the placement race may matter more.

The weak point is trust. High-quality synthetic video makes provenance a product requirement. Google said SynthID has watermarked more than 100 billion images and videos plus 60,000 years of audio assets, and that it is expanding SynthID and Content Credentials verification into Search and Chrome. That is the right move. It will still be tested by scale, adversarial reuse, and consumer indifference.

Where Agents Become Money

Universal Cart may be more commercially important than the video demos. Google says people shop across Google more than a billion times a day, supported by a Shopping Graph with more than 60 billion product listings. Universal Cart works across Search, Gemini, YouTube, and Gmail. It can watch price drops, flag compatibility issues, surface loyalty perks, and connect to agentic checkout through Universal Commerce Protocol.

This is not just convenience. It is Google trying to keep commerce intent inside its perimeter as agents start shopping on behalf of users. If successful, Universal Cart helps Google defend shopping ads, increase transaction proximity, and give merchants a controlled path into agentic commerce. The careful line is that the brand remains merchant of record. That line exists for a reason.

This is also where Google differs from most agent startups. A startup agent can help you shop. Google can see the search, compare the merchant, surface the video review, remember the Gmail receipt, connect payment, and close the loop through Shopping. That is not just a better assistant. It is a tighter transaction graph.

Workspace has a different revenue path. Gmail Live, Docs Live, Keep voice capture, AI Inbox, Pics, and Spark integration are not keynote fireworks. They are subscription retention features. They move work from typing to supervising. They also give Google a practical answer to Microsoft: Gemini is not only a model; it is embedded where the documents, messages, files, and meetings already sit.

Microsoft’s advantage is the enterprise desktop and Office relationship. Google’s advantage is web-native collaboration, Search, Gmail, Android, and consumer reach. The contest is not “which model writes a better paragraph.” It is which company can turn everyday workflows into agent workflows without forcing users into a new operating manual.

The Glasses Are Optionality

Android XR eyewear was framed carefully. Audio glasses come first this fall, with display glasses later. Google is working with Samsung, Qualcomm, Gentle Monster, and Warby Parker. The pitch is practical: spoken help, navigation, messages, photos, and Gemini assistance without pulling out a phone.

This is not Google Glass 2.0 in posture. It is Gemini looking for a socially acceptable body. The device matters less than the interface lesson: an agent that can see, hear, and respond in the moment becomes a different category of product.

The risks are familiar and not solved by better industrial design: ambient capture, social discomfort, battery life, latency, and permission fatigue. The opportunity is also familiar: if glasses work, the phone is no longer the only personal computing surface that matters.

The Bull Case, The Bear Case, And The Honest Middle

The bull case is strong. Alphabet has more AI distribution than any company in the world. It has Search, Android, Chrome, YouTube, Workspace, Cloud, custom silicon, and a subscription base already paying for AI plans. If agents become mainstream, Google can place them inside defaults.

The open-agent market makes that advantage clearer, not weaker. OpenClaw, Hermes, and similar projects show what the frontier user wants: agents that use tools, browse, remember, schedule, code, delegate, and keep working. Google is trying to take that behavior out of the terminal and put it inside the everyday internet.

The bear case is not that Google lacks AI. That argument is tired. The bear case is that agentic AI compresses margins, cannibalizes clicks, invites regulation, and forces an infrastructure arms race where even winners spend more to stay even.

The honest middle is where I/O landed. Google showed the most coherent agent distribution strategy in consumer technology. It also showed why the next phase of AI will be judged less by model demos and more by execution quality, trust design, and cash returns.

Google has spent two decades organizing information. At I/O 2026, it made a new promise: Google will organize action.

That promise is powerful. It is also the point at which users, publishers, merchants, regulators, and investors will ask for proof.

What To Watch Next

VC-backed companies don’t usually need PE. Three linked moves in less than two weeks between the two biggest labs and several of the largest alternative-asset managers tells you something has shifted underneath the asset class.

The bull read is that this is a distribution move, and a smart one. The labs have plenty of money. What they don’t have, and have never had, is a channel into enterprise. The bottleneck in enterprise AI was never awareness; it has always been implementation. Change management. Workflow redesign. Governance. Internal trust. PE portfolios are an unusually clean answer: a thousand mid-market companies with concentrated ownership, ROI mandates, and operating teams that move fast. Pair model capability with deployment muscle at that scale and the labs stop being model companies. They become operating platforms. Anthropic framed it as a shot at the consulting industry. OpenAI is taking the same shot, except they’re also underwriting the financial outcome.

That is the part to slow down on. You don’t normally promise institutional investors a fixed return on the hottest asset class of the decade. You don’t trade super-voting shares for $4B unless the structure of the deal matters to you more than the dilution does. Both are concessions a company usually makes only when distribution, capital structure, or capacity matters more than clean optics.

Being a frontier lab is much harder than the press releases make it look.

You run two compute footprints at once. One for inference, one for research, and they fight each other for the same scarce GPUs. You burn operating cash on training runs whose economic payoff depends on a generation of models you haven’t shipped. You price into a market where the buyer can replace you, on most workloads, with an open-weight model that landed yesterday. And every twelve months you have to renegotiate your supplier relationships, because the underlying technology curve keeps moving the floor under your feet.

The five biggest deals the labs have signed since October read as a single move: trading future equity, IP exclusivity, and multi-decade purchase commitments for capacity they can’t otherwise secure. OpenAI committed $300B to Oracle over five years for 4.5 GW of Stargate capacity. AMD got six gigawatts of OpenAI demand for the MI450, with a warrant letting OpenAI take up to 10% of AMD itself. Microsoft restructured the partnership in OpenAI’s favor in late April — non-exclusive IP through 2032, multi-cloud serving allowed, AGI clause removed — in exchange for $250B of incremental Azure spend. AWS added $5B of equity, $20B of milestones, and a $100B Anthropic-to-AWS commitment over a decade. SpaceX added 300 MW on top of that.

Roughly $700B of pre-committed compute, capital, and equity is now booked between the labs and their suppliers. None of that comes back at the prices the same labs are charging at the bottom of their menus.

Premium reasoning has converged near $25–30 per million output tokens. Workhorse has settled at $15. The floor has collapsed: Gemini Flash-Lite at $0.40 is roughly 75 times cheaper than GPT-5.5. Caching cuts another 90% on cache hits. Batch APIs cut 50%. Run that next to the $700B of committed compute and the 17.5% starts to look less like strength and more like a coupon you have to offer when the rest of the deal isn’t pricing itself.

Share

That is the bear read. The labs are nervous, they have reasons, and the deals are tells.

The bull read is one you have to take seriously. It is also the part that keeps the bear case from being too easy.

The Jevons paradox is simple: when a technology becomes more efficient, total consumption can rise instead of fall. Nineteenth-century Britain made steam engines more efficient; instead of burning less coal, it found more uses for coal. The cheaper input did not shrink the market. It expanded the surface area of demand.

Inference has the same setup. If a token gets 90% cheaper, companies do not merely spend 90% less on the same chatbot tickets. They start running model calls inside workflows that used to be too expensive to automate: compliance review, support QA, code migration, procurement, claims, email, internal search. Unit price falls. Usage explodes.

That pattern has shown up across compute-driven platform shifts. Electricity in the 1920s. Cloud in the 2010s. Bandwidth before that. Each time analysts modeled “what happens when this gets cheap,” they undershot, sometimes by an order of magnitude. Output per dollar kept falling, but dollar spend per company kept rising, because the workloads that became economically viable at the next price point were larger than the ones that were viable at the last.

If Jevons kicks in for inference the way it has for everything else like it, the labs grow into their $700B of pre-committed capacity. The 17.5% becomes easy to clear. The PE-portfolio deployment channel turns out to be the early flush of a much larger enterprise pipe. The deals that read as nervousness today read as foresight in 2028.

The deals the labs signed this month are the actions of operators who believe the demand curve bends up just in time. The PE coupon is what being early to that demand costs you if it shows up.

That is the question worth thinking about as an investor. So here is my view, and you can hold me to it.

Jevons holds. Total inference spend grows non-linearly over the next three years; the labs are right about demand. But the rising tide doesn’t lift the labs the way they’re pricing for. A big chunk of that demand migrates to open-weight models running on commodity infrastructure, because that is precisely the workload that becomes economically viable when the floor sits at $0.40 per million tokens. The frontier labs grow into the $700B in absolute terms. They don’t grow into it at premium per-token margin.

Both halves of the trap turn out to be real. PE-portfolio cost-reduction workloads flow to open weights — most of them shouldn’t even try a frontier model for what they’re doing. But Jevons-driven demand also creates new workloads at the top of the curve that genuinely need premium reasoning, and those compound into real revenue for whoever sits at the frontier. The labs end up bigger than today and proportionally less dominant of the stack than today, simultaneously. That is the part the consensus model hasn’t priced.

The consequence is for the layers in between.

Hardware wins under almost any version of this scenario. If volume explodes, NVIDIA wins. If open-weight serving wins the floor, NVIDIA still wins, because open-weight serving runs on the same silicon. AMD gets enough through the MI450 deal to be a real second source. The 70% gross-margin layer is the safest bet in the entire stack, and almost nobody writes about it because the story isn’t sexy.

The serving layer becomes interesting in a way it isn’t today. Fireworks, Together, and the next set of fast-inference companies aren’t intermediaries — they are how the open-weight slice of the substitution wave gets monetized. If open-weight share of the workhorse tier triples, you’re underwriting a serving market that triples with it.

The control plane gets pulled load-bearing if enterprises end up routing across five or more models in real time. Whether it stays a venture-fundable category depends on whether the labs let it stay neutral. They probably won’t, and that’s the risk.

Applications with real workflow lock-in keep doing fine. Cursor, Harvey, Sierra are not really model-economics businesses; they are software businesses that happen to use models. The companies that pretend models are their product end up exposed to whichever direction the pricing curve next moves.

Generic frontier API consumption — thin wrappers, model resellers, undifferentiated chat layers — gets compressed from both sides. That’s the layer that looked huge in 2024 and gets quietly written down in 2026.

The companies that compounded out of the cloud transition weren’t the ones that read the trend best in 2008. They were the ones who guessed correctly when each layer would unlock and got there one quarter before the consensus did. The same exercise is now live in inference.

So the trap is real on both sides, but it isn’t symmetric. The labs are mostly right about Jevons. They are probably wrong about how much of the rising demand they themselves keep. The best positioning is one layer over from where the frontier is fighting — silicon, open-weight serving, routing if it stays neutral, and applications that own a workflow customers can’t detach.

The 17.5% is the loudest tell we’ve gotten that the labs already know what they’re holding. The next chapter doesn’t go to whoever ships the smartest model. It goes to whoever follows the volume to where the silicon, the serving, and the workflow stack up — and gets there one quarter before the consensus catches the rotation.

Next week · Part II

The Inference Market Map

A layer-by-layer field guide to the inference stack. Six layers, twenty-eight categories, ~120 named companies, with funding, ARR, and a candid take on which layers compound and which get squeezed — and how the answer rotates as the demand curve scales. Plus the bets I’d make and the ones I’d avoid. Subscribe to get this in your inbox.

If agents impressed you, the next leap is harder to mentally price in. The most important AI products of the coming decade may not look like chatbots or browser agents at all. They may look like drones that inspect power lines, robots that work inside factories, medical devices that diagnose at the edge, satellites that turn earth observation into decisions, semiconductor systems that reduce dependency, and energy hardware that learns from the grid it supports.

Here the game changes. AI software is already not classic SaaS; inference has a cost, reliability has a cost, and every agentic workflow has to be evaluated, monitored and made trustworthy. Once AI leaves the browser, though, mistakes become physical. A bad answer can be regenerated. A drone crash, a failed battery pack, a faulty sensor or a robot arm near a worker cannot be debugged with the same casualness.

Reality shows up earlier. Dust. Heat. Supply chains. Regulation. Safety. Manufacturing yield. After-sales service. The demo is no longer enough.

Which is why India’s current deeptech push feels worth paying attention to. The country has spent the last decade proving that it can build large digital public rails, a serious startup ecosystem, and a growing electronics manufacturing base. Now the question is different: can India build companies that do not just sit on top of technology created elsewhere, but create hard technology themselves?

The stakes are not only startup-sized. They are national capability-sized. If Indian founders build only the software wrapper or services layer around foreign hardware, we participate in the next wave. If we build the sensors, compute systems, motors, materials, robotics, power electronics, testbeds and manufacturing know-how underneath it, we compound with it.

China is the uncomfortable comparison because it is already showing what this looks like at scale. Not because India should copy it. It cannot, and probably should not. China has its own distortions, from overcapacity to state-directed excess. But it makes the deeptech challenge concrete. Manufacturing capacity, domestic demand, supplier density, research spending and capital have become one compounding system. India is now putting more serious money behind deeptech. Necessary, yes. Sufficient, no.

Subscribe now

China’s advantage is a flywheel

China’s hardware lead usually gets told through brand names: DJI in drones, BYD and CATL in electric mobility and batteries, Unitree in robots, Shenzhen as the default mental image of rapid hardware iteration.

Under those logos is the more important story. China has built a dense loop between research, components, factories, customers and deployment. A robotics company can find suppliers nearby. A battery company can scale into a massive domestic EV market. A drone company can learn across consumer, agricultural, industrial and public-sector use cases. A factory automation supplier can sell into the world’s largest manufacturing base and get better because customers are close, demanding and numerous.

The International Federation of Robotics estimated that China represented 54% of global industrial robot deployments in 2024, with 295,000 installations. India, by comparison, reached a record 9,120 installations and became the sixth-largest installer worldwide. That is meaningful progress for India. It also shows the distance.

Batteries and EVs tell the same story. The International Energy Agency says China has nearly 85% of global battery cell production capacity, with substantial shares in cathode and anode active material production. Its Global EV Outlook 2025 noted that China sold more than 11 million electric cars in 2024, more than the world sold just two years earlier. The Chinese advantage is not one company. It is stack control.

And in embodied AI, the Chinese state is already worrying about a problem most countries would love to have: too many humanoid robot companies chasing similar products. In November 2025, China’s National Development and Reform Commission said the country had more than 150 humanoid robot companies, more than half of them startups or cross-sector entrants. The warning was about overheating, but it also revealed the depth of the pipeline.

A flywheel looks like this. Not a slogan, a loop. Capital funds companies. Companies use local suppliers. Suppliers improve because startups are demanding. Factories become customers. Customers create data and production feedback. Better products attract more capital. The state, for better and worse, keeps nudging the loop toward sectors it thinks matter.

Underneath the loop sits research spend. A NITI Aayog discussion paper puts China’s 2024 R&D intensity at 2.68% of GDP and India’s historical level around 0.7%. China’s own 2025 statistical communique says R&D expenditure reached 2.80% of GDP in 2025. That gap does not mean India cannot win. It means India has to be much more deliberate about where it tries to win first.

Software founders often talk about distribution as the moat. Hardware founders know that the moat can be the place where a thousand small execution details are easier for you than for anyone else.

Share

India is finally treating deeptech like a system

Encouragingly, India is no longer treating deeptech like a loose collection of science projects. Several policy moves now point in the same direction.

The Draft National Deep Tech Startup Policy, released for public consultation in 2023, was an early signal. Its framing was broader than funding: commercialization of research, IP, talent, procurement, infrastructure and national priorities. That matters because deeptech does not fail only because founders cannot raise money. It fails because labs, markets, regulators and capital often move on different clocks.

Then came more concrete capital formation. In July 2025, the Union Cabinet approved the Research Development and Innovation Scheme with a corpus of Rs. 1 lakh crore. The structure matters: long-term concessional loans, possible equity for startups, and potential contribution to deeptech fund-of-funds vehicles. Finally, at least on paper, financing that looks closer to the shape of R&D risk.

In April 2026, Startup India Fund of Funds 2.0 was notified with a Rs. 10,000 crore corpus. Its language is important. It explicitly includes deeptech startups with longer R&D cycles and higher costs, smaller AIFs and micro-VCs, tech-driven manufacturing startups, and a broader sector-agnostic segment. It also contemplates larger corpuses, longer-duration AIFs, higher contribution for deeptech and manufacturing, and a moderated investment multiplier for specific segments.

Private capital is waking up too. The Nasscom-Zinnov Indian Tech Start-up Report 2026 says India now has more than 4,200 deeptech startups and that deeptech funding rose 37% to $2.3 billion in 2025. A better base than the popular narrative admits. But the same report also says AI accounts for 84% of deeptech startups and 91% of deeptech funding. Strength, yes. Warning too. India’s deeptech boom can still become too software-shaped if hardware, manufacturing and components do not catch up.

Piece by piece, the stack is getting parts it did not have before. Electronics became India’s third-largest export category in FY 2024-25, and ECMS is trying to move the base from assembly toward components. India’s drone ecosystem now has more than 38,500 registered drones, nearly 39,890 certified remote pilots and 244 approved training organizations as of February 2026. Space has a dedicated Rs. 1,000 crore VC fund under IN-SPACe. The IndiaAI Mission has onboarded more than 38,000 GPUs for common compute. And after two more approvals on May 5, 2026, the India Semiconductor Mission reached 12 approved projects with cumulative investment of around Rs. 1.64 lakh crore.

I do not read these as isolated programs. I read them as India beginning to understand that deeptech needs a stack. Compute has to meet products. Capital has to meet components. Manufacturing has to move beyond assembly. Startups need customers willing to buy before the product is obvious.

Good news, with an asterisk: the stack still has gaps.

Capital alone does not suffice

Money is the easiest part to announce. That is the trap. Deeptech is partly a funding problem, but only partly.

Deeptech companies need patient capital because the technical risk is real. They also need early customers who can tolerate imperfect but improving products. They need testbeds where prototypes can fail safely. They need procurement systems that do not punish a young company for not looking like an old vendor. They need universities that translate research into companies without drowning founders in IP friction. They need component suppliers. They need manufacturing partners who can handle low-volume iteration before scale. They need regulatory clarity. They need engineers who have touched the material world, not only built cloud workflows.

Here deeptech stops behaving like normal venture. A SaaS company can fake completeness for longer. A robot cannot. A drone motor cannot. A battery management system cannot. The product either survives the field or it does not.

For that reason, China is a useful comparison. Its advantage is not that every policy worked. Many did not. The advantage is that when a sector becomes strategic, capital, demand, manufacturing and talent often point in the same general direction. India has to find its own democratic, federal version of that coherence.

Another easy mistake: assuming India’s first deeptech proof point has to be the glamorous one. Robotaxis make for great headlines. They are also a brutal first exam for India because open-road autonomy has to deal with chaotic traffic, liability, mapping, weather, infrastructure and public trust.

Autonomy absolutely belongs in India’s future. The first wins, though, may come from less cinematic places: mines, ports, warehouses, factories, campuses, defence corridors, farms, highways under controlled inspection workflows. The more bounded the operating domain, the easier it is to create a product that works, gets paid for, and keeps improving.

As an investor, this is the interesting part. India may not win by manufacturing a direct copy of every Chinese hardware category. It can win by starting where local demand is unusual, painful and large.

The first wedges

Some of the most investable Indian deeptech categories may look slightly boring from far away and incredibly important up close. Boring is not a weakness. Often, boring is where the budget sits.

For venture capital, the diligence lens changes. “How big is the market?” is still there, but it is no longer enough. Where does the company sit in the value chain? What part of the stack does it own? Can it manufacture or partner its way into production? Does the customer have budget today? Is the technical milestone tied to a commercial milestone? Does regulation help, hurt or simply slow the loop? Can the company learn faster in India than a global competitor can learn from outside India?

At first, the best Indian deeptech companies may not look like global category winners. They may look like oddly specific answers to very Indian constraints: a mining automation system that works in dust and heat, a crop-spraying drone network that understands smallholder economics, a power electronics company that survives grid variability, a medical device that works outside tier-1 hospitals, a satellite analytics company that sells decisions rather than imagery. Specific is not small. Specific is the wedge.

No fund announcement will decide India’s deeptech moment. The real test is more mundane: can a founder building a drone motor, a medical imaging device, an industrial robot arm, a semiconductor packaging tool, a satellite subsystem or a battery management platform find the capital, components, test site, manufacturing partner and first customer without spending half the company’s life stitching the ecosystem together manually?

Benchmark: not a press release, not a demo day, but a lower-friction path from lab to market.

Cautiously bullish, then. Not press-release bullish. The direction is right. FoF 2.0 recognizes that deeptech and manufacturing need longer-duration capital. The RDI Scheme recognizes that private-sector R&D needs different financing instruments. ECMS recognizes that electronics value capture depends on components. IndiaAI recognizes that compute is infrastructure. The drone, defence and space pushes recognize that government demand can help create early markets.

Still, the work ahead is less glamorous than the announcement. India has to make deeptech investable and buildable at the same time.

Winning the next AI wave will not merely mean having the best models. It will mean having the best loops between intelligence and production. China has built one version. India now has a chance to build another, suited to its democracy, markets, constraints and founders.

Becoming a slightly cheaper consumer of other people’s hardware is not enough. The goal has to be more ambitious than that: one of the places where the physical future gets invented, tested, manufactured and exported.

Harder game. Better game.

Share

References
Startup India Fund of Funds 2.0 Gazette notification, April 13, 2026.

• Cabinet approval for the Rs. 1 lakh crore RDI Scheme, July 1, 2025.

• Draft National Deep Tech Startup Policy public consultation release, July 31, 2023.

• International Federation of Robotics, World Robotics 2025 global release.

• International Federation of Robotics, India 2025 release.

• China NDRC update on embodied intelligence, November 27, 2025.

• International Energy Agency, Batteries and Secure Energy Transitions.

• International Energy Agency, Global EV Outlook 2025.

• Nasscom-Zinnov Indian Tech Start-up Report 2026 release.

• NITI Aayog discussion paper on India’s R&D vision.

• PIB update on Electronics Component Manufacturing Scheme, October 27, 2025.

• PIB update on India’s drone ecosystem, February 2026.

• Cabinet approval for Rs. 1,000 crore space VC fund under IN-SPACe, October 24, 2024.

• IndiaAI Mission compute update, April 2026.

• May 2026 semiconductor approvals under India Semiconductor Mission.

On December 27, 2024, Bench emailed roughly 12,000 customers to tell them that the bookkeeping platform they had been paying for was gone. Shut down the same day. No wind-down, no export flow, no tax-season safety net. Three days later, Employer.com bought the corpse and revived the login page.¹

For most of the last decade, Bench was the company venture investors pointed to when they wanted to describe tech-enabled services done well. It had raised over $100 million. It had built what looked like software: a dashboard, an API surface, an onboarding flow. But behind the UI were hundreds of bookkeepers in Vancouver, and as the company scaled, those bookkeepers did not get cheaper. Gross margin did not migrate in the direction SaaS investors were underwriting. Wage inflation compressed the very metric that was supposed to widen over time. The software never ate the service. The service ate the software.²

Three days after Bench disappeared from the market, Sequoia published the essay arguing that the next trillion-dollar company would look like a services firm on the surface, because selling finished work is a larger prize than selling access to the tool that produces it. They called the model Service as a Software.³ I took that thesis up in last week’s post; the short version of the conclusion was that the framing is correct, narrower than the slogans, and decided at the workflow level rather than the category level. This essay is the workflow-level version. I’m going to take a single domain — accounting — and read it carefully. It is the cleanest place in the whole economy to see where Service as a Software actually holds and where it doesn’t.

Subscribe now

Why accounting is the right domain to test the thesis

Every AI-services essay picks customer support as its proof case. I’ve written that essay too. Support is valid, but it is overfit. Volumes are high, every account has a knowledge base, and the action space is almost entirely text. The architecture generalizes poorly. If you want to pressure-test whether a category actually supports Service as a Software, pick a domain where the workflow touches a system of record, regulatory scrutiny, and real money — and then look for the specific sub-workflows inside it where the properties still hold.

Accounting is exactly that domain. About 1.6 million bookkeeping, accounting, and auditing clerks and another 1.6 million accountants and auditors work in the United States alone.⁴⁵ Their output is almost entirely structured. The ground-truth signal is a ledger, not an opinion. A significant portion of their time sits in recurring, bounded tasks with deterministic correctness: did the cash hit, does it match the invoice, does the sum reconcile to the bank statement, did the journal entry respect accounting policy. And — this is the commercially interesting bit — an enormous installed base already pays SaaS vendors for the tooling that surrounds those tasks: Intuit, NetSuite, Sage, BlackLine, FloQast. That tells you the budget line exists. It just sits on the wrong side of the execution fence.

But “accounting” is not one workflow. Invoice matching and SEC-footnote drafting both show up under that label and they have almost nothing in common architecturally. If you stop at the job title — a mistake I see in a lot of thesis decks — you end up with a $200B TAM story and no product. The honest map goes one level down, to the workflow slice. That is also how the new companies in this space are segmenting themselves, because the sub-workflow is the actual unit of competition.

The 2×2: where the accounting stack actually sits

The framework I'll use is the same 2×2 from last week's post: how verifiable the output is (ground truth), and how expensive a single mistake is (recovery cost). Sub-workflows inside accounting spread across all four quadrants, which is why the category cannot be discussed in aggregate. The matrix tells you which sub-workflow can support outcome pricing and which cannot, regardless of how good the model gets.

The accounting stack, workflow by workflow

Take the 2×2 seriously and you can map the actual companies being built against it. I’m going to go sub-workflow by sub-workflow, because the shape of the market only resolves at that level.

AP matching and invoice coding

This is the cleanest Q1 workflow in the entire back office. Every invoice has a purchase order or a contract against which it can be checked. A vendor either exists in the master file or it doesn’t. Coding rules are discoverable from prior behavior. The right answer is provably right. Vic.ai reports clients reaching 85% no-touch processing within six months; its transportation case study cites 99% coding accuracy at 84% no-touch.¹⁰ Ramp’s AP agents, designed around the same workflow, claim end-to-end autonomy on a meaningful share of invoices and have pulled close timelines forward by days for customers that move off legacy AP tools.¹¹ Pricing here is already per-invoice or per-document, which is the architectural tell: nobody in this segment sells by the seat anymore.

Bank reconciliation and cash application

Also Q1. This is where Numeric has built its wedge. At Brex, the company reports that cash-match rates moved from 30% to over 90% after Numeric’s reconciliation agent was deployed.⁶ That number looks small until you realize that reconciliation is the single biggest time sink in a monthly close, and that the distance from 30% to 90% is the distance between “close takes two weeks” and “close takes three days.” Numeric has since raised a $51M Series B on the back of that wedge and announced plans to expand across the close.⁷ The customer list — Brex, OpenAI, Plaid, Wealthfront — is the list you’d expect: AI-native finance teams who are comfortable replacing human match logic with a software agent because they already think in systems.

Bookkeeping for SMBs

This is the zone Bench tried to own and lost. The work is Q1 on paper — categorize transactions, reconcile accounts, produce monthly financials — but for businesses under $5M ARR, the real work is less “apply rules” and more “chase the founder for a receipt.” Unstructured input dominates. And the margin arithmetic is brutal: an average SMB bookkeeping contract is in the low hundreds of dollars per month, which is not enough revenue to support a dedicated human reviewer but is more than most of these businesses will pay for pure software. Puzzle has taken the software-first approach here (pricing tiers of $249, $399, and $599/month; 15% MoM growth at last disclosure; over $66M raised) and is working to pull the human review cost out of the bundle as automation improves.¹² Digits has gone harder — launching what it calls the Autonomous General Ledger in March 2025 and layering AI agents on top of it in June — and has ~$100M in capital to prove out the model.⁹ Truewind positions as an AI staff accountant sold through firms rather than directly, and counts EisnerAmper and Frank Rimerman among its customers.¹³

Pilot is the informative contrast. Pilot hit $27M in revenue in 2024 with ~250 employees, and has publicly committed to a hybrid architecture — AI plus a dedicated Nashville-based bookkeeper on every account.¹⁴ That is a deliberate product choice. Pilot’s bet is that the segment will reward trust and accountability more than raw margin compression, at least for another cycle. It is a real bet, and it also explains why Pilot’s numbers look like services numbers — because, structurally, they still are.

The tell: in SMB bookkeeping, the first question I ask is whether a new customer’s cost-to-serve in month twelve is meaningfully below month one. If yes, the company is on the AI-native curve. If no, they have built a better-looking Bench.

General ledger and ERP replacement

This is the biggest swing. Rillet, which raised a $25M Series A from Sequoia in May 2025 and a $70M Series B co-led by Andreessen Horowitz and ICONIQ three months later, is explicitly positioning itself as an AI-native replacement for NetSuite for mid-market companies.⁸ The pitch is simple: if you were going to rebuild NetSuite today with modern infrastructure and LLM-backed workflows, it would look nothing like NetSuite. The migration story — Rillet quotes 4-to-6 weeks to move off a legacy GL — is aggressive, but the customer list (Windsurf, Decagon, Postscript, Bitwarden) is real and the pace of capital formation is telling investors something. Note that this is not “agent replaces accountant.” It’s the system of record itself getting a rebuild, with agents as a native part of it.

Close management

This is Q3 in the matrix — mixed ground truth, serious consequence — and it’s where the honest pricing answer is “hybrid.” Numeric came up through close. FloQast has incumbency; BlackLine has the enterprise footprint. The agentic unlock here is the assembly and explanation of the close — flux analysis, variance commentary, tie-out packages — not the judgment about whether a transaction should have been booked in the first place. A 2025 MIT/Stanford field study of 79 SMBs found an average 7.5-day reduction in monthly close time for SMBs using generative AI in accounting, with AI-using accountants supporting 55% more clients per week.¹⁵ Those are sizable productivity numbers, but they are productivity numbers. They compound into faster closes and higher throughput per accountant, not a Service-as-a-Software business model.

Audit and judgment work

This is Q4 — where, I’d argue, it is a mistake to try to sell the outcome. A missed disclosure or a misapplied ASC 606 interpretation has a tail that outweighs any vendor’s willingness to carry the liability. The interesting companies here are the ones that explicitly choose not to pursue the outcome pitch. Double — the company formerly called Keeper, which rebranded in late 2025 and raised a $6.5M Series A backed by Jack Altman and Y Combinator — is explicit: its thesis is to double the accountant’s capacity, not replace the accountant.¹⁶ That is a Q3/Q4 product built honestly. It will look less glamorous than an autonomous ledger, and it will probably be a better business for a long time.

Pricing is a consequence

One reason this category gets muddled is that people treat pricing like a strategic lever when it is actually a tell. Pricing reveals what the vendor thinks it owns. If a company charges for seats, it thinks it has sold a tool. If it charges per task, it thinks it has sold a machine. If it charges per outcome, it thinks it has sold a finished result — and it is willing to carry the downside of being wrong about that claim.

In practice, almost every AI-services company sits on a ladder. The ladder has four rungs, and the rung tells you which architecture the company has actually built.

Most AI accounting companies sit on Rung 2 or Rung 3. Vic.ai and Ramp are Rung 2: they charge per document processed. Numeric and Rillet are Rung 3: they charge for the workflow they own. Nobody credible has tried to price SMB bookkeeping on Rung 4 — charge $X per month-end-closed — because the exception rate is too high. Bench, for what it's worth, was a subscription business wearing a services sweater; pricing signaled "bounded monthly service," but cost structure was "ongoing salary commitment," which is how it died.

The margin slope test

The honest way to distinguish tech-enabled services from AI-native is not to read the pitch deck. It’s to look at the gross-margin layer cake in year one versus year three. The shape is different. Classic SaaS was 80% gross margin from day one because the marginal cost of a new customer was near zero. AI-native ledger businesses are generally not 80% gross margin in year one — they carry meaningful inference cost, QA cost, implementation cost, and trust overhead. The question is whether those layers compress over time as coverage improves and the backlog of workflow templates grows.

For a tech-enabled services company, those layers don’t compress — they scale with revenue, because they’re denominated in humans. That’s the Bench failure mode. For an AI-native company, the hypothesis is that by year three, implementation is a one-week onboarding, QA is absorbed into the model, and the only meaningful variable cost is inference. That’s not a guarantee. It’s a bet, and it’s the bet that matters.

The incumbents don’t sit still

One of the lazy moves in every AI-agent deck is to assume a clean greenfield. Accounting is not greenfield. It has three well-funded incumbent classes and all of them are moving.

The legacy software vendors — Intuit, Sage, Oracle (NetSuite), Workday — own distribution and the system of record. Intuit has shipped Intuit Assist inside QuickBooks, NetSuite has AI layered into its finance suite, and the big platform vendors are well aware that a $100M-funded AI-native ERP is trying to reforge the GL beneath them. Their risk is not capability; it’s incentive. Every one of them prices per seat. A product that truly automates a seat cannibalizes its own revenue line. That’s the classic innovator’s-dilemma setup, and it explains why the AI features from incumbents look like tools (Rung 1) rather than agents (Rung 3 or 4).

The BPOs and global SIs — Accenture, Genpact, Concentrix, Deloitte — are moving the fastest of the three classes, and also have the most to lose. Accenture reported $2.7B in generative-and-agentic AI revenue for fiscal 2025 and $5.9B in AI-related bookings; Julie Sweet called out agentic AI by name on the earnings call as an inflection point.¹⁷ Genpact disclosed Data-Tech-AI revenue of $2.44B and Digital Operations revenue of $2.64B for 2025, positioning around agentic solutions.¹⁸ Concentrix named 2025 a pivotal year for its iX Hello and iX Hero autonomous-CX offerings.¹⁹ They have the workflows, the SLA experience, the trust relationships, and the labor pools. What they don’t have yet is the product-margin slope — their org charts are still indexed to headcount, and the AI revenue they report is often sold into the same workflows they currently staff.

The AI-native entrants are the interesting ones. They’re greenfield on product and pricing, carrying no legacy revenue to protect. But they have to build trust, integration depth, and operational maturity into a category that punishes sloppiness. It is not a foregone conclusion that they win. It is a foregone conclusion that they force the category to evolve.

What should Investors look for

Taking this seriously as an investor has changed the questions I ask. Model novelty barely enters the conversation now. Workflow economics are the whole conversation.

Bottom line

Service as a Software is real, and it’s narrower than the slogan. It shows up where a software vendor can credibly own the execution of a workflow without paying for the labor that workflow implies. That requires a specific set of properties: structured ground truth, bounded liability, enough repetition to keep improving, and an exception rate that falls over time. When those properties hold, the vendor can charge per completed unit of work and the business starts to behave like software. When they don’t, the same company behaves like Bench.

Accounting is the most legible place to watch this play out because the category has clean sub-workflows sitting in every quadrant of the 2×2. AP matching and bank reconciliation are already at Rung 2 pricing with margin slopes that look right. Mid-market GL is being rebuilt from scratch in a way NetSuite cannot defend with an AI feature flag. Close management and audit work will be accelerated, not owned. SMB bookkeeping is the hardest test and the most instructive one: it is where we find out if software can finally do the thing Bench tried to do, with an architecture Bench never had.

The temptation with every new category is to stretch the thesis until it fits everything. The more useful move here is the opposite: take the thesis at its strongest form, apply it only where the workflow properties support it, and underwrite the margin slope instead of the narrative. The companies that will matter from this cycle aren’t the ones that called themselves AI agents earliest. They’re the ones whose human-layer percentage shrinks, quietly, cohort over cohort, until the ledger is genuinely running itself — and the humans around it are doing the work software was never going to do anyway.

Share

References

1. Bench to be acquired after abruptly shutting down, TechCrunch, Dec 30, 2024. Confirms ~12,000 customers cut off on Dec 27, 2024 and Employer.com acquisition three days later.

2. “Bench Accounting — Why Tech-Enabled Services Without Moats Break,” Build Mode, on Bench’s labor-indexed gross margin and inability to compress delivery cost at scale.

3. Sequoia Capital, “Services: The New Software,” Sonya Huang and Pat Grady, March 2026.

4. U.S. BLS: Bookkeeping, Accounting, and Auditing Clerks. ~1,613,400 jobs (2024).

5. U.S. BLS: Accountants and Auditors. ~1,579,800 jobs (2024).

6. Numeric grabs $28M Series A to automate accounting using AI, TechCrunch, Oct 10, 2024. Brex’s cash-match rate moved from 30% to over 90% after deploying Numeric.

7. Numeric Series B announcement, November 2025; total disclosed funding ~$89M with $28M Series A (Menlo, Oct 2024) and $51M Series B.

8. Rillet Series B announcement, August 2025; $25M Series A (Sequoia, May 2025) and $70M Series B (a16z + ICONIQ, Aug 2025). ~200 customers including Windsurf, Decagon, Postscript, Bitwarden.

9. Digits Autonomous General Ledger and AI Agents launch; ~$100M raised from Benchmark, SoftBank, GV.

10. Vic.ai: Countsy case study; 84% no-touch, 95% accuracy. See also product page citing up to 85% no-touch within six months.

11. Ramp: Agentic AI for Accounts Payable; end-to-end AP autonomy with close acceleration.

12. Puzzle: $30M round announcement; ~$66.5M total; tiered pricing starting at $249/month.

13. Truewind YC profile; 100+ customers, firm partners include EisnerAmper and Frank Rimerman.

14. Pilot.com 2024 revenue and headcount (via Latka); $27M ARR, ~250 employees, ~1,000 customers.

15. “AI cuts monthly financial close time by 7.5 days,” CFO Dive on MIT/Stanford 2025 field study of 79 SMBs.

16. Double (formerly Keeper) Series A announcement, December 2025; explicit augment-the-accountant thesis.

17. Accenture FY25 results: $2.7B generative + agentic AI revenue, $5.9B AI bookings.

18. Genpact FY2025 results; $2.44B Data-Tech-AI revenue, $2.64B Digital Operations revenue.

19. Concentrix 2026 Annual Report; iX Hello and iX Hero positioned as autonomous CX offerings.

There was Sam Altman. There was Dario Amodei. There was Prime Minister Narendra Modi. There were the usual smiles, the ceremonial photo-op choreography, the forced warmth of people who spend most of their lives competing for money, talent, chips, attention, and moral authority.

Then came the moment.

Everyone was supposed to hold hands.

Sam and Dario did not.

Maybe it was nothing. Maybe it was stage confusion. Altman later said he did not know what was happening. Fair enough. These things happen. But the internet does not run on fairness; it runs on symbolism. And this one was too perfect.

Because Altman and Amodei are not just two CEOs who happen to run rival AI labs. They are two heirs to the same broken house.

That house is OpenAI.

Before OpenAI became the company behind ChatGPT, before it became a Microsoft-backed juggernaut, before it became the main character in every argument about artificial intelligence, it was supposed to be something stranger and nobler: a nonprofit lab built to make sure artificial general intelligence benefited humanity.

Elon Musk was there at the beginning. Sam Altman was there too. So were researchers who believed that AI was too important to be left to the usual Silicon Valley incentives.

Then reality arrived, carrying an invoice.

Frontier AI does not run on ideals. It runs on GPUs, data centers, power contracts, elite researchers, cloud partnerships, and oceans of money. The original OpenAI mission was clean. The business of making it real was not.

That is where the story starts to split.

Musk eventually left OpenAI. OpenAI later created a capped-profit structure. Microsoft came in with billions. Altman became the public face of the AI boom. ChatGPT turned OpenAI from an ambitious research lab into a consumer phenomenon. Then, almost overnight, OpenAI became the company everyone else had to define themselves against.

Including Anthropic.

Dario Amodei did not come from nowhere. He was one of OpenAI’s key research leaders. He worked on the generation of language models that made the current AI race possible. Then he left and helped build Anthropic, along with other OpenAI alumni.

That detail matters. Anthropic is not just OpenAI’s competitor. It is OpenAI’s counterargument.

OpenAI says: scale the technology, ship useful products, build fast enough to lead the world instead of watching someone else do it.

Anthropic says: slow down just enough to not lose the plot.

Of course, that distinction gets messier in practice. Anthropic is not a monastery. It has raised huge sums, cut giant cloud deals, sells to enterprises, and competes aggressively. But its brand is different. OpenAI is the rocket launch. Anthropic is the seatbelt. OpenAI has Sam Altman doing the world-leader circuit. Anthropic has Dario Amodei sounding like the guy at the table who actually read the risk memo.

Then Elon re-enters the frame.

Musk has spent years arguing that OpenAI betrayed its founding mission. He has sued. He has posted. He has built xAI. He has made it clear that he sees OpenAI, and Altman in particular, as central characters in a story of idealism curdled into corporate capture.

Now SpaceX is partnering with Anthropic.

On paper, it is a compute deal. Anthropic gets access to SpaceX’s Colossus 1 data center capacity: hundreds of megawatts and a mountain of NVIDIA GPUs. That matters because the AI race is no longer just about who has the smartest researchers or the cleanest product demo. It is about who can get enough electricity, chips, cooling, land, capital, and political permission to keep training bigger systems.

But come on. Nobody is going to read this as just a compute deal.

SpaceX is Elon Musk’s empire. Anthropic is OpenAI’s most credible philosophical defector. OpenAI is Sam Altman’s kingdom. Put those pieces together and you get the kind of headline Silicon Valley pretends not to enjoy but absolutely lives for.

Elon supplying compute to Anthropic does not prove he is personally trying to crown Dario as the anti-Sam. Business is rarely that simple. But it does put Musk’s infrastructure behind one of OpenAI’s most dangerous rivals. And in this industry, infrastructure is not plumbing. It is power.

That is the part people still underestimate.

The AI war used to look like a software war. Better chatbot. Better coding model. Better reasoning. Better benchmark. Better demo.

Now it looks much more industrial.

Who has the chips?
Who has the power?
Who has the data centers?
Who has the cloud contracts?
Who has the government relationships?
Who can afford to lose billions now to own the platform later?

This is why the SpaceX-Anthropic deal hits differently. It suggests the frontier AI race is becoming vertically integrated, geopolitically sensitive, and brutally physical. The future of intelligence is being negotiated through substations and server racks.

And hovering above all of it is the old OpenAI rupture.

Musk helped start OpenAI, then turned against what it became.
Amodei helped build OpenAI’s research engine, then left to build a safer rival.
Altman turned OpenAI into the most famous AI company on Earth, then survived a boardroom coup that made the company look both powerful and unstable.

That coup still hangs over everything.

In November 2023, OpenAI’s board fired Altman, saying he had not been consistently candid with them. The move detonated instantly. Employees revolted. Microsoft maneuvered. Investors panicked. Within days, Altman was back. The board was reshaped. The king returned to the castle.

But the myth changed.

Before that weekend, OpenAI could still present itself as the careful steward of transformative technology. After it, the world saw something more human and more chaotic: ego, governance failure, fear, loyalty, money, mission, and power all colliding in public.

And during that chaos, Anthropic was reportedly approached as a possible escape hatch. Dario Amodei was floated as a potential replacement. A merger was discussed. He declined.

Imagine that for a second.

The company founded by OpenAI defectors was briefly considered as the solution to OpenAI’s own crisis. That is not a footnote. That is the family drama revealing itself.

Which brings us back to the photo-op.

Sam and Dario not holding hands would be meaningless in almost any other industry. In this one, it became an accidental poster for the current order: polite smiles, shared stages, private distrust.

Altman, meanwhile, has been working hard to retake the narrative. He is more visible. More personal. More active on X. More willing to present himself not just as the CEO of OpenAI, but as a person trying to explain himself in real time.

That is not random. Public trust is now part of the product.

When your company is building systems that people believe could change labor, education, warfare, media, politics, and the basic economics of knowledge, you do not just need users. You need legitimacy. You need people to believe you are not reckless, captured, delusional, or hiding something.

OpenAI knows this. Anthropic knows this. Musk knows this better than anyone.

That is why each character has his own gospel.

Sam Altman’s gospel is scale with responsibility. Build the thing, deploy the thing, learn from the world, and stay ahead because someone is going to build it anyway.

Dario Amodei’s gospel is controlled acceleration. Build powerful systems, yes, but admit more openly that nobody fully understands where the edge is.

Elon Musk’s gospel is betrayal and correction. OpenAI lost the plot, so he will build or back the alternatives that force the race back onto his terms.

All three stories are convenient. All three contain truth. All three are also attached to enormous commercial interests.

That is what makes this moment so fascinating. The AI leaders are not just selling models. They are selling moral positioning.

OpenAI sells usefulness at planetary scale.
Anthropic sells seriousness.
xAI sells anti-establishment speed.
SpaceX now sells the raw horsepower underneath the whole contest.

The industry likes to talk about alignment, but the people are not aligned at all.

They came from the same origin story and scattered into rival camps. They argue about safety while chasing customers. They warn about concentration of power while building companies that require unprecedented concentration of power. They speak the language of humanity while negotiating the future through private capital and closed infrastructure deals.

So yes, the SpaceX-Anthropic partnership is about compute.

But it is also about the breakup of OpenAI’s founding myth.

The dream was that artificial intelligence could be guided by a unified mission: build it safely, share the benefits, avoid capture by narrow interests.

The reality is messier and more interesting. The people who once stood near the same mission now compete to define what that mission was supposed to mean.

Elon has his version.
Sam has his.
Dario has his.

And somewhere between the lawsuits, the data centers, the summit stages, the X posts, and the non-hand-holding, the AI race has become what Silicon Valley always becomes when the stakes get big enough:

a moral argument with a cap table.

That is the real backstory.

The cold war in AI is not coming.

It is already here.

And this time, it runs on GPUs.

Subscribe now

Thesis: the next important AI services companies will not just sell software seats or staff augmentation. They will sell trusted execution: agents for work, humans for judgment, software for repeatability, and pricing tied increasingly to outcomes.

There is a useful what-if in Indian technology history. Back in 2015, Infosys’ CEO Vishal Sikka was close enough to the original OpenAI story, Sikka was publicly listed as an adviser. Years later, Sikka said Infosys had explored a meaningful OpenAI investment but did not go ahead. The counterfactual is irresistible: what if a global IT services company had owned a piece of the AI platform layer before AI became the defining enterprise technology wave?
Every enterprise technology cycle creates two markets. One is visible: software platforms, licenses, usage, seats, dashboards. The other is execution: implementation, migration, integration, support, managed operations, compliance, change management, and the messy human work required to make technology function inside a real enterprise.
Cloud had both. SaaS had both. AI will have both. The difference is that, this time, the service provider may not need to be organized as a pyramid of people.

Services are not bought like software

Most AI services commentary starts with the wrong mental model. It asks whether agents can replace people. That is directionally right but commercially incomplete. Enterprises do not buy services the way they buy tools. A tool improves productivity. A service absorbs responsibility. A tool still leaves work inside the customer organization. A service promises that the work will get done.

That is why the winning AI services companies will not be thin chatbot wrappers. The job is not merely to answer a question. The job is to understand context, follow policy, execute a workflow, escalate the exception, document the action, and measure the outcome.

The best early openings are the workflows that already have a budget, repeat often, and can be measured.

That is where an agent starts to look less like software and more like a service.

The market map: six collision zones

AI services is not one category. It is a collision between traditional IT services, SaaS workflow platforms, AI-native agent companies, vertical professional services automation, software engineering agents, and the infrastructure used to make AI reliable. The funding signal is strongest where agentic execution can attack an existing services or labor budget.

Funding is moving toward labor budgets

The public signals are no longer theoretical: strategic M&A in internal operations, mega-rounds in CX and legal AI, and billions of AI bookings at services incumbents all point to the same shift.

Where new AI services companies can break in first

Not every services workflow is a good startup wedge. Low-value work becomes a feature inside existing software. Highly bespoke work is too hard to standardize. The best early wedges sit in the middle: painful, frequent, measurable, and structured enough for automation.

This also explains why many early winners will look vertical before they look horizontal. Legal AI, customer support AI, employee service AI, and coding agents each start with a bounded domain where the workflow, data, buyer, tooling, and ROI are legible.

Subscribe now

What separates a chatbot from an AI-native service

The strongest companies will not sell agent access. They will sell a delivery system. The pattern is already visible: connect to enterprise systems, normalize context, encode policy, execute with agents and deterministic workflows, escalate exceptions to humans, and report outcome metrics.

The real fight: who owns the workflow?

There are three plausible company-building paths.

1. Direct AI services company Sell the outcome directly to enterprises. Highest value capture, but the trust barrier is highest.

2. Channel through incumbents Sell agentic delivery infrastructure to SIs, MSPs, BPOs, and consulting firms. Faster distribution, but less customer ownership.

3. Embedded in systems of record Build around Salesforce, ServiceNow, SAP, Workday, Atlassian, or Microsoft. Lower friction, but higher platform dependency risk.

A hybrid path may be the most attractive. Own the first workflow directly. Prove the ROI. Turn repeated delivery into software and data. Then distribute through channels or systems of record after the automation pattern is proven.

How to diligence AI services companies

My initial mistake was to diligence the demo instead of the service. A clean demo proves that an agent can complete a happy-path task. It does not prove that the company can own messy enterprise work, absorb exceptions, protect the customer, and improve margins as volume scales.

The diligence should start with one blunt question: what existing service budget is this replacing or expanding? If the answer is only “software productivity,” the company is probably a SaaS tool. If the answer is headcount, outsourced delivery, support capacity, implementation effort, or managed operations, the opportunity starts to look like AI services.

The test is simple: if the company disappeared tomorrow, would the customer replace a tool, a headcount plan, or an outsourced service provider? The closer the answer is to service capacity, the more interesting the company becomes.

Subscribe now

Why the category exists now

For decades, identity in the enterprise has been an administrative concern: provision the user, assign the role, rotate the password, run the quarterly access review. AI agents break that entire model.

Traditional software follows a path its developer can describe before a single line ships. The developer knows which APIs the service will call, provisions a service account with the right scopes, and relies on stable role assignments to keep things running. The identity layer manages who and whether, but it does not need to reason about what a piece of software will decide to do at the moment of execution.

Agents change this in a fundamental way. They discover tools dynamically. They select actions at runtime based on context, prompts, and intermediate reasoning. They chain calls across multiple systems. They take high-impact actions with minimal human oversight. And they operate at machine speed, which means the window between “access granted” and “damage done” can collapse to milliseconds.

This is why agent identity is not just “another AI security category.” The underlying control problem is identity-centric. Every meaningful enterprise deployment of an AI agent eventually collides with the same questions: Under what identity does this agent act? With what scope? For how long? Under whose policies? And who is accountable when something goes wrong?

The identity question at the center of every agent deployment: Which principal is the agent acting as? What is it authorized to do right now, at this moment, in this context? Can we revoke that authority in real time? And does the audit trail make it clear who approved what?

Those questions land squarely in the territory of IAM, IGA, and PAM. But they land in a way those systems were never designed to handle. The result is a gap: existing identity products remain foundational, but they cannot, by themselves, govern the behavior of non-deterministic software principals that reason, select tools, and execute at machine speed.

Why IAM, IGA, and PAM are necessary but insufficient

It would be a mistake to dismiss the existing identity stack. IAM, IGA, and PAM are not going away. They are the foundation that any agent identity solution must integrate with. But each was designed for a different era, and each breaks in a specific way when agents enter the picture.

IAM: The system of record

IAM handles authentication, federation, and base policy. It tells the enterprise who a principal is and whether it should be allowed to start a session. But classic IAM reasons in terms of users, applications, sessions, groups, and coarse scopes. It was not designed to understand the intent of a single agent action or to continuously narrow privilege on a per-tool-call basis.

IGA: The compliance layer

IGA answers the governance questions: Who should have access? Who approved it? Who must certify it? Is the lifecycle in policy? For agents, IGA is useful for ownership, registration, certification, and the discovery of shadow AI. But IGA operates on the lifecycle plane, not the execution plane.

PAM: The closest neighbor

PAM is the most relevant legacy category. It already understands standing privilege as a problem and has decades of practice in just-in-time elevation, approval workflows, credential vaulting, and session recording. But most PAM products were built for human operators and deterministic workloads. AI agents need privilege to be narrowed and re-evaluated continuously at runtime, often at the level of a single action rather than an entire session.

The six-layer framework

The market becomes much clearer once you separate it by control point. The first three layers are foundational. The next three are where the category is being rewritten.

For investors, the most important distinction is between companies that see risk and companies that block risk. Both create value. But the companies in the runtime control path tend to have stronger lock-in, higher switching costs, and more defensible pricing.

Representative market map

The tables below are not exhaustive. They capture the companies most relevant to how we think about the category today.

Capital formation and market signals

Funding alone will not determine winners. But the pattern of capital formation reveals which sub-segments investors and acquirers believe matter.

Key M&A milestones

2024

Astrix closes $85M. Permiso raises $28.5M. Entro raises $18M Series A. Category formation accelerates.

Early 2025

Token Security raises $27M. Aembit announces IAM for Agentic AI. Keycard launches with $38M.

Late 2025

Oasis Security raises $195M. Descope launches Agentic Identity Hub. MCP auth fabric emerges as a sub-category.

Early 2026

CrowdStrike acquires SGNL. Palo Alto completes CyberArk acquisition. ServiceNow acquires Veza. Microsoft ships Entra Agent ID. The category is being pulled into major platforms.

The M&A activity in early 2026 is particularly telling. When CrowdStrike, Palo Alto Networks, and ServiceNow all make identity-related acquisitions within weeks of each other, it confirms that incumbents view agent identity as must-have, not niche.

What this means for investors

This category matters because it sits in the control plane of enterprise AI adoption. As agents move from read-only copilots to systems that can take actions across production environments, identity becomes the gating layer that determines whether enterprises can deploy them safely at scale.

Risks and open questions

No investment thesis is complete without a clear-eyed accounting of what could go wrong. Four risks deserve particular attention.

Incumbent bundling. When Microsoft ships Entra Agent ID, when CrowdStrike acquires SGNL, and when ServiceNow acquires Veza, it becomes harder for standalone startups to maintain pricing power. The counter-argument is that incumbents solve for their own estate first and leave multi-cloud edge cases to startups. But the risk is real.

Category confusion. Multiple vendors describe adjacent capabilities using identical language. “AI identity” can mean runtime authorization, NHI inventory, lifecycle governance, or just a marketing slide bolted onto an existing product.

Protocol volatility. MCP is gaining traction, but the standards around OAuth for MCP and client registration are still evolving. Companies that bet on a single protocol face re-architecture risk.

Adoption timing. It is easy to overestimate how fast fully autonomous agents will reach production in heavily regulated environments. The category can still be attractive if the near-term wedge is copilots and supervised workflows, but it affects go-to-market and the patience capital needs to have.

An additional question: Where does the buying center land? Some products sell to IAM teams, some to security operations, some to platform engineering, some to developers. Unclear buying center creates friction for startups and makes repeatable GTM harder.

Bottom line

Agent identity is emerging as one of the most important control planes of the agentic stack. But the right mental model is not “new category replaces IAM.” The better model: agents are forcing IAM, IGA, PAM, identity security, and AI-native authorization to converge around runtime access control.

For investors, the opportunity is to identify the companies that become the indispensable runtime and privilege layer for agentic systems. For operators, the challenge is to separate identity registration, governance, privilege brokering, and action enforcement rather than treating them as one undifferentiated market.

The category is early, but it is no longer theoretical. Product launches, funding rounds, and strategic acquisitions across 2024 to 2026 already show the market moving from experimentation to platform formation. The question is no longer whether agent identity matters. The question is which companies will own the control path.

Glossary

IAM (Identity and Access Management): System of record for authentication, federation, and base policy. IGA (Identity Governance and Administration): Ownership, approvals, certification, and lifecycle compliance. PAM (Privileged Access Management): Just-in-time privilege, credential brokering, and session controls. ITDR (Identity Threat Detection and Response): Monitoring and defending against identity-based threats. NHI (Non-Human Identity): Any machine principal, including service accounts, API keys, bots, and AI agents. MCP (Model Context Protocol): An emerging standard for how AI agents discover and invoke external tools. JIT/JEA: Just-in-time and just-enough access. AuthZ: Authorization, the decision about whether a principal is allowed to perform a specific action.

Enterprise AI moved from pilot to production faster than most security programs could absorb. The pattern is now familiar: a business unit rolls out a chatbot for employees, then a customer-facing assistant, then an internal “agent” wired into ticketing, CRM, finance approvals, and code repos. Capabilities arrive as APIs. Permissions get bolted on later.

Risk is rising at the same pace. The World Economic Forum’s Global Cybersecurity Outlook 2025 captured the gap bluntly: 66% of organizations expected AI to have the biggest impact on cybersecurity, but only 37% reported having processes to assess AI tool security before deployment. World Economic Forum

In that gap sits a comforting misconception: guardrails will take care of it.

In most enterprises, “AI guardrails” means some mix of:

• a system prompt that tells the model what it must refuse

• a moderation filter for inputs and outputs

• a few regexes to strip secrets

• a policy that blocks certain topics

• a vendor promise that the model is “hardened”

Those controls matter. They also get mistaken for a security boundary. In 2026, that mistake is getting more expensive—because AI systems are no longer “just generating text.” They browse, call tools, create tickets, pull documents, and execute workflows. Microsoft’s own 2026 trend piece frames agents as workplace participants and argues they need “similar security protections as humans,” warning about “double agents” carrying unchecked risk. Source

Once the model can take actions, “guardrails” stop being a safety feature and start being a false sense of containment.

Why guardrails fail in practice

Guardrails fail for the same reason most “string-based security” fails: they try to impose deterministic rules on a system that was never designed to treat text as a privileged control plane.

1) Models don’t separate instructions from data

SQL injection was mitigated at scale by separating code from data: parameterized queries, prepared statements, strong parsing boundaries. Large language models don’t have an equivalent boundary. The UK’s NCSC put it plainly: “As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated…” Computer Weekly

That’s not rhetorical pessimism. It’s a property of how LLM applications are built: the prompt is a single sequence that often mixes system rules, user intent, retrieved documents, tool outputs, and untrusted web content.

When enterprises treat the prompt as a secure channel, attackers treat it as an input surface.

2) Prompt injection bypasses “refuse” logic by attacking the workflow, not the model

A lot of guardrails focus on content policy (“don’t output X”). Prompt injection is often about behavior policy (“do Y instead of what the user asked”).

OpenAI’s December 2025 write-up on hardening “ChatGPT Atlas” (a browser agent) describes prompt injection as a long-term issue for agent security and states: “Prompt injection… is unlikely to ever be fully ‘solved’.” OpenAI
That post also shows why: when an agent reads an email, a doc, or a webpage, malicious instructions can ride along inside content the agent is expected to ingest. In OpenAI’s demo, an injected instruction causes the agent to send an unintended resignation email. OpenAI

A refusal policy doesn’t help if the attack is “continue the task, but quietly route the output to my endpoint” or “add this hidden step in the middle of the workflow.”

3) Indirect prompt injection turns everyday content into an exploit carrier

Direct prompt injection is the user typing an attack prompt. Indirect prompt injection is worse for enterprises because it weaponizes normal inputs: a helpdesk ticket, a vendor PDF, an internal wiki page, a Jira description, a marketing landing page.

Brave demonstrated this with an “agentic browser” scenario: an attacker hides instructions in third-party content; the assistant summarizes the page; the hidden instructions attempt to drive navigation and exfiltration inside the user’s authenticated session. Brave

Google’s Chrome security team calls indirect prompt injection “the primary new threat facing all agentic browsers,” explicitly linking it to outcomes like financial transactions and data exfiltration. Google Online Security Blog

This is a core reason guardrails fall short: the same model that must read untrusted content is asked to make privileged decisions based on it.

4) Agents amplify blast radius through tool access and privilege

Traditional chatbots were mostly a reputational risk. Agents can be an operational risk.

The OWASP Top 10 for LLM Applications puts “Prompt Injection” (LLM01) and “Excessive Agency” (LLM08) at the top of the list because crafted inputs can lead to unauthorized access and because unchecked autonomy leads to unintended consequences. OWASP Foundation

A concrete example came out of research on ServiceNow’s Now Assist: a low-privileged agent can generate an internal task that a higher-privileged agent executes, resulting in data exfiltration—what the reporting calls “second-order prompt injection.” AppOmni’s Aaron Costello described the risk in operational terms: “it isn’t a bug in the AI; it’s expected behavior” under certain defaults. TechRadar

Guardrails that screen user prompts won’t stop an agent-to-agent escalation path if tool permissions and workflow checks are weak.

5) “Blocklists and slogans” are easy to route around

Enterprises still deploy phrase-based filters (“ignore previous instructions”), deny-lists, and brittle classifiers. Attackers don’t need a novel exploit; they need paraphrases, multilingual variants, obfuscation, or simply moving the payload into retrieved context.

The NCSC explicitly warns against simplistic approaches like deny-listing phrases and argues instead for deterministic safeguards that constrain actions, plus least-privilege principles applied to LLM systems. Computer Weekly

This is where the guardrail myth becomes dangerous: it encourages spending on controls that look like governance while leaving the real control plane—identity, privileges, execution, and monitoring—underbuilt.

Guardrails aren’t useless. They’re just not a boundary.

Enterprises should keep “guardrails” in their toolbox:

• They reduce accidental policy violations.

• They lower the frequency of obvious misuse.

• They help with UX, safety messaging, and coarse risk reduction.

What they don’t do is provide reliable containment against a motivated adversary with time to iterate. Treating them as a boundary is like treating a WAF as a substitute for authentication, authorization, patching, and monitoring.

The better mental model: guardrails are friction; security is architecture and operations.

Red teaming, in AI terms

“Red teaming” started as a military term for an adversarial group tasked with breaking assumptions. In cybersecurity, it evolved into structured offensive testing: simulate real attackers, chain vulnerabilities, prove impact, then feed fixes back into engineering.

In AI, red teaming has expanded in two directions:

1. Model behavior testing

   • Can the model be induced to reveal sensitive data?

   • Can it be manipulated to violate policy?

   • Can it be coaxed into unsafe tool selection?

2. System-level adversarial testing

   • Can an attacker drive the end-to-end product into an unsafe state?

   • Can they exploit retrieval, plugins, connectors, permissions, and workflow logic?

This second form is where most enterprise risk sits, because it maps to business outcomes: data loss, fraud, unauthorized access, destructive actions, and compliance failures.

The leading labs now treat red teaming as continuous work, not a launch checklist. OpenAI describes “automated red teaming” using reinforcement learning to discover prompt injection attacks against agents and then harden both the model and surrounding safeguards. OpenAI
Google describes automated red teaming for agentic Chrome features, measuring attack success rates to prevent regressions as engineering changes land. Google Online Security Blog

Government frameworks are converging on the same idea: testing needs to be systematic, repeatable, and tied to validation. CISA has been pushing TEVV (test, evaluation, verification, validation) concepts into AI evaluations; one summary of its guidance emphasizes borrowing proven methods rather than inventing a new discipline from scratch.
NIST’s AI RMF also embeds TEVV into its lifecycle framing and highlights that companion playbook guidance evolves based on community feedback. NIST Publications

Red teaming matters because it replaces arguments with evidence. It produces exploit chains, not theoretical risk registers. It also forces the uncomfortable question that guardrails try to avoid: what happens when the model fails anyway?

AI security in 2026: what’s getting harder

If you’re building or funding enterprise AI products in 2026, the security landscape has a few defining traits.

Prompt injection is no longer a niche problem

It’s now a default threat model for any product that:

• reads external text (web, email, docs, tickets)

• uses retrieval (RAG)

• calls tools

• acts across accounts or systems

The NCSC’s position that prompt injection may never be “totally mitigated” like SQL injection is a warning about the limits of model-centric defenses. Computer Weekly
OpenAI’s stance that prompt injection is unlikely to be fully solved echoes the same reality from the platform side. OpenAI

Jailbreaking is still a problem, but it’s not the enterprise-killer

Jailbreaking focuses on getting disallowed content out of a model. That matters for consumer misuse, brand risk, and regulated domains.

Prompt injection focuses on steering behavior inside a system. That’s what turns an assistant into an attacker’s workflow engine.

Sander Schulhoff’s December 21, 2025 appearance on Lenny’s Podcast—summarized in the episode notes—frames this as a key distinction and argues that commercial “guardrails” are being oversold relative to what attackers can do, especially as agents get more capable.

For enterprise products, that’s the hinge: content moderation issues are visible; workflow hijacks can be silent.

Data poisoning and AI supply chain risk are moving up the agenda

Enterprises are assembling models and products out of:

• foundation models and fine-tunes

• vendor tools and plugins

• open-source libraries

• datasets and synthetic data pipelines

• vector databases and embedding services

OWASP explicitly calls out “Training Data Poisoning” (LLM03) and “Supply Chain Vulnerabilities” (LLM05). OWASP Foundation
WEF’s 2025 outlook also flags supply chain vulnerabilities as a top ecosystem risk, citing visibility and oversight gaps across suppliers. World Economic Forum

This matters for AI products because poisoning doesn’t need to compromise your core model training pipeline to hurt you. It can enter through retrieval corpora, “approved” documentation, or third-party connectors.

Agentic AI shifts security from “outputs” to “actions”

Microsoft’s 2026 trend piece is unusually specific for a corporate forecast: give agents identities, limit what they can access, manage the data they create, and protect them from attackers. Source
Google’s Chrome architecture work similarly treats agent actions as high-risk operations that require gating, isolation, and confirmations. Google Online Security Blog

The direction is clear: as soon as AI can take actions, enterprises need the same controls they use for human operators—plus extra controls for the ways models fail.

What does work: building secure AI products like secure distributed systems

Enterprises don’t need a miracle defense. They need to stop treating the model as the security boundary and start treating it as an untrusted component inside a security-designed system.

1) Make the control plane deterministic

A practical rule: the model proposes; the system disposes.

Concrete patterns:

• Tool allowlists and schemas. Tools should be explicit, typed, and validated. Free-form “call anything” is how prompt injection becomes RCE-by-proxy.

• Policy-as-code checks on every tool call. Enforce tenant boundaries, data classification, and intent constraints outside the model.

• Constrained output formats. If the model outputs JSON that drives automation, validate it strictly and reject anything off-schema (OWASP’s “Insecure Output Handling” is still a common failure mode). OWASP Foundation

Google’s “User Alignment Critic” is a concrete example of splitting the system: the planning model is exposed to untrusted web content; a separate model vets proposed actions using structured metadata instead of raw page text. Google Online Security Blog

Whether you use a second model or deterministic code, the point is the same: don’t let untrusted text directly shape privileged actions.

2) Put hard boundaries around where agents can operate

Agentic systems need something like network segmentation, but for information flow.

Google describes extending origin isolation with “Agent Origin Sets,” limiting what origins the agent can read and what origins it can write to, enforced by gating functions not exposed to untrusted content. Google Online Security Blog
This is the right direction for enterprise apps too:

• restrict which internal systems an agent can touch per task

• restrict which document collections a retrieval step can access

• prevent cross-domain data flows by default

Brave’s research shows why: indirect prompt injection often aims to exploit the user’s authenticated context. If your agent can wander freely, your users’ existing sessions become the prize. Brave

3) Apply least privilege to models and agents—literally

“Least privilege” needs to move from policy slides into implementation details:

• short-lived credentials for tool access

• per-agent service principals

• per-action authorization checks

• no shared “super-agent” identity

• explicit separation of read vs write capabilities

The NCSC calls this out directly: LLMs should not have more privileges than the parties interacting with them. Computer Weekly
Microsoft’s 2026 trends piece makes the same point with a human analogy: agents need identity and constrained access to avoid becoming “double agents.” Source

4) Engineer for failure: confirmations, holds, and reversibility

You can’t “train away” every failure mode. OpenAI’s Atlas post explicitly frames deterministic guarantees as hard, and leans on layered defenses plus rapid response cycles. OpenAI

Enterprises should borrow a concept from payments and infrastructure: two-phase commit for high-impact actions.

• Phase 1: the agent prepares a proposed action with a human-readable diff (“send $X to vendor Y; attach invoice Z; email approvals to A and B”).

• Phase 2: a user or policy engine approves execution, or the system routes to an alternate workflow.

Google’s Chrome design uses confirmations for sensitive steps like purchases and sign-ins. Google Online Security Blog
This is not UX decoration. It’s a containment mechanism.

5) Treat retrieval as an attack surface, not a feature

If you use RAG, you’re ingesting untrusted text and giving it influence over outputs.

Controls that hold up better than “prompt telling”:

• provenance tracking for retrieved chunks (where did this come from, when, who approved it)

• retrieval allowlists by source type and sensitivity

• document-level malware-style scanning for injection patterns in high-risk corpora

• “quoted context” UX (show the user what sources influenced the answer)

• aggressive separation between “facts the model may cite” and “instructions the model may follow”

This is where many enterprise assistants fail quietly: the model is “following policy,” but the system fed it poisoned context.

6) Observability: log the right things, at the right layer

Most enterprises already know how to monitor applications. AI adds new signals.

Minimum viable AI security telemetry:

• full prompt assembly with structure (system / user / tool / retrieved context), redacted for secrets where required

• tool call logs (arguments, success/failure, latency, downstream resource identifiers)

• authorization decisions (why a tool call was allowed or blocked)

• origin / tenant / data classification tags for every retrieval and action

• user overrides and confirmation events

The NCSC explicitly recommends extensive logging—up to full inputs and outputs and any tool/API calls—and live monitoring for failed tool calls as a sign of attack iteration. Computer Weekly

Detection content that pays off quickly:

• repeated blocked tool calls (agent “probing”)

• unusual tool sequences (data export followed by external notification)

• sudden expansion of reachable sources (new origins, new connectors)

• drift in “normal” agent behavior after a prompt or model update

Pipe it into your SIEM. Create playbooks. Treat it like a new workload class.

7) TEVV and red teaming as a product discipline, not a one-off exercise

NIST’s AI RMF is explicit about being a living framework with an update schedule and frequent playbook updates; it frames governance as cross-cutting and expects organizations to map, measure, and manage risks across the lifecycle. NIST Publications+1

What that looks like in practice for an enterprise AI product:

• threat model every new capability (new tool, new connector, new memory feature)

• build an eval harness with regression tests for known attack classes (prompt injection, data leakage, unsafe tool calls)

• run continuous red teaming (human + automated where it makes sense)

• track fixes like vulnerabilities: severity, exploitability, exposure, patch status

• treat model updates as releases that require security sign-off

Google explicitly measures attack success rates to prevent regressions as features change. Google Online Security Blog
OpenAI describes a rapid response loop that uses attack discovery to drive adversarial training and improvements in the broader defense stack. OpenAI

That’s the direction enterprise teams should copy: security as a loop, not a checklist.

Tools and solutions that help more than “guardrails”

Enterprises are buying an emerging AI security stack. Some of it is real engineering. Some of it is content filtering in a trench coat. Tool selection should reward systems that can be tested, monitored, and reasoned about.

Categories that tend to create durable security gains:

AI gateways and policy enforcement layers

Look for:

• per-request policy evaluation (identity, tenant, data classification)

• tool call mediation and allowlisting

• strict schema validation for structured outputs

• centralized logging and trace correlation

Adversarial testing and evaluation platforms

Look for:

• reproducible test suites for prompt injection and tool abuse

• regression tracking across model versions and prompt changes

• support for multi-step agent scenarios (not just single-turn chat)

Agent runtime sandboxes

Look for:

• isolation boundaries (network, filesystem, secrets)

• deterministic controls over what can be executed

• explicit permissions for each capability (browsing, emailing, ticket creation, exports)

Data security controls integrated into AI flows

Look for:

• DLP hooks at ingestion and egress

• redaction/tokenization before sending data to models

• retrieval access control tied to IAM, not “prompt instructions”

Supply chain and provenance tooling for AI assets

Look for:

• versioning and signing for prompts, policies, and retrieval corpora

• audit trails for dataset and document updates

• dependency scanning for model plugins and connectors

What to be skeptical of:

• vendors claiming they “stop prompt injection” outright (the NCSC explicitly cautions against this framing). Computer Weekly

• tools that can’t show you what they blocked, why, and how you can test it

• “secret sauce” classifiers with no measurable regression tracking

Guardrails can be part of a product. They just can’t be the product.

Where this leaves enterprise leaders and investors

The most useful question in AI security isn’t “does it have guardrails?” It’s:

• What can this system do, under what identity, with what privileges?

• What happens when an attacker gets text into its context?

• Can we observe misuse in time to stop it?

• Can we prove fixes stick across releases?

The security path forward looks less like content moderation and more like the last 20 years of secure systems engineering: isolation, least privilege, policy enforcement, testing, telemetry, incident response. The model sits inside that system. It is not the system.

If you accept, as both OpenAI and the NCSC do in different words, that prompt injection won’t be “solved” in a neat, final way, then the enterprise job becomes clear: reduce the probability of compromise and shrink the blast radius when it happens.

That’s how secure AI products get built—without pretending a few lines of prompt text can do the work of architecture.

Coding is where this shows up first for boring reasons. The work already lives in text. The feedback loop is tight. You can run tests, see a red stack trace, and ask the tool to fix it. You can diff the change. You can roll it back.

That combination creates a clean scoreboard: did it compile, did it pass CI, did the PR merge, did prod stay up.

Who’s “winning” depends on which layer you mean

People talk about “the coding tool market” like it’s one thing. It isn’t. There are at least three distinct layers, and different players can lead each.

1) Default distribution inside existing workflows (Copilot)

Microsoft said 15 million developers were already using GitHub Copilot as of May 19, 2025.

GitHub’s Octoverse (Oct 28, 2025) said nearly 80% of new developers on GitHub use Copilot within their first week.

That’s not “best model” talk. That’s default behavior.

2) The new “agentic editor” surface (Cursor)

Cursor’s own Series C post (Jun 6, 2025) said it had grown to >$500M ARR and is used by over half of the Fortune 500.

TechCrunch also reported (Jun 5, 2025) that Cursor’s ARR had passed $500M and was accelerating quickly.

Cursor is a good reminder that “surface” can beat “distribution” when the workflow feels faster.

3) Model-native, terminal-first delegation (Claude Code)

TechCrunch reported (Oct 20, 2025) that Claude Code grew 10× in users since its broader launch in May 2025, and that it accounts for >$500M annualized revenue for Anthropic.

So your instinct (“Anthropic is winning in coding”) is credible in one slice: model-native, agentic coding. It’s just not the only slice.

Why the gap is so wide: the product is the loop, not the model

Most tools can write a function. The winners feel like they’re running a mini software team:

• Read the repo.

• Make a plan you can skim.

• Change files with a diff you can review.

• Run unit tests or lint.

• Fix what breaks.

• Open a PR.

That loop matters more than clever prompts.

If a tool can’t close the loop, it becomes a toy. You get a nice snippet. Then you spend 30 minutes wiring imports, fixing types, and chasing edge cases. People quietly stop using it.

The tools that compound do three things well

1) They reduce “time to first merged PR”

Not “time to first output.” Time to first result that sticks.

A practical example: updating a payments SDK across 30 services. The real work isn’t writing code. It’s hunting for call sites, adjusting tests, updating docs, and making sure CI passes. The best tools treat that as one job. The weak ones give you a patch and walk away.

2) They make context cheap and safe

Context isn’t just “long prompts.” It’s access to the right things at the right time: repo, build logs, error output, package manager quirks, internal APIs, sometimes a ticket in Linear/Jira.

This is where “agent” work starts looking like systems design:

• permissions (what can it read/write/run?)

• sandboxes (where does it execute?)

• audit (what did it touch?)

OpenAI’s Codex pitch is explicit about this: a cloud-based agent where each task runs in its own cloud sandbox preloaded with your repo, and it can propose PRs for review.

3) They ship a review-first interface

The most useful UI in coding is still the diff. If your tool makes diffs easy to inspect and revert, people push it into real work. If it hides the diff behind chat, it stays in prototype land.

Cursor’s own CEO has talked about the risk of leaning on AI without understanding the code, calling out “shaky foundations.”

That warning doesn’t kill vibe coding. It tells you what the product needs: diff-first, tests by default, and a clean escape hatch.

The survival traits you can actually underwrite

A bunch of “vibe coding” products will not last. The ones that do tend to have at least one of these moats:

A) Own a surface

If you live where developers spend their day, you get habit and retention. Copilot has this through IDE integration and GitHub workflow gravity.

Cursor has it through a full editor experience that people adopt as a daily driver.

If you don’t own a surface, you are one browser tab away from churn.

B) Own a workflow that feels like a “unit of work”

The market is moving from “assist me” to “take the ticket.” That means the product boundary shifts. The user buys outcomes: upgrade dependency, fix flaky test, refactor module, add endpoint, write migration, update docs, open PR.

Codex is leaning into this with “many tasks in parallel” and a packaging system for repeatable workflows.

C) Own trust and governance

As agents get more permissions, the threat model changes. The tool is now reading untrusted text (issues, PR comments, docs) and taking actions. Prompt injection stops being a demo trick and starts being a real operational risk.

OpenAI’s own Atlas hardening post says prompt injection is “unlikely to ever be fully ‘solved’,” comparing it to scams and social engineering on the web.

Coding agents aren’t browsers, but the same pattern shows up whenever an agent consumes untrusted text and has tools. Enterprises will pay for guardrails if you make them real (sandboxing, policy, audit logs, least privilege), not marketing.

D) Control your unit economics

There’s a hard business constraint here: a delegative tool can drive usage way past “per seat.” Every retry, test run, and refactor is tokens and inference cost.

Reuters flagged this directly in June 2025: many coding startups rely on third-party foundation models, which pushes operating costs high and can produce negative gross margins.

That doesn’t mean they’re doomed. It means margins become a first-order product choice: how often you run tools, how you cache context, when you summarize, when you stop.

What “vibe coding” teaches about AI products outside coding

Coding is just the cleanest lab. The lessons generalize when the domain has a tight loop and a clear notion of “done.”

Good domains look like this:

• The agent can take actions (API calls, edits, writes).

• There’s fast feedback (tests, reconciliations, alerts, approvals).

• You can review changes (diffs, ledgers, before/after states).

• You can roll back.

Think: IT ops runbooks, FinOps cleanup, security triage, data pipeline fixes. Anything where the job is a sequence of actions with logs.

Bad domains are the ones where success is fuzzy and the loop is slow. The tool can talk forever, but nobody trusts it to ship.

The coding winners aren’t magic. They picked a domain where the loop can.

Why this matters now

Cloud did not win because it was cheaper. Cloud won because it pushed teams to rethink networks, services, identity, security, and observability from first principles. AI is creating the same kind of pressure. The stack is changing. The loops are changing. The control plane is changing.

Think of AI as software with uncertainty built in. That means new habits. Tests evolve into evaluations. Systems return confidence, not just pass or fail. Some paths end in “I am not sure” with a safe fallback to a human. None of this works without tracing, versioning, and policy in code.

What carries over from cloud

• Abstract the messy. Treat models like services with clear contracts.

• Automate the boring. Manage prompts, policies, and datasets as code with review and rollback.

• Observe everything. Wire traces, evaluations, and cost telemetry from day one.

These habits scaled cloud. They will scale AI as well.

What is different with AI

• Probabilistic behavior. You will need golden tasks, eval suites, and confidence thresholds.

• Traceability. You must know the model, the prompt, the data slice, the tool calls, and the version.

• Many models, not one. Route by speed, cost, and quality. Sometimes a small fast model beats a big one.

• Humans in the loop. Oversight, review, and retraining are part of the product, not an afterthought.

The AI stack has a cloud base

You are not leaving cloud. You are layering AI on top.

• Kubernetes for orchestration

• Object storage for knowledge and artifacts

• Queues and event buses for tools and agents

• GPUs and accelerators inside VPCs

• Hybrid and multi cloud to reduce lock in

Cloud vs AI, at a glance

Cloud era concerns mapped to AI era equivalents and why they matter in 2025.

Subscribe now

Adoption is broad, maturity is not

Many teams report the use of AI in at least one function. Far fewer feel truly mature. That is normal for a platform shift. The pattern is simple. Data readiness and policy come first. Then workflow automation. Then broader copilots.

Common blockers:

• Messy or siloed knowledge

• Undefined access controls and governance

• Gaps in evaluation and tracing

• Skills and training for product, data, and engineering

Risks you can manage

• Quality and security debt. Code assistants can increase churn and duplication if you skip reviews and evaluations. Make evals part of the CI loop.

• Sustainability. Electricity demand from data centers is rising fast. Efficient models, right sized workloads, and clean power strategies help with cost and carbon.

• Shadow systems. If you do not provide a central model and prompt gateway, teams will build their own. Cost and safety will drift.

A concrete picture

Customer support is a good mental model.

1. Retrieve the relevant policy from your knowledge base

2. Draft a response

3. Check rules and thresholds

4. Send if confident, or escalate to a human

5. If a refund is needed, call a scoped tool with clear permissions

This pattern blends fuzzy generation with strict edges. Many companies start here and discover what should be automated, what should be tool based, and what should remain with a human.

The AI native stack

Five layers on top of cloud primitives

Layers in plain terms

• Data and Knowledge. Collect sources, protect PII, build embeddings, refresh often. Most systems answer from your content, so retrieval quality matters.

• Models. Serve multiple models. Route by latency, cost, and quality. Fine tune where it pays.

• Orchestration. Plan, call typed tools, handle retries and timeouts. Add memory where it helps.

• Experience. Copilots, chat, and APIs with streaming and citations. Give users history and controls.

• Control plane. Traces, metrics, safety, cost, and versioning. Policies as code. Strong isolation.

Where to start without boiling the ocean

• Pick one workflow with a clear KPI like CSAT, AHT, deflection, or lead to demo.

• Start with retrieval over your own content.

• Add a small number of trusted tools.

• Stand up a basic control plane early. Tracing and evals will save you weeks later.

• Form a small platform group that owns the gateway, evaluations, datasets, and safety.

• Treat prompts, datasets, and policies like code with review and rollback.

What not to do

• Do not bury rules in prompts that no one can test. Put them in code and tools.

• Do not ship demos you cannot trace, measure, or replay.

• Do not assume every step must be fuzzy. Some steps want low temperature and deterministic tools.

• Do not skip training and documentation. Talent gaps stall pilots more than technology.

Takeaway

Cloud forced a rebuild around new primitives and new ways of working. AI is doing that again. If you design around data, models, orchestration, experience, and a control plane, you get faster loops, safer systems, and better economics. If you ignore this and bolt models onto old patterns, you will get clever demos and little durable value.

Share

Leave a comment

Share

Share Rishi Maheshwari

Krishna has been in this game longer than most. He was building neural networks for Bing in the early 2000s, back when search was just keyword matching. Later, at Facebook, he ran the technical teams behind Newsfeed ranking.

That experience at Facebook is where the problem became obvious. They had built a massive "black box" to decide what millions of people saw every day, but even the engineers couldn't always explain why a specific post appeared. They had to build internal tools just to debug their own creation.

That was the seed for http://fiddler.AI.

We talked about why the current wave of AI is giving enterprise CIOs heartburn, and why "observability" for LLMs looks nothing like monitoring a database.

The end of deterministic software

We are used to software that follows rules. You write code, it executes. If A, then B. It is deterministic. When it breaks, it’s usually because of a logic error or a server timing out.

AI is different. It is probabilistic.

You can give an LLM the same prompt twice and get two different answers. One might be helpful; the other might be a hallucination or a security leak.

Krishna argues that you can’t monitor this with traditional tools like Datadog or Splunk. Those tools watch for latency and CPU spikes. They don’t know if your chatbot just promised a customer a 90% discount that doesn't exist.

FOMO vs. FOMU

Every enterprise leader right now is stuck in a vice.

On one side, there is FOMO. Boards are demanding an AI strategy. They want GenAI features shipped yesterday.

On the other side, there is FOMU: Fear of Messing Up.

No CIO wants to be the reason the company stock creates a crater because an agent leaked PII or went rogue on social media. This tension effectively freezes production. Companies are stuck in "PoC purgatory"—running endless experiments but terrified to flip the switch for real users.

Fiddler’s bet is that you can’t resolve FOMU without a control plane. You need a layer that sits between the model and the user, monitoring for bias, hallucinations, and safety violations in real-time.

The compound system

The complexity is ramping up. We aren't just hitting a single model anymore. We are building "compound AI systems" and agentic workflows.

Krishna broke down what a modern support agent actually does:

1. It interprets user intent.

2. It converts that intent into a SQL query.

3. It pulls data from a CRM like Salesforce.

4. It feeds that data into an LLM to summarize an answer.

If the answer is wrong, where did it break?

Did the SQL query fail? Was the retrieval from Salesforce bad? Or did the LLM hallucinate during summarization?

Without tracing—the ability to follow the request through every hop in that chain—you are flying blind.

The dual-shore advantage

Since we were in Hyderabad, we talked about talent. Fiddler operates on a "dual-shore" model, splitting teams between the Bay Area and India.

Krishna sees a clear distinction in value. The Bay Area still owns the "zero-to-one." The density of ideas, the meetups, the sheer volume of noise and signal—you can’t replicate that "vibe continuum" elsewhere.

But India owns the scale. Engineers here have spent the last fifteen years building distributed systems for Uber, Google, and Amazon that serve hundreds of millions of users. When you need to build the heavy infrastructure to monitor billions of AI predictions, that systems engineering talent is hard to beat.

The trust gap

I want AI agents to handle the drudgery. I want them to file my taxes, book my flights, and renegotiate my internet bill.

But I won’t let an agent touch my bank account if I don’t trust it.

We are moving from chatbots that simply talk to agents that act. The stakes for error are getting higher. Until we have a control plane that proves these systems are doing what we asked them to do, most of us will keep our hands on the wheel.

You can watch the full episode here