Nobody talks about Hadoop anymore.

Fifteen years ago it was the most important word in enterprise data. If you weren't standing up a cluster, you were behind. Today you can go a whole year without hearing the word. Hadoop didn't fail. The problem it solved got absorbed into something you no longer have to think about.

I came up in the Bill Inmon and Ralph Kimball era, building and leading data teams in consulting, so I watched this shift play out across one enterprise after another. The AI agent market is repeating it right now, almost beat for beat. Here is how it goes, and where it leaves your money.

When storage was expensive, discipline was the job

In the years before Hadoop, storage and compute cost real money, so discipline was the whole game. You kept analytics off your transactional systems, modeled data warehouses and data marts by hand, and earned your keep by being clever, because you couldn't just throw hardware at the problem.

Then the constraint disappeared. HDFS let you spread data across a pile of cheap commodity machines: a master node (the NameNode) tracking where everything lived, and an army of workers (the DataNodes) doing the work. Storage and compute were suddenly cheap and, for practical purposes, endless. We stopped talking about warehouses and started talking about "data lakes," a term James Dixon coined in 2010: pour everything in raw, sort it out later. And discipline went out the window. Why model carefully when you can keep everything and add another node?

The real bill was always the people

Here is the trap, and we are walking straight back into it. A single hundred-node cluster took four to eight full-time engineers just to keep alive, and a loaded Hadoop administrator ran well north of $400 per hour as a contractor. That talent barely existed: McKinsey warned of a shortage of 140,000 to 190,000 people with deep data skills in the US alone.

Enterprises hired the scarce people and built it in-house anyway, because owning your data lake felt like owning your future. Cloudera and Hortonworks offered a middle path, a packaged and supported Hadoop, but you still ran the cluster yourself. Even they couldn't make the math work: Cloudera booked $261 million in revenue the year before its IPO and lost $187 million doing it, and the two merged in 2019. Packaging the plumbing was never the winning move.

The storage got cheap. The people who could run it never did.

The winning move came from AWS, Microsoft, and Google, and it was different in kind. They did not hand you a better cluster to operate. They made the cluster disappear and sold you the result, rented by the query. Everybody migrated. Nobody brags about the size of their cluster now; we just talk about outcomes.

I have seen this movie before

Now look at AI agents. Intelligence just got cheap and abundant, the same way storage did, and the cost curve is running the same script. The blended price of a million tokens fell 67% in a single year, from $18.40 to $6.07, and frontier prices are down roughly 95% since early 2023.

So watch the bills. Even as the unit price collapsed, total enterprise AI spend jumped 320% in a year, and nearly three-quarters of companies say AI costs blew past budget. Uber handed Claude Code to 5,000 engineers and burned its entire 2026 AI budget in one quarter, then capped spend at $1,500 per tool per month. Agentic work is why: an agent can burn a thousand times the tokens of a person in a chatbot.

When the unit gets cheap, the bill gets big. It happened with terabytes. It is happening with tokens.

The rest of the pattern is all there too: the scarce specialists, the homegrown frameworks and evals, the agents that drift and need re-tuning every time a model ships. Running a fleet of agents in production feels a lot like babysitting a cluster, and the instinct is identical: build it yourself, because owning your AI feels like owning your future.

"But AI is different"

The pushback I hear most is not about technology. It is about value. Founders and boards, from seed-stage startups to the largest enterprises, believe that owning and controlling the AI is what compounds their valuation. If AI is the product, the thinking goes, you cannot rent the thing that makes you worth more.

The second objection is data. Sending your data to someone else's models feels like a risk, and for some that fear is reviving the urge to build private infrastructure, even their own data centers, to keep everything in-house.

We heard both arguments last time. Owning the data lake was supposed to be the moat and the multiple. It wasn't: value followed your data and your judgment about what to do with it, never the cluster underneath. Companies that poured capital into owning infrastructure mostly bought a tax on operating margin that only a handful could justify, and building your own data center to run agents is that same bet at a scale almost no one can afford. The data worry is now solved for serious providers: isolation, no training on your data, and private deployment give you control without owning the plumbing. MIT studied hundreds of enterprise AI efforts and found internal builds succeeded only about a third as often as buying, while 95% of pilots showed no measurable P&L at all.

Where this goes

The abstraction is coming, and it will win, because it always does. In a few years most companies won't build or manage agents any more than they run their own distributed file systems today. They won't think about agents at all. They will think about operations: capacity, velocity, throughput, work getting done without bottlenecks.

Yesterday's flex was the size of your cluster. Today it is how many tokens you burn.

Nobody brags about the size of their clusters anymore. Someday, nobody will brag about their agent framework either.

Before you commit the budget, a few questions to consider:

• Are you hiring to produce a business outcome, or to operate infrastructure? If the roles mostly keep agents alive, that is infrastructure management in disguise.

• When a better model ships next quarter, who does the migration, and what does it cost you?

• Can you name the KPI your agents move, in dollars? If the honest answer is "productivity, somewhere," you are in the 95%.

The durable skill was never wiring the plumbing together. It is knowing which outcomes to demand and running the operation that delivers them. That is the move from builder to architect, and it survives every turn of the technology.

So watch which layer each vendor is really selling. The agent platforms and frameworks that hand you a better way to build and run your own agents are this era's Cloudera: useful, but still your cluster to babysit, and riding for the same fall. The layer that lasts makes the agents disappear and delivers the result. That is the bet we've made at Qurrent , a managed digital workforce measured on outcomes written into the contract, with the technology swapped underneath as it changes. You can weigh where AI is today however you like; the pattern holds either way.

Nobody talks about Hadoop anymore. In a few years, nobody will talk about AI agents either. The winners won't be the ones who built the most impressive clusters. They will be the ones who spent that time getting the work done.

Originally published on LinkedIn

Sources

• Uber's 2026 AI budget spent in a quarter, $1,500 per-tool cap: Fortune, Benzinga

• Token price down 67% while total AI spend rose 320%, 73% over budget, agents ~1,000x tokens: The Modern Data Company

• Frontier LLM prices down ~95% since 2023: DeepLearning.AI

• MIT: 95% of GenAI pilots show no P&L impact; buying beats building: Fortune

• McKinsey: 140,000 to 190,000 deep-analytics talent shortage: McKinsey Global Institute

• Hadoop TCO (4 to 8 engineers per cluster, loaded admin cost): The Ramp / Jim Scott

• Cloudera and Hortonworks losses: Fortune, The Register

• "Data lake" coined 2010: James Dixon; Cloudera and Hortonworks merger 2019: Cloudera