Every Era of Compute Requires Its Own Data Layer
There is a pattern in computing that almost nobody names out loud. A new kind of compute shows up. For a while it belongs to a small priesthood, the few people who can wrestle the raw machine into doing something useful. Then someone builds the layer that hides the hard parts of working with data on that machine. The moment that layer arrives, the gates fall open. Everyone can build. The era goes from a research curiosity to an industry.
The compute is never what opens the door. The data layer is.
We hit this pattern early, building our own stack, and it explains every hard call we made along the way. Every computing era has been carried not by its processor but by the data management layer built on top of it. The edge is the newest era. Edge data management, the layer that would let everyone build on it, does not exist yet. That gap is not a footnote. It is the whole opportunity.
The mainframe got a data layer, and computing got an industry
Mainframes were the most powerful machines on Earth, and for years working with data on them was miserable. Data lived in hierarchical and network databases where you navigated the records by hand, following pointers through a structure that mirrored the physical layout of the disk. Your program was welded to that layout. Change how the data was stored and the program broke. Only specialists could touch it, and every new question meant new code.
In 1970, a researcher at IBM named Edgar Codd published a paper that proposed something close to heresy. Separate the logical shape of data from how it is physically stored. Put data in plain tables of rows and columns. Ask for it with a query language instead of hand-written navigation. He called the key idea data independence, and it meant a person could ask a question of the data without knowing, or caring, where the bytes lived.
IBM was in no hurry. It took eight years for the first relational database to ship. Oracle got there as a business, the query language became the standard every vendor spoke, and the effect on the industry was enormous. An analyst could now ask a question in an afternoon that used to take a specialist a week. A company could build a payroll system, an inventory system, a banking ledger, without a team that understood the bare metal. The mainframe stopped being a machine for the few and became a platform thousands of businesses could build on.
The processor did not do that. The data layer did. Same silicon, completely different world, because the hard part of using it had been abstracted away.
The cloud got its own data layer, and it looked nothing like the last one
Then the ground moved. Cheap servers, rented by the hour, reachable from anywhere. The cloud arrived around 2006, and almost immediately the relational assumptions from the mainframe era started to crack.
A single database on a single server, the thing that had served the previous era beautifully, could not hold a planet's worth of users hammering an app at once. So a new data layer grew to fit the new compute, and it was a different shape on purpose. Object storage let you stash a file and get it back by name without ever thinking about a disk. Key-value and document stores threw out rigid tables for speed and scale. The story here is sharper than people remember. Amazon found that a traditional database could not give its shopping cart the always-on availability it needed at its scale, so its engineers built a new kind of store to get it, and that work seeded a whole generation of the key-value and document systems a wave of startups were built on. Warehouses split storage apart from compute so you could query enormous piles of data without owning a server farm. Managed databases meant a developer never touched a machine at all.
The pattern repeats, beat for beat. The cloud's compute was available in 2006. The cloud era did not become something two people in a garage could build on until the data layer caught up to the compute, a few years later, when you could spin up a globally redundant database with a credit card and a few minutes. That, not the raw rented servers, is what made the cloud universal. The data layer was the kingmaker again.
This is the part worth sitting with. The mainframe's data tools did not stretch to cover the cloud. Nobody ran a global social network on a hierarchical mainframe database. The new era did not borrow the old era's data layer. It grew its own, shaped to its own reality. That has happened every single time.
Edge compute is here. Its data layer is not.
Which brings us to now.
The compute has moved to the edge. The silicon crossed the line where on-device work stopped being a compromise. The runtimes for on-device models shipped. The models got small enough and the chips got fast enough. The compute side of this era is real and it is here.
The data side is not. And the reason is the same mistake, made in reverse. The edge inherited the cloud's data layer like a hand-me-down, and it does not fit.
Almost every tool a developer reaches for to store, sync, or govern data carries one buried assumption: there is a server in the middle, it is always reachable, it is the single source of truth, and it is where trust lives. That assumption is the entire cloud data layer. At the edge, that assumption does not bend. It breaks.
Start with where the data is born. Gartner projects that 75 percent of enterprise data will be created and processed outside a traditional data center, up from about 10 percent a few years earlier. The total amount of data the world generates has been climbing toward 175 zettabytes, a figure measured in trillions of gigabytes. A single autonomous machine shows how steep this gets. IDC estimates that one self-driving vehicle generates over 3 terabytes of data per hour. Multiply that across a fleet, a factory floor, a wind farm, a constellation of satellites, a port full of automated cranes. The data is being created in exactly the places the cloud is not. Shipping all of it back to a server to be managed is the precise thing edge compute exists to avoid. Yet that round trip is wired into nearly every cloud-era database and sync client on the market.
The mismatch goes deeper than volume. Cloud data tools assume a single source of truth, one authoritative server that every client checks with. The edge has many peers, often unreachable, that each have to hold truth on their own and reconcile when they meet again. Cloud sync clients treat being offline as an error to recover from. At the edge, offline is not the exception, it is the normal condition, and a tool that panics without a connection is the wrong tool. Cloud databases keep identity, permissions, and the audit trail on a server, so trust means asking that server. Edge devices and the agents running on them need trust to travel with the data itself, provable on the spot without phoning home, a shift we walked through in Edge First - Cloud Last. And cloud schemas come from the center, where one party owns the structure and migrates everyone at once. At the edge, many parties evolve their own data on their own clocks and still have to understand each other.
Why do so many impressive on-device demos never turn into a product? We dug into this in Why Data Is the Bottleneck for the Next Generation of AI [LINK PLACEHOLDER]. The short version: the compute works. The model runs on the chip. The thing that breaks is underneath, in a data layer that was built for a world that always assumed the cloud.
The pattern tells you what happens next
History is not subtle about where this goes. A new compute paradigm has never once been served well by the previous era's data layer. Not the mainframe's, not the cloud's. Each time, the new era grew a data layer in its own shape, and only then did it open up to everyone. The edge will not be the exception. It cannot be, because the shape of the problem is too different.
The stakes are higher this time, because of what people are trying to build on the edge. Edge AI, the agents and on-device models everyone is racing to ship, is a data problem wearing a model's clothing. An agent is its memory, its retrieved context, its plan, its record of what it did. Strip those away and the model in the middle has nothing to stand on. An agent with no durable, portable, trustworthy place to keep its state is not a product. It is a demo that forgets you the moment the connection drops.
So the data layer for the edge will look nothing like the cloud's, the same way the cloud's looked nothing like the mainframe's. It will assume many peers instead of one server. It will treat offline as the normal state of the world. It will carry trust and history inside the data, so a device or an agent can prove what it holds without asking a server for permission. It will let many parties change and reconcile their data without a referee in the middle.
Whoever builds that layer does for the edge exactly what the relational database did for the mainframe and what cloud data tools did for the cloud. It takes a powerful but stubborn new kind of compute and turns it into something every developer, at a company of any size, can build on. That is how an era stops belonging to a priesthood and starts belonging to everyone.
The compute is already here. The data layer is the thing still being written. It is the most important piece of infrastructure of the decade, and it is the piece we are building.
Each era's data layer looked different because each era's compute was different. The edge is the newest, and it needs its own. An edge database has to start from that reality, not the cloud's. At Source, we are building DefraDB, the database built for the edge.
