Quality over quantity. True in life, true in love, and true in data. In the initial pell mell of AI training that has defined the last decade, it has always been about more data. Endless gristle for the mill. Now, though, awareness has risen about the need for quality data. Data that is reliable, authentic,  and curated. Data that is free from errors, inconsistencies, and bias. The new emphasis is on ‘clean’ data, and properly established data provenance that leads to desired outcomes, particularly when it comes to LLMs.

Garbage In, Garbage Out

Bad data leads to bad models. “Garbage in, garbage out” (GIGO) has always been a fulcrum of computer science, and nowhere more obvious does this occur than with AI. The strength of the grand-scale mimic god is reliant on what exactly it is mimicking. This is why - to date - models have been shown to collapse and degrade when trained recursively on their own output. They lose diversity, they reinforce systemic errors, and begin hallucinating wildly. This is not just for LLMs, but in variational autoencoders and Gaussian mixture models too. 

The same principles are why large-scale data-scraping from the internet has already nearly hit its ceiling in terms of LLM functionality, and why the initial exponential rate of AI development seen in the last few years has stalled. There are no more easy wins in terms of data collection. One of the best sources of training data are the prompts that millions of human users around the world are typing in within their controlled environment.

Ilya Sutskever stated “Data is the fossil fuel of A.I, and we used it all!”. The limitation for AIs, particularly LLMs, to progress, lies in the quality of data it can train on - often referred to as the “entropy gap.” The more rich, diverse, and high quality the dataset is, the better the AI is at generalizing to unseen scenarios. Bridging the gap requires data to be of unimpeachable provenance, highly curated, uncorrupted, and not loaded with bias. All things that creep into data over its lifecycle, as datasets move from place to place become filled with ever more variegated input. Even LLMs that were working perfectly can begin to degrade if they use inopportune data for inference. 

The difficulty of acquiring and husbanding these datasets is a major reason why AI companies have shifted focus to compute. Quite frankly, it’s easier to build trillion dollar giga clusters than it is to curate high quality datasets. That seems wildly unintuitive - but it’s absolutely true. Quality data is the mana that powers the magic of our modern technological progress, and it’s not just increasingly hard to come by, it’s increasingly hard to maintain. It’s not just about how the data is sourced, but how it is handled in post.

The Unseen Killer of AI Models

Traditional cloud architecture leads to data corruption. Replication errors, “Bit Rot” due to poor hardware, multi-tenant interference of data sets. When GitHub lost data due to a corruption issue in their cloud server and a bug in their multi-region replication system, a lot of data ended up deleted. In a world where data is more valuable than ever, this is catastrophic. That’s not to mention the potential damage caused by adversarial attacks or the effect of synthetic data contamination. When AWS S3 suffered an outage, many datasets were corrupted or lost. The data being used to train LLMs was then compromised, affecting organizational outcomes across a variety of sectors.

Then there is of course the human flaw. Bad labeling and poor annotations, ETL (extract-transform-load) errors introducing inconsistencies or missing values, and versioning conflicts - where different teams modifying and working on datasets create conflicting versions. LLMs trained and delivered through the cloud are thus always at risk of corruption. This is funny when AIs deliver childish output in response to a user prompt, nightmarish when the AI takes control of our energy system. 

Clean Data From Cradle to Grave

It’s not therefore just about what is generating the data, but the entire lifecycle of that data and how it is handled. Missteps at any stage and corrupt dataset quality lead to inopportune outcomes. Generation, collection, storage, cleansing and transformation, utilization and even archiving and deletion should all keep data pristine, to deliver clean results. Appropriate handling of data is a colossal challenge that is essential even in a pre-LLM world. As we seek to use that data to train better models, it becomes imperative if we are to have a chance of success. It’s also essential if organizations are to remain compliant with different data governance jurisdictions like GDPR, CCPA and PIPL. Maintaining data lifecycles across global, differentiated frameworks is a massive challenge, even before taking into account the effect of poor data lifecycling on organizational outcomes.

So, effective data must be clean, without duplication, inconsistency or bias. Data must be provenanced, with unimpeachable lineage that is trustworthy, auditable and uncorrupted, while data lifecycles must be pristine, managed from the cradle to the grave of creation to deletion. If we can achieve all these things, we can not just create better LLMs, but create better coordinated data infrastructure environments that can maximise productivity outcomes in every sector.

How Source Network Fixes the Data Pipeline

Source Network is the distributed data management stack that delivers clean, provenanced, and pristine data right across any infrastructure network, no matter how fast it scales and no matter how variegated its device fleet.

DefraDB’s CRDTs ensure data is not corrupted through version conflicts and that data synced or replicated across devices remains intact. LensVM creates uniform schemas across datasets so that it can be both interoperable with all devices and consistent. SourceHub, our blockchain, is the essential keystone that creates unparalleled trust auditing, reconciliations and recoverability so that should data be poisoned, corrupted, hacked or attacked - through malicious or accidental action - the source of that corruption can be traced and the dataset can be recalibrated to its pristine previous state. All of this is maintained through distributed cryptographic security through Orbis Secrets Management, meaning that changes to datasets are always permissioned, always on a per device or user basis, and trackable. 

In an average supply chain, inventory systems are often updated by multiple vendors and shipping partners in real time. This can lead to conflicts, especially when there is an outage, disruption, or if an air-gapped network fails to update for a period of time. Supply chains need traceable and trackable data about what’s happening to then be able to ensure traceable and trackable products. DefraDB’s use of Merkle-CRDTs can resolve conflicts between data coming from multiple sources simultaneously, including concurrent writes. LensVM ensures that data is homogenised and useful. SourceHub, the trust protocol, then provides tamper-proof provenance of each product’s journey and leaves an auditing trail for how conflicts were resolved across the edge device network, and provide a rollback failsafe in case of catastrophic downtime. In this way, Source Network creates a way for highly sophisticated and variegated supply chains to have a verifiable data lifecycle - even with multiple organizational participants. 

Clean Data, Smarter Systems 

By implementing Source Network’s distributed data stack, we can guarantee pristine data sets that can be monitored effectively at every stage of the life cycle, with no multi-tenant problems, no versioning conflicts, and no replication errors. It can also democratize and canvas a greater slice of data and create regional, industry and linguistic diversity for training - without compromising on the verifiability or veracity of those datasets. With data auditing as standard across devices, it gives organizations a god's-eye view of the data being processed by their edge device network and the ability to ensure consistency, at scale, throughout their infrastructure.

Local LLMs deployed by companies can be fed this clean, provenanced and managed data and operate, infer, and train on data whose lifecycle can be traced, maintained, and compliant at all times - even in the face of unusual or adverse data conditions in the network. This is simply not achievable with 100% certainty with our current cloud-based architecture, and it’s not practical without using distributed tooling to achieve it. 

In a world where clean matters more than ever - and is more valuable than it has ever been - Source Network is the tooling that ensures that our mana wells remain clean, and our spells become more powerful every time we cast them.