AI data storage splits durable archives from fast access
AI data storage has to reconcile two different requirements: retaining expanding datasets affordably and delivering the smaller active portion fast enough to keep graphics processing units (GPUs) working. Backblaze has documented a two-tier design in which B2 Cloud Storage holds the durable copy of datasets, derived files and checkpoints, while NVIDIA AIStore keeps frequently used data on local storage close to compute. A checkpoint is a saved training state that lets work resume or be reused later.
The design reflects the way AI projects use data. Teams may retain source datasets, curated revisions and successive checkpoints, but an individual job reads only a changing subset. Curation can revisit raw material whenever its parameters change, multiple engineers can reuse the same revision, and training may read one revision over many epochs, meaning complete passes through the training data. Keeping the entire archive beside the GPUs can be expensive, while repeatedly reading it from remote object storage can affect performance.
AIStore is an open-source, scale-out object store, meaning it can spread data and requests across multiple machines as demand grows. It can attach a remote B2 bucket through an S3-compatible API, an interface designed to work like Amazon S3. The first request for an object retrieves it from the remote system of record. Subsequent requests can use the cached local copy while it remains available and valid. Supported S3 applications can point at the AIStore endpoint through configuration rather than replacing their storage-access code.
The software also provides operations aimed at training workloads. A prefetch job loads a known dataset revision before processing begins. Write-through stores a local copy and uploads the result to the remote record in the same request. GetBatch combines many small objects into one ordered stream instead of issuing a separate request for each object. In the layered design, data travels from B2 through AIStore to the application and GPU memory for decoding, batch preparation and model execution. Results and checkpoints travel back through AIStore into B2.
Backblaze built two sample workflows to demonstrate the pattern. The video-curation example uses NVIDIA NeMo Curator, an open-source, GPU-accelerated library for preparing large generative-AI datasets. Full runs prefetch source videos, while limited runs or runs with prefetch disabled retrieve uncached inputs when needed. Curator produces clips and Cosmos-Embed1 embeddings, numerical representations of content used by AI systems, then publishes them through AIStore to B2. Later curation runs can reuse valid videos still held in the cache.
The robotics-training example covers trainers that require an ordinary local directory. NVIDIA Isaac GR00T is an open vision-language-action model for humanoid robot skills. Its demonstration datasets contain many small files, including per-episode tables, camera video and metadata. The sample stages one revision from AIStore onto the GPU node, checks it against a manifest, which is an inventory of the expected files, and then starts training without changing the existing data loader. Checkpoints are published through AIStore to B2. Staging that revision again can read the cached copy rather than retrieving it from B2.
Backblaze sets out five operational rules for keeping these tiers independent. First, size the fast tier for simultaneous working sets and likely reuse, not for the whole archive. It also needs enough headroom for prefetched data to remain resident until staging finishes. For remote buckets, AIStore enables least recently used, or LRU eviction, by default. It starts removing older cached data when disk use crosses a cluster-wide high watermark, which is 90% by default. Operators therefore need to set that threshold around the workload rather than treating it as an incidental default.
Second, prefetch data when predictable startup matters. Third, treat dataset prefixes as write-once locations: each revision should receive a new path, such as raw/revision-001/, rather than modifying an existing revision. AIStore can validate cached objects against B2 on every read through versioning.validate_warm_get for out-of-band changes. Unique, unchanging prefixes avoid that per-read check for completed revisions. For data retained for audits or reproducibility, B2 Object Lock can prevent a locked object version from being deleted during its retention period.
Fourth, publish a verifiable manifest with every result. Both samples initially mark a _SUCCESS.json record as pending. After every object arrives, they mark it complete and record each file's size and SHA-256, a cryptographic value used to detect whether a file has changed. When an input manifest exists, the output provenance also carries its hash so the originating dataset revision can be identified.
Fifth, measure initial loading, warm-cache reads, processing and publication separately because each phase responds to different changes. Publication may be the largest transfer: checkpoints contain model weights and optimizer state, and Backblaze says even its 20-step GR00T demonstration can publish tens of gigabytes. The sample repositories compare direct B2 retrieval with cold and warm AIStore reads. Their integration tests confirm that warm reads make zero B2 GET requests.
Credential boundaries also matter. Curator and GR00T child processes do not receive the B2 key. AIStore targets and utilities that connect directly to B2 do require it. This limits where the remote-storage credential must be distributed without changing the application's S3 workflow.
The financial case depends on reuse. Jobs sharing one AIStore cluster can consume the same cached objects without downloading them repeatedly. Backblaze says this can reduce reads counted against B2's free egress allowance, which is three times average monthly storage. It also says egress to compute partners such as CoreWeave is free without limit. For a single sequential pass, however, streaming sharded data directly from B2 into the data loader is simpler.
AIStore remains infrastructure that a team must operate. Kubernetes with ais-k8s is one production deployment option. Write-through adds a local write before each upload, although NVIDIA's benchmarks show that this overhead shrinks as object size increases. The trade-off is that published results stay cached for another job to reuse. Operators should compare that benefit with the additional upload work for each workload.
For organisations considering this separation of capacity and performance, AEU Data provides European cloud and data infrastructure as an option for the underlying data layer.