VAST Data says AI needs a new kind of parallel file system

Interview. Our article attempting to explain parallel file systems prompted Jan Heichler, Director of AI/HPC PreSales EMEA at VAST Data, to get in touch and put across the view that parallel file system (PFS) needs in the AI era are quite different from those of traditional or classical HPC PFS. The basic dividing lines now are not striping vs non-striping but metadata structure and coping with large numbers of small files and objects.

Blocks & Files: Which is more important in workload and metadata architecture now – striping or non-striping?

Jan Heichler: Classical striping parallel file systems (PFS) were originally designed for traditional HPC workloads, where many processes write concurrently into the same large file, and metadata subsystems were relatively weak, so striping compensated for that.

For modern AI workloads, data is often organised very differently. There’s usually no requirement to aggregate data into huge monolithic files. In fact, with a strong metadata architecture, treating images, embeddings, or documents as individual files can provide much better parallelism than striping a single multi-terabyte file.

So the real distinction today is not striping itself, but the metadata design, which determines how much concurrency the system can sustain, how small-file workloads are handled, and how the system scales in AI, not just HPC.

Blocks & Files: You say modern architectures differ from classical parallel file systems. How so?

Jan Heichler: Your article’s categorisation is right at a high level, but there are a few architectural distinctions worth adding.

Many classical PFS designs historically relied on centralised metadata components (e.g., Lustre’s MDS). Systems like GPFS/IBM Spectrum Scale evolved beyond that model, but the general pattern in traditional parallel file systems has been one of “shared-nothing” storage nodes, a namespace stitched together from independent building blocks, and attempts to minimise global knowledge across the cluster for scalability reasons.

This architectural pattern has consequences. Global data reduction becomes difficult or impossible, rebuild domains become siloed, and metadata becomes the gating factor for concurrency. Performance can depend heavily on where a file is placed, or which node “owns” it.

By contrast, architectures like VAST employ a disaggregated, shared-everything (DASE) model where data and metadata are scattered everywhere across everything. Clients aren’t tied to specific controllers or owners, the system doesn’t rely on fixed block or chunk sizes, but adapts layouts dynamically, and global knowledge is a feature, not a scaling limiter.

This is why DASE can support global data reduction, globally consistent metadata, and failure domains that are not tied to isolated building blocks.

Blocks & Files: If striping is not so important with AI, what is?

Jan Heichler: AI workloads stress metadata, concurrency, and multi-protocol behavior. AI pipelines generate a workload mix very different from HPC. They involve millions of small, high-fanout reads, extremely parallel random I/O, concurrent access from GPUs, agents, or microservices, and hybrid file + object usage patterns.

Your article rightly notes object storage’s scaling advantages. But the bigger story is multi-protocol behavior. AI developers overwhelmingly prefer object interfaces for cloud-native development, portability, and library support, while file remains essential for POSIX, ML tooling, NFS-based pipelines, and existing applications.

The emerging requirement is a single platform capable of full parallel file semantics, full-performance object semantics, and shared metadata and namespace between them.

This is one of the key differences between classical PFS designs (which are primarily file-oriented) and modern AI data platforms.

Blocks & Files: Are there more AI-relevant architectural distinctions?

Jan Heichler: Yes. Global data reduction: traditional shared-nothing PFSes avoid global knowledge for scalability reasons, which means they can’t perform global dedupe or compression intelligently. VAST’s DASE makes this possible.

In classical designs, data typically rebuilds within isolated silos. In DASE-style systems, the entire cluster participates in recovery, making rebuilds faster and reducing the risk of correlated failures.

Legacy PFS clients are often tied to specific nodes or controller paths, making for concurrency limits. In DASE, any client can talk to any component at any time, so concurrency grows with the cluster.

Blocks & Files: Summarize your view of the shift from classical PFS to AI PFS.

Jan Heichler:Both systems support parallel access, but they achieve it through fundamentally different architectural principles. A useful way to frame the shift might be that HPC parallel file systems are striping-oriented, shared-nothing, sometimes centralized metadata, and optimized for few large files. AI Parallel Systems feature disaggregated metadata, shared-everything design, multi-protocol (file + object), and are optimized for massive small-file parallelism and GPU concurrency.