Multi-PetabyteWholeGenomeImagingFabric
400GbE RoCEv2 storage cluster and parallel file system delivering 120 GB/s sustained throughput for high-throughput gene sequencers.
The Brief
The St. Jude Genomic Research Pavilion runs sixteen high-throughput Illumina NovaSeq sequencers and four cryogenic electron microscopy (cryo-EM) suites dedicated to pediatric cancer biology. Each sequencing run outputs terabytes of raw fluorescence imagery that must be written directly to high-speed scratch storage without frame buffering, or the entire flow cell run is invalidated.
The existing storage area network (SAN) suffered severe bottlenecking whenever more than six sequencers were running simultaneously. Buffer overrun errors caused three ruined sequencing runs in a single quarter, wasting hundreds of thousands of dollars in chemical reagents and delaying critical patient tumor profiling.
Apache Spark was brought in to architect, cable, and benchmark a non-blocking 400GbE storage fabric and parallel file system capable of sustaining 120 gigabytes per second of raw sequential ingestion.
Engineering Approach
We engineered an end-to-end NVMe over Fabrics (NVMe-oF) storage architecture utilizing RoCEv2 (RDMA over Converged Ethernet) with Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). The physical plant features MPO-16 fiber trunk cabling connecting high-density flash storage nodes to dual 400GbE spine switches. We deployed Lustre parallel file systems tuned for large sequential writes alongside an automated S3-compatible cold tiering daemon that migrates verified datasets to tape and object stores upon pipeline completion.
Operational Outcome
Achieved 142 GB/s verified sustained write throughput, eliminating sequencer buffer overflow errors entirely. All 16 sequencers and cryo-EM rigs now operate concurrently at 100% capacity with zero dropped capture frames.
Schematic Delivery Pipeline
Sequencer optical outputs connect via 100G links directly to leaf switches.
Zero-loss 400G spine fabric routes RDMA packets with zero CPU copy overhead.
All-NVMe storage arrays ingest parallel write stripes across 24 high-density storage nodes.
Automated policy engine migrates completed alignment outputs to high-density cold tiers.
Legacy vs Modernized Architecture
Technical Plates & Trace Graphs
High-density 400GbE MPO-16 fiber optic trunk routing in clean room data suite.
RDMA throughput benchmark graph showing 142 GB/s saturation plateau.
Instrument interface cable dressing for Illumina NovaSeq integration racks.
“Our sequencers generate an ocean of data every day. Apache Spark built the plumbing that handles it effortlessly. We have not lost a single sequencing run since day one of cutover.”