DeepSeek Open-Sources Fire-Flyer File System (3FS): DeepSeek’s Storage Breakthrough for AI Workloads

DeepSeek has unleashed a seismic shift in AI infrastructure with the open-source release of ​Fire-Flyer File System (3FS), a distributed storage system engineered to obliterate data bottlenecks in modern AI workloads.

As the final bombshell of its five-day “Open Source Week,” 3FS has already garnered ​1,100+ GitHub stars​ in under 2 hours, with developers dubbing it “the storage backbone for the AGI era” 💥

Project URL: https://github.com/deepseek-ai/3fs

---

​Why 3FS Redefines AI Storage​

Traditional file systems crumble under the dual pressures of ​petabyte-scale datasets​ and ​real-time inference demands. 3FS solves this through:

1️⃣ ​Disaggregated Architecture:

• Pooling thousands of SSDs and hundreds of storage nodes
• Enabling location-agnostic data access at 6.6 TiB/s throughput

2️⃣ ​CRAQ-Based Strong Consistency:

• Chain Replication with Apportioned Queries protocol
• Eliminates synchronization nightmares in distributed training

3️⃣ ​AI-Optimized Workflows:

• 40 GiB/s KVCache throughput for LLM inference
• Zero-copy checkpointing at 3.66 TiB/min

---

​Performance That Shatters Limits​

Tested on 180-node clusters (2×200Gbps InfiniBand, 16×14TB NVMe SSDs):

​Scenario​	​Metric​	​Result​
Stress Testing	Read Throughput	6.6 TiB/s
GraySort Benchmark	110.5 TiB Sorting Time	30m14s
LLM Inference	KVCache Throughput	40 GiB/s

---

​Core Innovations​

1. Hardware-Obsessive Design​

• Full utilization of RDMA networks and NVMe SSDs
• 4K random read latency under 50μs

2. Unified File Interface​

• FoundationDB-backed metadata service
• POSIX-compliant API requiring zero learning curve

3. End-to-End AI Pipeline Support​

• Training: Instant dataset shuffling without prefetching
• Inference: Cost-effective DRAM replacement for KVCache

---

​Smallpond Synergy​

The companion framework (also open-sourced) supercharges 3FS:

pythonimport smallpond  
sp = smallpond.init()  # PB-scale data processing in 5 LOC[7](@ref)

• 40% faster data preprocessing
• 30% GPU memory reduction

---

​Developer Frenzy​

@AI_Engineer: “3FS cut our checkpoint I/O time from hours to minutes—this changes everything!”
@MLOps_Leader: “Finally, a storage system that scales with our 100B-parameter models!”

---

​The New Storage Paradigm​

3FS completes DeepSeek’s open-source AI stack:

• Day 1: FlashMLA (580 TFLOPS inference)
• Day 5: 3FS (6.6 TiB/s storage)

As DeepSeek’s CTO declared: “Democratizing AI starts with demolishing storage bottlenecks—3FS is our crowbar.”

👉 Clone 3FS on GitHub