 |
ML Newsnew | past | best | rss | submit | about |
| SparseLoCo: Communication-Efficient LLM Training (arxiv.org) | |
| 19 points by synapz_org on Sep 1, 2025 | hide | past | pdf | 1 comment on HN | |
|
In plain words: When training sites sync, this sends only the largest few changes and rounds them to 2 bits, instead of shipping every change at full size. It cuts the data sent by 97-99% and still ends with a lower loss than the usual dense approach. Abstract · Overcoming the Communication-Performance Tradeoff in LLM PretrainingCommunication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwidth-constrained settings, such as across datacenters and over the internet. While these local-update methods achieve communication reduction through reduced synchronization, they still require communicating dense model-sized pseudo-gradients, resulting in a communication bottleneck even for cross-datacenter links. While quantization is often applied to reduce the pseudo-gradient's size, in the context of LLM pre-training, existing approaches have not been able to leverage sparsification without incurring significant performance degradation. In this work, we introduce SparseLoCo, a communication-efficient training algorithm for LLMs that can effectively leverage Top-k sparsification and 2-bit quantization to reach extreme sparsity in the communicated pseudo-gradient, as high as 97-99%, while achieving lower final loss than dense DiLoCo. In our empirical study of language model pre-training, we demonstrate that SparseLoCo's improvements over DiLoCo in performance and pseudo-gradient compression are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing the number of workers, and increasing communication intervals. | |
| add comment on HN |
Templar AI has developed SparseLoCo, a distributed training algorithm that achieves extreme compression ratios (1-3% sparsity + 2-bit quantization) while outperforming existing methods like DiLoCo and DeMo on both loss and communication efficiency.
The Core Problem
Training LLMs across data centers or over the internet is bottlenecked by communication: as model scale grows, each synchronization can require transferring hundreds of gigabytes of pseudo-gradients. DiLoCo reduces the frequency of synchronizations, but the communication remains dense and large. This makes distributed training impractical for many scenarios, especially internet-scale collaboration.
Technical Approach
Our key insight: The infrequent communication of DiLoCo can be aggressively compressed via TOP-k sparsification while improving performance.
Algorithm highlights:
* Replace global momentum with per-replica error feedback * Apply TOP-k magnitude compression (1-3% density) + 2-bit quantization to pseudo-gradients * Maintain infrequent communication (H=15-250 steps) like DiLoCo * Use chunked TOP-k for better parallelism and reduced index overhead
Results
Communication reduction: With >97× compression, SparseLoCo outperforms DiLoCo across all benchmarks. Sparse aggregation appears to provide regularization benefits beyond just compression.
Communication infrequency: Consistently outperforms DiLoCo across communication frequency ∈ {15, 30, 50, 100, 250} on 512M parameter models.
Real deployment: Currently running on Bittensor with a 70B model and 20 participants in the gather operation (out of many more total participants): 70 seconds communication with <500Mbps bandwidth. Our previous successful deployment of a medium sized (200B token) run of an 8B parameter model and 20 gather participants achieved communication average of 12 seconds vs 4.5 minutes compute time.
Key Technical Contributions
1. Local momentum approximation: Show that DiLoCo's global outer momentum can be well-approximated by local accumulators (>90% cosine similarity)
2. Error feedback as momentum: Demonstrate that TOP-k + error feedback naturally provides similar benefits to outer momentum
3. Sparse aggregation benefits: Find that sparse aggregation actually improves performance over dense methods—likely due to emphasis on high-saliency components
4. Extreme quantization: Error feedback enables 2-bit quantization without additional accumulators or performance drops
Implementation Details
* Chunked TOP-k (4096 elements/chunk) reduces index transmission overhead * Custom index compression: 8.9, 6.6, 5.6 bits per value for different sparsity levels * Drop-in replacement for DiLoCo all-reduce operations * Compatible with existing distributed training frameworks
Limitations & Future Work
* Tested on 512M parameter models (though deployed on 8-70B) * Chunk size optimization could be further explored * Random-k performs significantly worse than TOP-k
This work makes distributed training viable over commodity internet connections and opens possibilities for global AI training collaborations that were previously bandwidth-prohibited.