FL+FSDP and FL+HSDP Hybrid Algorithms Reduce Sharded Data Parallelism Overhead

To mitigate communication bottlenecks in large-scale sharded data parallelism on heterogeneous interconnects, these hybrid algorithms interleave FSDP with FedAvg-style aggregations. This decouples large deployments into smaller, loosely-coupled federation groups, minimizing inter-group traffic while maintaining global batch size constraints.