RDMA transfer library with EFA and libfabric

https://le.qun.ch/en/blog/2024/12/25/libfabric-efa-0-intro/

This post from Lequn Chen introduces a hands-on series about saturating the 3,200 Gbps inter-server network on AWS P5 instances, which contain eight NVIDIA H100 GPUs connected by NVSwitch.
The author built a proof-of-concept with RDMA, AWS EFA, libfabric, and GPUDirect, eventually reaching 3,108 Gbps—about 97.1% of the theoretical bandwidth.
Why build it instead of relying on NCCL?

• NCCL’s collective communication assumes a global group of participants, making dynamic node changes awkward.
• Its synchronous programming model is less natural for the author than asynchronous systems such as gRPC.
• Building the networking stack directly is educational and fun.
  The remainder of the 16-part series progressively covers RDMA/EFA fundamentals, libfabric, send/receive operations, GPUDirect RDMA, topology, multithreading, CPU pinning, state sharding, batching, and lazy posting. The largest improvements came from CPU-core pinning, state sharding, batch posting, and finally lazy posting.
  In short: this is the series overview and motivation, not the detailed implementation. Its main promise is a beginner-oriented explanation of how careful low-level system design can drive AWS P5 networking from a small fraction of capacity to nearly full utilization. The accompanying code is available in the author’s libfabric-efa-demo GitHub repository.