NVIDIA AI Networking Professional (NCP-AIN) · Foundations

Foundations: why AI traffic is different, RDMA, GPUDirect and NCCL

Why AI training traffic breaks ordinary data center networks, how RDMA works over InfiniBand and RoCE v2, the three layers of GPU-to-GPU communication, and NCCL collectives and benchmarks.

30 min read6 chapters2 labs15 quiz7 scenarios15 interview Q&A

This first module is free: read the lesson and take the quiz. Labs need an account: sign up free to run up to 3 hands-on labs.

Log inStart free trial
This reading material cannot be printed. Read it on config.nwkings.com.
01

Start here: why AI traffic breaks ordinary data center networks

Welcome to the NCP-AIN track. This foundations module is the prerequisite for everything that follows. It has no exam weight of its own, but every tested topic is built on these ideas, so please do not skip it.

What you will learn in this module

  1. This chapter: why AI training traffic breaks ordinary networks.
  2. RDMA in plain words: over InfiniBand and over RoCE v2.
  3. GPU-to-GPU communication: NVLink, the compute fabric and GPUDirect RDMA.
  4. NCCL and collectives: the operations that create the traffic.
  5. Settings and benchmarks: NCCL variables and nccl-tests.
  6. Summary and exam checklist.

Prerequisites

You should know CCNA-level routing, BGP basics and VXLAN basics. If AI clusters are new to you, take the AI Infrastructure and Operations Associate track first as a warm-up.

Analogy: a class project

Imagine a class project where the group may start step 2 only after every student has handed in their part of step 1. If 1,023 students finish in ten minutes and one student takes twenty, the whole class waits twenty minutes. An AI training network is exactly this: thousands of GPUs finish a calculation, exchange their results, and nobody continues until the slowest exchange is complete.

Three terms first

A GPU is the processor that does the training maths. The compute fabric is the network that connects the GPUs of different servers to each other. East-west traffic moves between servers inside the data center, while north-south traffic enters or leaves it (users, storage, management).

Four ways AI traffic is different

  • Few, huge flows. A flow is one conversation, identified by source and destination addresses and ports. Training creates a small number of very long, very fast flows (called elephant flows) instead of millions of small ones. Switches use ECMP (equal-cost multipath): when several equal paths exist, the switch hashes the packet headers to pick one, so all packets of a flow take the same path and stay in order. With millions of flows the hash evens out. With a handful of elephants it cannot: two can land on one link while another link sits idle.
  • Synchronized bursts. All GPUs finish computing at almost the same moment and all send at once. Many senders pushing to one receiver is called incast; very short spikes that fill a switch buffer are microbursts.
  • Tail latency rules. A collective operation (an exchange in which all GPUs of the job take part) finishes only when its slowest transfer finishes. One congested link, or one dropped packet that must be resent, stalls every GPU.
  • East-west dominates, and it is kept separate. GPU-to-GPU traffic has its own compute fabric. Storage, management and user traffic use other networks, and you keep them apart on purpose.
Flow A 400GFlow B 400GFlow C 400GFlow D 400GLeafuplink 1: A + B share it, 200G eachuplink 2: one flowuplink 3: one flowuplink 4: idleSpine

Four elephant flows, four equal uplinks, and an ordinary hash that stacks two flows on one link while another link does nothing.

Why average throughput is the wrong yardstick

A normal enterprise network is judged by average throughput. An AI network is judged by job completion time (JCT): the wall-clock time the whole training job needs. Because each step waits for its slowest transfer, JCT depends on the worst link and the worst moment, not on the average. This one idea explains lossless behaviour, congestion marking, adaptive routing and rail design later in the track.

Worked example. A leaf switch has four 400 Gb/s uplinks. Four GPU NICs each send one elephant flow at 400 Gb/s, and each flow must move 10 GB (80 gigabits) in one step. With a perfect spread, every flow takes 80 / 400 = 0.2 s. A random hash puts the four flows on four different links in only 4! / 44 = 24 / 256 = about 9.4% of cases, so about 90.6% of the time at least two flows collide. Suppose two share one uplink: each gets 200 Gb/s, so those transfers take 80 / 200 = 0.4 s. The step ends when the slowest transfer ends, so the step takes 0.4 s instead of 0.2 s. The job runs at half speed while the average uplink utilisation looks like a healthy 75%.

Illustrative monitoring view (not from a real switch):

Uplink     Speed   Carried   Utilisation
uplink 1   400G    400G      100%    (flows A and B, 200G each)
uplink 2   400G    400G      100%
uplink 3   400G    400G      100%
uplink 4   400G    0G        0%      (idle)
Average utilisation: 75%   Step time: 0.4 s (expected 0.2 s)

Common mistake. Judging an AI fabric by average link utilisation or one simple throughput test. A fabric can show 75% average and still halve the job speed. Always look at per-link numbers and job completion time.

Exam trap. When a question asks what matters most for an AI training network, the answer is job completion time and tail latency, not average throughput.

Slow run, healthy dashboard

A team ran a 128-GPU training job on 16 servers with 8 GPUs each. Step time was 0.41 s against a benchmark of 0.20 s. The dashboard showed uplinks averaging 60%, so the network looked fine. The engineer then pulled per-link counters on one leaf and found two uplinks at 100% and two at 20%: several elephant flows had hashed onto the same two uplinks. Switching to per-packet load balancing (adaptive routing, covered in the Spectrum-X module) brought every uplink to an even 60% and the step time to 0.21 s.

Lesson: The average was 60% both before and after. Only the per-link view and the step time showed the problem.

"Why can ECMP hurt a training job even when average link utilization is low?"

Training produces a few very large, long-lived flows, and ECMP assigns each whole flow to one path by hashing its headers. With so few flows the hash cannot balance the load, so two flows can share one link while another is idle. A collective finishes only when its slowest transfer finishes, so the job runs at the speed of the most crowded link even though the average looks fine. The fix is finer-grained balancing such as adaptive routing.

Key takeaways

  • AI training makes a few huge flows, sends in synchronized bursts and waits for the slowest transfer.
  • ECMP hashes whole flows to links, so a handful of elephant flows can collide while other links idle.
  • An AI fabric is judged by job completion time and tail latency, not by average throughput.
  • The compute (east-west) fabric is kept separate from storage, management and user networks.
  • This module is a prerequisite with no exam weight, but all the tested domains build on it.

Upgrade to keep reading

The free plan lets you read chapter 1 of this module. Upgrade to open the remaining chapters.

  1. RDMA in plain words: InfiniBand and RoCE v2
  2. GPU-to-GPU communication in three layers: NVLink, the compute fabric and GPUDirect RDMA
  3. NCCL and collectives: All-Reduce, All-Gather and the traffic they create
  4. NCCL settings and benchmarks: reading busbw with nccl-tests
  5. Summary and exam checklist: foundations

Join the community