Remote Direct Memory Access (RDMA) over Converged Ethernet, or RoCE, is often deployed on lossless Ethernet fabrics to work efficiently. Then why does RoCE run on UDP instead of TCP?

TCP provides reliable transport through mechanisms such as acknowledgments and retransmissions. On the surface, TCP looks like a good fit for carrying RDMA traffic over an inherently lossy medium like Ethernet.

To understand why TCP isn’t necessary, we need to look more closely at how RoCE transport works. RoCEv1 runs directly over Ethernet; RoCEv2 adds UDP/IP encapsulation.

A simplified RoCEv2 stack looks like this, from the RDMA operations down to Ethernet:

Layer	Role
RDMA operations	Remote memory operations
InfiniBand transport (RC)	Sequence numbers, acknowledgments, retries
UDP	Lightweight encapsulation
IP	Routing
Ethernet	Link-layer delivery

Note: RDMA has different transport modes. This discussion focuses on RoCEv2 using Reliable Connected (RC) mode. In RC mode, the NIC handles acknowledgments and retransmissions. Unreliable Connected (UC) and Unreliable Datagram (UD) do not provide this transport-level recovery; higher application or protocol layers must handle loss if reliable delivery is required. See NVIDIA's transport-mode guide for details.

The distinction is between reliable delivery and a lossless network:

• Reliable delivery: The endpoints detect missing packets and retransmit them, subject to retry limits.
• Lossless network: The network tries to prevent congestion-related packet drops in the first place.

RoCE uses InfiniBand (IB) transport mechanisms, which provide reliability in RC mode. These mechanisms run in hardware on an RDMA network interface card (NIC). Reliability does not mean delivery under every failure: a connection can fail when its retry limits are exceeded.

This explains why TCP is unnecessary for reliability here. TCP would introduce another transport layer when IB transport already handles acknowledgment and recovery. UDP provides a lightweight, stateless way to encapsulate those transport packets over IP.

Note: A flavor of 'RDMA over TCP' actually exists, in the iWARP protocol suite. See RFC 5044

Also, using TCP would also bring the CPU and kernel socket stack back into the data path. One of RDMA’s main advantages is that the RNIC can bypass this path and move data directly to or from registered application or GPU memory. Using RoCE’s RDMA transport avoids requiring the OS TCP stack to maintain connection state, acknowledgements, and retransmissions.

If RoCE provides reliable delivery, why use lossless Ethernet?

Packet recovery can be expensive and inefficient, resulting in suboptimal RoCE performance. It takes time to detect lost packets and then retransmit them, usually with a Go-Back-N mechanism or selective repeat on newer RNICs. With Go-Back-N, a single packet loss can result in several later packets being retransmitted, amplifying the cost. A lossless Ethernet fabric commonly uses Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN). PFC tells the sender to pause selected traffic. With ECN-based congestion control, switches mark packets traveling toward the receiver, and the receiver sends congestion notification packets (CNPs) back to the sender so it can reduce its rate. These mechanisms help prevent congestion-related drops and reduce the need for recovery. NVIDIA’s RoCE configuration documentation describes both lossless and lossy modes.

RoCE can also operate on lossy Ethernet. How well it performs depends on the NIC’s recovery and congestion-control capabilities and the network conditions. NVIDIA’s Zero Touch RoCE implementation is one example.

Reliable RDMA needs reliable transport semantics, but it does not inherently require a network that never drops packets. Preventing congestion-related drops reduces the recovery work that the endpoints must do.