Link Layer (L2 / PFC)

Scope: Days AGO and Legacy Days. link.mode accepts None and Pfc:

[link]
mode = "Pfc"

None is the default. Legacy Pfc requires cargo build -p days-legacy --features l2_pfc. Days AGO supports lowered PFC on Scalar, CPU, Metal, and CUDA without that legacy feature.

Legacy PFC

Legacy inserts an egress gate, frame serializer, and ingress port on each directed fabric edge. [link.pfc] is optional in this mode because the actor runtime supplies defaults.

[link.pfc]
xoff = [65536, 65536, 65536, 65536, 65536, 65536, 65536, 65536]
xon = [49152, 49152, 49152, 49152, 49152, 49152, 49152, 49152]
pause_quanta = [65535, 65535, 65535, 65535, 65535, 65535, 65535, 65535]
buffer_capacity = [0, 0, 0, 0, 0, 0, 0, 0]
refresh_interval = 0.0002
drain_interval = 0.000001

Every vector must contain eight entries, one for each 802.1Q priority.

The gate converts pause quanta to seconds as pause_quanta * 512 / link_rate_bps.

Days AGO PFC

Days AGO requires [link.pfc] with eight-entry xoff, xon, and buffer_capacity arrays (or, on the Spectrum-X topology, the by_tier and headroom_by_rate tables below). It implements edge-triggered XOFF/XON. Omit refresh_interval and drain_interval, or set them to zero; nonzero timer values produce this rejection:

PFC refresh/drain timers are outside the T25 executor mechanism; use edge-triggered XOFF/XON

For an enabled priority, validation requires xon < xoff <= buffer_capacity. Set xoff = 0 to disable that priority. A zero pause_quanta entry also disables its priority during lowering. Exact execution models pause/resume state, not a timed pause duration.

Each switch egress port monitors every switch-to-switch link that feeds it, separately per link and priority: the thresholds apply to the bytes that arrived over that link and wait in that port's queue. A paused upstream port serves the first queued packet whose priority is not paused; the packet in service is never preempted.

Host links

By default only switch-to-switch links are PFC-controlled, so a switch never pauses a host and a host sends at its own rate. Set host_links = true to also monitor each host-to-switch link with the same thresholds, so that switches pause host NICs:

[link.pfc]
host_links = true                 # default false
xoff = [0, 0, 0, 60256, 0, 0, 0, 0]
xon = [0, 0, 0, 57184, 0, 0, 0, 0]
buffer_capacity = [0, 0, 0, 97756, 0, 0, 0, 0]

A paused host keeps queuing what its flows send and transmits the first queued packet whose priority is not paused, as a switch port does. Pause and resume at a host appear in the PFC records with the host as the node and queue 0.

RoCE queue pairs also stop sending while their data class is paused at their host. The pacer's next tick sends nothing, adds no pacing credit, and parks the pacer. The resume restarts it at its next pacing tick after the resume. An ACK, NACK or timeout during the pause restarts the pacer as usual, and its next tick parks it again. Other flow types keep generating packets into the host queue while paused.

Because the thresholds are per link, a port fed by many links can hold up to the sum of their buffer_capacity values before every link is paused. Size xoff and buffer_capacity so that this sum fits the switch queue (switch.capacity packets), or the queue drops before PFC holds it. host_links changes the lowered image only when it is true. It runs on Scalar, CPU, Metal, and CUDA.

Thresholds by switch tier on the rail fabric

On the Spectrum-X topology, two tables replace the flat arrays. by_tier gives the XOFF and XON arrays of the leaf (asw) and spine (psw) switches separately, as SimAI sets one threshold per switch from its port count; headroom_by_rate gives each monitor's headroom by the rate of the link it controls, so a monitor's buffer capacity is its XOFF plus that headroom on every enabled priority:

[link.pfc]
host_links = true
by_tier = [
    { tier = "asw", xoff = [0, 0, 0, 4168818, 0, 0, 0, 0], xon = [0, 0, 0, 4165746, 0, 0, 0, 0] },
    { tier = "psw", xoff = [0, 0, 0, 4154756, 0, 0, 0, 0], xon = [0, 0, 0, 4151684, 0, 0, 0, 0] },
]
headroom_by_rate = [
    { rate_bps = 100000000000, bytes = 30574 },
    { rate_bps = 400000000000, bytes = 75000 },
]

by_tier must list the asw and psw tiers once each, with eight XOFF and XON entries that enable the same priorities; it replaces xoff and xon, and it is refused off the rail fabric ("unsupported link.pfc.by_tier off the SpectrumX rail fabric; switch tiers are rail tiers"). headroom_by_rate needs one positive headroom per distinct link rate, replaces buffer_capacity, and must have a row for every controlled link's rate ("link.pfc.headroom_by_rate has no row for a 100000000000 b/s controlled link"). The usual rule xon < xoff <= buffer_capacity and the validator's headroom minimum still apply. The AICB adapter derives both tables from SimAI.conf; see Workloads.

Propagation interaction

link.propagation_ns works with mode = "None". Legacy rejects propagation on PFC links. Exact lowering treats PFC and propagation as separate mechanisms and can place the configured link delay in the lowered image. See Topology and routing for uniform and tiered propagation.