Flows

Scope: Days AGO and Legacy Days. Days accepts explicit [[flow]] entries and generated [[flow_set]] entries. Legacy Days supports the full schema below, except the [flow.traffic.dcqcn] table: since P16 the two engines read different DCQCN keys, and the legacy keys are on the Legacy flows page. Days AGO accepts a deterministic subset and returns a specific error for unsupported fields or mechanisms.

Explicit flow

[[flow]]
flow_type = "PacketDistribution"
priority = 0
graph = [[0, 1]]

[flow.traffic]
initial_delay = 0.0
size = 10000
arr_dist = { type = "Uniform", low = 0.001, high = 0.001 }
pkt_size_dist = { type = "DiscreteUniform", low = 512, high = 512 }

For an explicit path, its first and last IDs must match the source and sink attachment switches. A configured path cannot accompany routing = "ShortestPath" or "ECMP". If you set routing = "PathFromConfig", you must also provide path.

Flow set

[[flow_set]]
first_flow_id = 100
flow_type = "PacketDistribution"
flow_count = 10
priority = 0
pairing = "Random"

[flow_set.traffic]
initial_delay = 0.0
duration = 10.0
arr_dist = { type = "Uniform", low = 0.001, high = 0.001 }
pkt_size_dist = { type = "DiscreteUniform", low = 512, high = 512 }

flow_count is required. first_flow_id, dependencies, priority, and routing have the same legacy meaning as their explicit-flow counterparts. pairing accepts:

• Random (default), seeded from seed
• SwitchOffsetHalf
• SameSwitchNext

See Topology and routing for structural-pairing constraints.

Traffic

Each [*.traffic] table requires arr_dist, pkt_size_dist, and exactly one termination field:

• size: byte bound
• duration: simulated seconds after the flow starts

initial_delay defaults to 0.0. Legacy validates finite, non-negative times and valid distribution bounds.

Supported distribution forms are:

arr_dist = { type = "Exp", lambda = 1.0 }
arr_dist = { type = "Uniform", low = 0.001, high = 0.002 }
pkt_size_dist = { type = "DiscreteUniform", low = 1000, high = 1500 }

Legacy samples those distributions. Exact lowering requires a positive constant packet size. PacketDistribution also needs a positive constant arrival interval; it rejects exponential and nonconstant inputs because the image stores deterministic precomputed traffic.

TCP

[flow.traffic.tcp]
cc_algorithm = "TCPReno" # or "TCPCubic", "TCPBBR"
ecn = false

Legacy requires this table for TCP. TCP packet size must be a positive fixed integral MSS. Byte-bounded legacy TCP retains arr_dist only as a compatibility token and does not sample it.

Legacy supports Reno, CUBIC, and BBR. CUBIC accepts:

[flow.traffic.tcp.cubic]
beta = 0.7
c = 0.4
fast_convergence = true

Exact lowering supports byte-bounded Reno and CUBIC. It rejects BBR, duration-bounded TCP, TCP ECN, and non-default CUBIC parameters. Metal and CUDA execute the supported exact TCP state and warm-start checkpoints.

DCQCN

Scope: Days AGO. The legacy engine's DCQCN keys are on the Legacy flows page; the two tables differ.

Days AGO runs the Mellanox form of DCQCN: the reaction point of HPCC's ns-3 model (CC_MODE 1) and of SimAI, in exact integer arithmetic. [flow.traffic.dcqcn] configures it on an unreliable flow_type = "DCQCN" flow and on a RoCE queue pair. Only max_rate_gbps is required; the other controller keys default to the values SimAI and HPCC ship.

[[flow]]
flow_type = "DCQCN"
priority = 0
graph = [[0, 1]]

[flow.traffic]
initial_delay = 0.0
size = 200000
arr_dist = { type = "Uniform", low = 1, high = 1 }
pkt_size_dist = { type = "DiscreteUniform", low = 1000, high = 1000 }

[flow.traffic.dcqcn]
max_rate_gbps = 10.0              # required: the line rate
rate_gbps = 10.0                  # default max_rate_gbps
min_rate_gbps = 0.1
g = 0.00390625
ai_rate_gbps = 0.05
hai_rate_gbps = 0.1
alpha_resume_interval_ns = 1000
rate_decrease_interval_ns = 4000
rp_timer_ns = 900000
fast_recovery_times = 1
clamp_target_rate = false
pacing_interval_ns = 800          # one 1,000-byte packet per tick covers 10 Gbps
cnp_interval_ns = 0               # unreliable flows only
cnp_priority = 0                  # unreliable flows only; default priority

Rates are exact decimals in gigabits per second and lower to integer bits per second. They must satisfy min_rate_gbps <= rate_gbps <= max_rate_gbps, and the minimum must be at least 3 bit/s. g is held as a 63-bit binary fraction, rounded half to even: exact for a dyadic value such as 1/256 or 1/1024, and within 2^-64 otherwise; it accepts at most 19 significant digits and 38 decimal places. The three intervals and pacing_interval_ns must be positive, and fast_recovery_times below 4294967295.

The paper-form keys of earlier releases, mi_factor, rtt_ns and increase_byte_threshold, are gone, not deprecated. A scenario that sets one fails to lower:

unsupported paper-form DCQCN key `mi_factor` (removed in P16: the Mellanox-form cut is alpha / 2); see docs/content/docs/configuration/flows.mdx

The rate-increase period that rtt_ns used to name is rp_timer_ns.

What the controller does

The controller keeps a current rate R_C, a target rate R_T, the estimate alpha and a stage counter. Until its first feedback it is pristine: R_C = R_T = rate_gbps, alpha = 1, and no timer runs.

• Feedback. On an unreliable flow, feedback is a CNP arrival. On a queue pair, it is an ACK or NACK whose ECN echo is set (see below). The first feedback at time t starts the alpha update at t + alpha_resume_interval_ns and then every interval, and the rate-decrease check at t + rate_decrease_interval_ns + 1 ns and then every rate_decrease_interval_ns; the first check always cuts. Every later feedback marks both the alpha update and the decrease check as pending; the first feedback marks only the decrease check, so the first alpha update decays unless a second feedback arrives before it.
• alpha update. alpha becomes (1 - g) * alpha + g if feedback arrived since the last update, and (1 - g) * alpha otherwise. alpha is a 63-bit fraction and each update floors once.
• Rate-decrease check. With feedback since the last check: R_T is set to R_C when clamp_target_rate is on or the stage is not 0; then R_C = max(min_rate_gbps, floor(R_C * (1 - alpha / 2))), the stage returns to 0, and the rate-increase timer restarts rp_timer_ns after the cut.
• Rate-increase timer. Every rp_timer_ns after the last cut: below fast_recovery_times, fast recovery; at it, R_T grows by ai_rate_gbps; above it, by hai_rate_gbps, clamped to max_rate_gbps in both cases. Then R_C = floor(R_C / 2) + floor(R_T / 2) and the stage advances. The timer keeps firing until the flow completes.

The controller owns no events. Its three timers are applied at the flow's next transition (a feedback, a pacing tick, an ACK, a timeout), in the order the eager model would have run them: at one instant, feedback arrivals first, then the alpha update, then the rate increase, then the rate decrease, then the flow's own pacing tick or timeout. The result is identical to running the timers as events, without the per-flow event stream. This order departs from ns-3's in two cases, taking ns-3's order of equal-time events to be its scheduling order, which the P16 design evidence infers from the scheduler rather than reads from a run. First, when rp_timer_ns equals rate_decrease_interval_ns, ns-3 runs the decrease check before the increase at their shared instant, and Days AGO runs the increase first; lowering does not reject that pair of values, and the shipped defaults (900000 and 4000) do not have it. Second, when a feedback (an echoing ACK or NACK, or a CNP) lands on the exact nanosecond of a decrease check, ns-3 runs the check first, because it was scheduled earlier, and Days AGO applies the arrival first, so the cut happens at that check instead of one rate_decrease_interval_ns later. The second case can occur under any configuration whenever an arrival time falls on the check grid.

The controller freezes when the flow completes: at the ACK that acknowledges a queue pair's last byte, or at the pacing tick that emits an unreliable flow's last byte. Feedback that arrives later changes nothing.

An unreliable DCQCN flow's receiver is the notification point: a CE-marked data packet makes it send a CNP in class cnp_priority, at most one per cnp_interval_ns. The sender's pacer draws credit from R_C on every tick; the flow has no acknowledgments and no retransmission.

DCQCN sources emit ECN-capable data. Use the ECN ramp (switch.ecn or switch.ecn_by_rate) to create CE feedback. Lowered DCQCN, including its controller and the notification point, runs on Scalar, CPU, Metal, and CUDA, with any cnp_priority.

RoCE queue pairs

A RoCE queue pair is a reliable DCQCN flow. It uses the DCQCN controller and pacer above and adds Go-back-N reliability: a PSN on every data packet, cumulative ACKs, NACKs, and a retransmission timeout. Its congestion feedback is the ECN echo its ACKs and NACKs carry; a queue pair sends no CNP. A queue pair completes when its last byte is acknowledged. Queue pairs are an exact-lowering feature; the legacy simulator does not run them.

[[flow]]
flow_type = "RoCE"
priority = 3                      # data class; lossless when link.pfc.xoff[3] is nonzero
graph = [[0, 1]]

[flow.traffic]
initial_delay = 0.0
size = 2_000_000                  # required: a queue pair is byte-bounded
arr_dist = { type = "Uniform", low = 1, high = 1 }
pkt_size_dist = { type = "DiscreteUniform", low = 1000, high = 1000 }   # the MTU

[flow.traffic.dcqcn]              # the controller: the DCQCN keys above, without cnp_interval_ns and cnp_priority
max_rate_gbps = 100.0
ai_rate_gbps = 0.02
hai_rate_gbps = 0.2
rp_timer_ns = 300000
pacing_interval_ns = 80

[flow.traffic.roce]
retransmit_timeout_ns = 1_000_000
window_bytes = 250_000

The PSN is the byte offset of a packet's first byte. Every packet is pkt_size_dist bytes except a short last one, so a retransmission resends exactly the bytes of the original. The pacer sends at most one packet per pacing_interval_ns tick, so the tick must be short enough for the line rate: at 100 Gbps with 1,000-byte packets, at most 80 ns.

On a NACK or a timeout, the sender rewinds to the first unacknowledged byte and resends everything from there (Go-back-N). Retransmissions are paced like first transmissions.

ECN echo. Each ACK or NACK echoes the CE bit of the data packet whose arrival produced it. With ack_every_packets above 1, only the packet that produces the ACK is echoed; a NACK that the NACK interval suppresses echoes nothing. The sender's controller takes one feedback per ACK or NACK whose echo is set, until the pair completes. ACKs and NACKs ride feedback_priority; a class other than the data class is not held back by a pause of the data class.

Window. With window_bytes above 0, the window is w = window_bytes, or with variable_window the larger of 1 and floor(window_bytes * R_C / R_max), with R_C the controller's current rate and R_max its max_rate_gbps, both in bits per second; the product is exact and does not wrap. A pacing tick that has a packet to send and finds next_psn - snd_una >= w sends nothing, adds no credit and parks the pacer, as a tick in a paused class does. Only a fresh ACK (one that moves snd_una), a NACK at or above snd_una (the usual NACK: it rewinds next_psn to snd_una without moving snd_una), or a timeout restarts it; a rate increase or a host-link RESUME does not. SimAI sets m_win to a bandwidth-delay product, (2 * propagation delay + one packet's transmission time) * line rate in bytes, of the pair's path or the largest over all host pairs; Days AGO takes the value from window_bytes.

With the timeout off, only a NACK repairs a loss. A NACK needs a later packet of the same queue pair to arrive out of order. If the last packet is lost, or the last ACK, or a resent packet is lost again when nothing follows it, the queue pair stalls for the rest of the run. It never completes, and the run does not report an error. This is the NACK-only Go-back-N of the HPCC ns-3 model; keep the timeout on unless you need that behavior.

With [link.pfc] host_links = true, a switch can pause a queue pair's host. The queue pair then stops sending in its data class until the resume; see host links.

Exact lowering refuses [flow.traffic.roce] on other flow types, [flow.traffic.tcp] on a queue pair, cnp_priority on a queue pair (use feedback_priority), cnp_interval_ns on a queue pair, variable_window without window_bytes, duration-bounded queue pairs, and an MTU above 4294967295 bytes. A queue pair with cnp_interval_ns fails with:

unsupported `cnp_interval_ns` on RoCE traffic: a queue pair's receiver echoes ECN on its ACKs and NACKs and sends no CNP (P16)

Queue pairs also carry collective stages (see Collectives). Queue pairs and collective stages run on Scalar, CPU, Metal, and CUDA.

Days AGO field limits

Exact lowering rejects non-empty dependencies, explicit flow IDs, per-flow routing, and explicit paths. It derives scenario-local IDs and deterministic routes. Root [routing] remains available for ShortestPath and FatTreeEcmp.

An unsupported combination fails during lowering. The compiler does not silently drop the field.