Internals

Scope: Days AGO. This page covers the Days AGO executor, which compiles a scenario into a SimulationImage and runs it with the Scalar, CPU, Metal, or CUDA backend. The legacy Nexosim runtime is documented separately under Legacy Days because its model, time, and scheduling APIs do not apply to Days AGO.

Executor map

The Days AGO path has four stages:

1. src/scenario/compile.rs lowers supported configuration into a SimulationImage with stable semantic IDs.
2. executor/src/validate.rs checks image invariants and rejects mechanisms that the selected backend cannot execute.
3. One of scalar.rs, safe_horizon.rs, cpu.rs, metal.rs, or cuda.rs applies the shared transition semantics.
4. The backend returns a normalized RunResult containing canonical final state and observations.

Days AGO is an LP executor, not an actor runtime. A logical process (LP) owns one host or one switch egress queue. Events have fixed payloads and a canonical key ordered by (time_ns, phase, origin_node, origin_seq); there are no callbacks in the event image. See executor/src/image.rs and executor/src/event.rs.

Scalar and parallel execution

Round execution starts at the minimum pending LP frontier. Its exclusive horizon is the smaller of the run boundary and frontier + minimum_channel_lookahead. Every active LP drains events strictly before that horizon. Cross-LP outboxes are then sorted by the canonical event key and merged at the barrier before the next frontier is chosen.

This rule is implemented in executor/src/safe_horizon.rs. A zero-delay channel is valid for global scalar execution but is rejected by scalar-round, CPU, Metal, and CUDA execution because it provides no positive safe horizon.

Device planning and bounded arenas

Metal and CUDA plan storage before launching kernels. executor/src/device_sizing.rs derives arena layouts from the compiled image, configured caps and floors, and the run boundary. Record arenas are striped by the entity that owns them (for example an LP, channel stream, TCP receiver class, or flow), so one busy entity does not force every row to use the same capacity.

Three rules bound the arenas of a stage image (executor/src/stage_sizing.rs). First, a host's stage flows are charged by its width, the most stages that can be active there at once, rather than by their count: the stages of one operation at a host form chains below its roots, an operation that follows another there starts only once that one completed, and a greedy chain cover over each issue stream sums the widest operation of each chain. Each active stage can leave at most one completed generation of stale resends behind it, so the remote-staging, event-heap and outbox charges of a host's stages are min(sum of the per-flow bounds, 2 * width * largest bound). An image without a join (every P14 and P15 image) is a forest and keeps the earlier leaf count, so its plan is unchanged byte for byte. Second, a RoCE queue pair with a window and no retransmission timeout has at most ceil(window_bytes / mtu) + 1 data packets unacknowledged, and as many feedback packets; that bounds its host queues and its remote staging on every link of its route. Every other queue pair and TCP stage keeps the lookahead-horizon bound. Third, a stage notify adds, at its source, a timer and an outbox slot while unfired, one event on its lane's stream, and its arrival in the target's heap when streams are off.

Full-observation logs (departures, arrivals and observed packets) are sized for the requested run, not the whole scenario: a generator with a pacing interval emits at most one packet per tick between its first tick and the run's end, feedback at most one packet per data packet, and each count is the smaller of that bound and the whole-run count (observation_packet_counts in executor/src/planner_capacity.rs). The bound grows linearly with the cutoff, because it counts every queue pair as active for the whole window. A short bound causes a typed capacity retry, never a different result.

Before allocating, each device compares the plan's bytes with its memory limit and refuses a larger plan with DeviceMemoryExceeded { planned_bytes, limit_bytes }. On Metal the limit is the device's recommended working set (recommendedMaxWorkingSetSize); above it Metal fails a command buffer out of memory, and past physical memory it completes command buffers whose writes do not land. On CUDA the limit is the free device memory after a stream sync on a discrete GPU, and total - max(8 GiB, total / 16) under managed provisioning (an integrated GPU sharing system memory, such as the GB10: 122,004,221,952 bytes of 130,594,156,544), because there free moves with the host's page cache. The optional max_device_bytes field of MetalConfig and CudaConfig only lowers the limit. The check is one snapshot: memory that other processes allocate afterwards is not seen, so a plan under the limit can still run out of memory on a shared host.

Device capacity is refuse or run. If a kernel reports that an arena is too small, that attempt produces no result. The host records the fault, raises the relevant plane, stream, class, or flow floor from the reported demand, rebuilds the plan, and retries up to the configured limit. Caps remain hard limits. The retry types and growth rules live in executor/src/device_capacity.rs.

The device PFC row of a host with egress pause state (host-link PFC) lists the host's queue pairs and, per class, a parked bitset of ceil(Q / 64) words for Q queue pairs (executor/src/device_pfc.rs). A paused pacing tick sets the pair's bit in its class; the RESUME that unpauses the class walks only the set bits and clears them, so a RESUME examines the pairs it restarts rather than the host's whole list. The bits are derived state, checked against Scalar's parked lists at readback.

A successful device run returns a CapacityWarmStart. Despite the name, this is only a sizing hint for a later run of the same image: it seeds the capacity floors that a previous retry chain learned. It does not contain simulated state, resume an interrupted run, or represent a compiled-pipeline cache. Passing the hint to run_with_observations_warm_started can avoid repeating known failed sizing attempts without changing simulation semantics.

Compact readback

Large striped record arenas are not copied back at their full planned capacity. Metal and CUDA first read the arena metadata, build a unique plan from each entity's offset, capacity, ring head, and live count, and run a gather kernel into a compact buffer. The destination offsets follow the same entity order and ring-slot arithmetic as the canonical decoder.

Compaction is lossless: it changes transfer size, not record order or decoded bytes. Invalid metadata produces a typed refusal instead of truncated output. The shared planning and validation logic is in executor/src/device_compaction.rs; each device backend owns its gather and readback implementation.

Device event records and the drain path

Device future-event storage does not pay one fixed record size for every event kind. The fallback record is 14 u64 words, and each proven-monotone stream class stores a compact per-kind encoding: channel events in 11 words, generator events in 10, service events in 5, and remote staging records in 12. On the benchmark mix this roughly halves record traffic. The encodings are lossless: every slim record decodes to the same bytes as its wide form, and a test checks the round trip. The shared encoder is in executor/src/device_event_record.rs, with matching device-side codecs in both kernel files.

The drain kernels themselves carry three measured optimizations, selected per backend rather than assumed portable:

• a restrict-qualified kernel ABI with explicit launch bounds (CUDA);
• a same-time continuation fast path, guarded by its own conformance oracle (same_time_continuations in the device record) so the fast and general paths are provably interchangeable (CUDA);
• worklist compaction for the round's active-LP set (Metal and CUDA).

Metal keeps only worklist compaction. The other two optimizations, faster on NVIDIA hardware, measured as regressions on Apple GPUs and are compiled out there. The engine does not choose the parallel launch width either: it is a per-run parameter (--round-threads-per-block on the days CLI), and the days_protocol record prints the value used, so a reported time names the geometry that produced it.

Backend capability validation

validate(image, backend) runs before execution. Compilation may accept a mechanism that a particular backend then rejects deliberately.

Metal and CUDA also require their feature, platform, and device runtime to be available. Device runs return the canonical state and measurement surface, but diagnostic-only observation planes remain a scalar/CPU facility. See executor/src/validate.rs and executor/src/scalar.rs for the authoritative checks and result contract.