Floating-Point Reproducibility¶

Floating-point reproducibility is a correctness property of the collective offload, not only a numerical-analysis detail. In an in-network collective, the effective reduction order can be affected by packet arrival time, VOQ arbitration, backpressure, egress scheduling, tree shape, and rank-to-port placement. This is a problem for IEEE-style floating-point addition: floating-point addition is rounded after each operation and is therefore not associative.

PRISM studies this problem at the packet-processing boundary. The goal is not to make a whole distributed application deterministic by itself. Local computation, tensor partitioning, compiler choices, GPU kernels, and host-side reduction fragments may still introduce nondeterminism before packets reach the switch. The narrower question is:

This matters for debugging, regression testing, numerical validation, checkpoint/restart workflows, and scientific or safety-critical workloads where bitwise differences make failures hard to reproduce. It also matters operationally: in a real cluster, the same job may be launched with the same number of ranks but mapped onto different nodes, leaf ports, or aggregation paths.

Candidate strategies¶

PRISM exposes reproducibility as a design space rather than as a single hard-coded arithmetic choice.

flowchart LR
    S{Accumulator profile}

    S -->|FP32_ARRIVAL| B[Arrival-order FP32 add]
    B --> O1[Fast but order-dependent]

    S -->|FP32_ORDERED| C[Wait only for next Contributor ID]
    C --> D[Streamed FP32 fold by Contributor ID]
    D --> O2[Bitwise stable for fixed logical order]

    S -->|KULISCH_FP32| E[Decode sign, exponent, significand]
    E --> F[Exact fixed-point deposit]
    F --> G[Normalize and round once]
    G --> O3[Order-independent finite sum]

The important distinction is that deterministic scheduling fixes the order, while a superaccumulator removes the order as a numerical variable. The first approach is cheaper and useful as a baseline. The second is more expensive, but it better matches the semantics expected from an order-independent collective reduction.

Why try superaccumulators now?¶

The motivation is stronger now than it would have been a decade ago.

First, in-network collective offload is no longer an exotic idea. NVIDIA/Mellanox SHARP¹ has made aggregation nodes, aggregation trees, and collective offload part of the vocabulary of production HPC and AI fabrics. SHARP² demonstrates that moving reduction work into the network can reduce redundant endpoint traffic and free CPU/GPU resources from communication processing.

Second, distributed AI and HPC workloads increasingly operate at scales where collective communication is a first-order performance limit. Systems such as SwitchML³ showed that aggregating model updates in the network can improve end-to-end distributed training performance by reducing exchanged data and bypassing some host-side communication overheads.

Third, modern accelerators already rely on mixed precision. NVIDIA H100-style FP8⁴ uses distinct E4M3 and E5M2 formats, typically choosing E4M3 for precision-sensitive forward values and E5M2 for higher-dynamic-range gradients. This makes accumulation semantics more important, not less: low-precision payloads are often acceptable only because accumulation is performed in a wider or more stable format.

Fourth, conventional programmable-switch datapaths were designed primarily for packet processing, not for application-level floating-point arithmetic. Work on FPISA⁵ explicitly identifies floating-point data in distributed training and query processing as a case where existing switch hardware needs expensive workarounds or architectural support. PRISM is therefore positioned as an RTL research vehicle: instead of emulating floating point inside a restricted P4/RMT pipeline, it studies what a purpose-built FPGA datapath would cost.

Finally, exact or reproducible accumulation maps naturally to hardware research questions: accumulator width, carry-save versus carry-propagate organization, banked accumulator files, BRAM/URAM pressure, initiation interval, Fmax, backpressure behavior, and final rounding latency. These are precisely the trade-offs that a compact RTL prototype can expose.

Superaccumulator semantics¶

For a finite binary floating-point input, the reducer decodes:

where \(p\) is the significand precision including the hidden bit, \(M\) is the integer significand, and \(e\) is the unbiased exponent. Subnormal values are also exact multiples of the smallest subnormal unit:

A Kulisch-style scalar-sum accumulator chooses this smallest unit as its fixed-point least significant bit. Each incoming floating-point operand is deposited into the corresponding fixed-point position. Since the intermediate representation is exact integer state, adding two deposits is associative and commutative up to the capacity of the accumulator. The final floating-point result is produced by normalizing the exact integer sum and rounding once according to the selected rounding mode.

This gives a clean separation between network timing and arithmetic semantics:

Special values must be handled outside the fixed-point datapath. NaNs, infinities, signed zeros, invalid operations, and overflow of the final target format should be tracked with deterministic sideband flags and a documented canonicalization policy. The finite superaccumulator should only receive finite numeric deposits.

Accumulator sizing for scalar reductions¶

The accumulator width depends on the operation. For PRISM scalar reductions,

the relevant bound is not the dot-product bound. Let:

• \(R\) be the maximum number of participants contributing to one reduction element;
• \(p\) be the significand precision including the hidden bit;
• \(E_{\min}\) be the minimum normal exponent;
• \(E_{\max}\) be the maximum normal exponent.

For exact finite scalar summation including subnormals:

and the signed two's-complement accumulator width is:

The \(+\lceil \log_2 R\rceil\) term is the carry headroom for summing \(R\) same-sign maximum-magnitude operands. The final \(+1\) is the sign bit.

Therefore, a single scalar-reduction accumulator covering FP8, FP16, BF16, and FP32 is dominated by FP32/BF16 exponent range:

For example, with \(R=256\) ranks, an exact signed FP32 scalar-sum accumulator requires:

bits per reduction lane, before adding implementation metadata such as valid bits, exception flags, operation tags, contribution bitmaps, or bank-selection state.

Parallelism and line rate¶

For \(b_d\) bits per element, \(L_d\) lanes, clock \(f\), and initiation interval \(II_d\):

At 250 MHz and \(II=1\), FP32 needs 13 active lanes for 100 Gbit/s, FP16/BF16 need 25, and INT8 needs 50. The current core keeps TILE_ELEMENTS=16 and forms an issue group from 1, 2, or 4 tiles. It therefore accepts 16 FP32, 32 FP16/BF16, or 64 INT8 elements per cycle, equal to 128 Gbit/s in every case. Output conversion can impose a second limit: 64 INT8 inputs converted to FP32 produce 2048 result bits per cycle.

Why not only preserve the FP32 order?¶

Preserving a strict floating-point order is a valid baseline, but it is not equivalent to exact accumulation.

A deterministic FP32 reducer still rounds after every addition. It can make the result repeatable, but the selected order becomes part of the numerical algorithm. A different rank count, different chunking, or different local partial sums can still change the result. In contrast, an exact finite accumulator makes the reduction independent of the order in which finite operands are deposited; only the final rounding step remains.

Preserving order can also fight the purpose of in-network computing. If reproducibility is achieved by gathering operands to a root, serializing the reduction, or forwarding intermediate FP32 accumulators across endpoints, the design may reintroduce communication and synchronization overhead that INC was meant to avoid. A local superaccumulator is attractive because it keeps the reproducibility cost inside the aggregation node rather than increasing the payload format on the wire.

PRISM therefore compares three profiles:

1. FP32_ARRIVAL, the smallest arrival-ordered baseline.
2. FP32_ORDERED, which defines ascending Contributor-ID order while still streaming ready ranges.
3. KULISCH_FP32, the exact-accumulator profile for studying order-independent finite summation and its hardware cost.

Each profile defines both its state representation and contributor-update rule. The target FP32_ORDERED semantics uses a per-range reorder buffer. The current two-contributor RTL folds the ordered pair directly; with more contributors it waits for the complete range before applying the ordered fold. KULISCH_FP32 uses the one accumulator width configured in the bitstream. Admission derives the required width from the Reduction Operator, input format, and Maximum Reduction Terms and reports ACCUMULATION_INEXACT if that requirement exceeds the configured width. Protocol 1.0 uses the scalar SUM bound; a future dot-product operation requires the wider product bound.

Expected outcome¶

The expected contribution is not the claim that superaccumulators are universally better. The expected contribution is an implementation-level measurement of the trade-off:

For PRISM, this is the central research point: if a switch-like datapath already contains parser state, VOQs, shared packet memory, arbitration, backpressure, and aggregation slots, then the open question is whether exact or near-exact floating-point accumulation can be integrated without destroying line-rate behavior.