Reproducibility¶

Floating-point reproducibility is a correctness property of the collective offload, not only a numerical-analysis detail. In an in-network collective, the effective reduction order can be affected by packet arrival time, VOQ arbitration, backpressure, egress scheduling, tree shape, and rank-to-port placement. This is a problem for IEEE-style floating-point addition: floating-point addition is rounded after each operation and is therefore not associative.¹

MiniSharp studies this problem at the packet-processing boundary. The goal is not to make a whole distributed application deterministic by itself. Local computation, tensor partitioning, compiler choices, GPU kernels, and host-side reduction fragments may still introduce nondeterminism before packets reach the switch. The narrower question is:

This matters for debugging, regression testing, numerical validation, checkpoint/restart workflows, and scientific or safety-critical workloads where bitwise differences make failures hard to reproduce. It also matters operationally: in a real cluster, the same job may be launched with the same number of ranks but mapped onto different nodes, leaf ports, or aggregation paths.

Why this is a switch datapath problem¶

In a host-side collective library, the reduction order is often an implementation detail of the selected collective algorithm. In a switch datapath, the problem becomes more explicit. A streaming aggregation engine sees packets as timed events and must decide whether to consume operands in arrival order, in a rank-indexed order, or in an order-independent representation.

From a switching perspective, the relevant hazards are:

• arrival-order dependence, where the first packet to reach an accumulator slot becomes the first addend;
• VOQ and egress arbitration dependence, where unrelated congestion changes the timing of reduction packets;
• tree-shape dependence, where different aggregation trees produce different parenthesizations of the same sum;
• fragmentation dependence, where vector chunks or packet beats are reduced in different partial orders;
• epoch-completion dependence, where a reduction result is emitted before all expected participants have contributed.

For this reason, MiniSharp treats floating-point reproducibility as part of the data-plane contract of a collective offload. The reducer is not just a packet parser plus an FP adder; it is an epoch-indexed aggregation engine with rank metadata, contribution tracking, accumulator state, and a finalization rule.

flowchart LR
    S{Reduction mode}

    S -->|Baseline| B[Arrival-order FP32 add]
    B --> O1[Fast but order-dependent]

    S -->|Deterministic FP32| C[Wait for contribution bitmap]
    C --> D[Rank-ordered FP32 tree]
    D --> O2[Bitwise stable for fixed logical order]

    S -->|Superaccumulator| E[Decode sign, exponent, significand]
    E --> F[Exact fixed-point deposit]
    F --> G[Normalize and round once]
    G --> O3[Order-independent finite sum]

Candidate strategies¶

MiniSharp exposes reproducibility as a design space rather than as a single hard-coded arithmetic choice.

The important distinction is that deterministic scheduling fixes the order, while a superaccumulator removes the order as a numerical variable. The first approach is cheaper and useful as a baseline. The second is more expensive, but it better matches the semantics expected from an order-independent collective reduction.

Why try superaccumulators now?¶

The motivation is stronger now than it would have been a decade ago.

First, in-network collective offload is no longer an exotic idea. NVIDIA/Mellanox SHARP has made aggregation nodes, aggregation trees, and collective offload part of the vocabulary of production HPC and AI fabrics.² SHARP demonstrates that moving reduction work into the network can reduce redundant endpoint traffic and free CPU/GPU resources from communication processing.³

Second, distributed AI and HPC workloads increasingly operate at scales where collective communication is a first-order performance limit. Systems such as SwitchML showed that aggregating model updates in the network can improve end-to-end distributed training performance by reducing exchanged data and bypassing some host-side communication overheads.⁴

Third, modern accelerators already rely on mixed precision. NVIDIA H100-style FP8 uses distinct E4M3 and E5M2 formats, typically choosing E4M3 for precision-sensitive forward values and E5M2 for higher-dynamic-range gradients.⁵ This makes accumulation semantics more important, not less: low-precision payloads are often acceptable only because accumulation is performed in a wider or more stable format.

Fourth, conventional programmable-switch datapaths were designed primarily for packet processing, not for application-level floating-point arithmetic. Work on FPISA explicitly identifies floating-point data in distributed training and query processing as a case where existing switch hardware needs expensive workarounds or architectural support.⁶ MiniSharp is therefore positioned as an RTL research vehicle: instead of emulating floating point inside a restricted P4/RMT pipeline, it studies what a purpose-built FPGA datapath would cost.

Finally, exact or reproducible accumulation maps naturally to hardware research questions: accumulator width, carry-save versus carry-propagate organization, banked accumulator files, BRAM/URAM pressure, initiation interval, Fmax, backpressure behavior, and final rounding latency. These are precisely the trade-offs that a compact RTL prototype can expose.

Superaccumulator semantics¶

For a finite binary floating-point input, the reducer decodes:

where \(p\) is the significand precision including the hidden bit, \(M\) is the integer significand, and \(e\) is the unbiased exponent. Subnormal values are also exact multiples of the smallest subnormal unit:

A Kulisch-style scalar-sum accumulator chooses this smallest unit as its fixed-point least significant bit. Each incoming floating-point operand is deposited into the corresponding fixed-point position. Since the intermediate representation is exact integer state, adding two deposits is associative and commutative up to the capacity of the accumulator. The final floating-point result is produced by normalizing the exact integer sum and rounding once according to the selected rounding mode.

This gives a clean separation between network timing and arithmetic semantics:

Special values must be handled outside the fixed-point datapath. NaNs, infinities, signed zeros, invalid operations, and overflow of the final target format should be tracked with deterministic sideband flags and a documented canonicalization policy. The finite superaccumulator should only receive finite numeric deposits.

Accumulator sizing for scalar reductions¶

The accumulator width depends on the operation. For MiniSharp scalar reductions,

the relevant bound is not the dot-product bound. Let:

• \(R\) be the maximum number of participants contributing to one reduction element;
• \(p\) be the significand precision including the hidden bit;
• \(E_{\min}\) be the minimum normal exponent;
• \(E_{\max}\) be the maximum normal exponent.

For exact finite scalar summation including subnormals:

and the signed two's-complement accumulator width is:

The \(+\lceil \log_2 R\rceil\) term is the carry headroom for summing \(R\) same-sign maximum-magnitude operands. The final \(+1\) is the sign bit.

Therefore, a single scalar-reduction accumulator covering FP8, FP16, BF16, and FP32 is dominated by FP32/BF16 exponent range:

For example, with \(R=256\) ranks, an exact signed FP32 scalar-sum accumulator requires:

bits per reduction lane, before adding implementation metadata such as valid bits, exception flags, epoch tags, contribution bitmaps, or bank-selection state.

Why not only preserve the FP32 order?¶

Preserving a strict floating-point order is a valid baseline, but it is not equivalent to exact accumulation.

A deterministic FP32 reducer still rounds after every addition. It can make the result repeatable, but the selected order becomes part of the numerical algorithm. A different rank count, different chunking, or different local partial sums can still change the result. In contrast, an exact finite accumulator makes the reduction independent of the order in which finite operands are deposited; only the final rounding step remains.

Preserving order can also fight the purpose of in-network computing. If reproducibility is achieved by gathering operands to a root, serializing the reduction, or forwarding intermediate FP32 accumulators across endpoints, the design may reintroduce communication and synchronization overhead that INC was meant to avoid. A local superaccumulator is attractive because it keeps the reproducibility cost inside the aggregation node rather than increasing the payload format on the wire.

MiniSharp therefore compares both paths:

1. Deterministic FP32 mode, useful as a lower-area baseline and for studying scheduling determinism.
2. Superaccumulator mode, useful for studying order-independent finite summation and the hardware cost of exact accumulation.
3. Wait-all epoch mode, useful as a control policy to prevent early emission and arrival-order dependence, but not an arithmetic solution by itself.

Expected outcome¶

The expected contribution is not the claim that superaccumulators are universally better. The expected contribution is an implementation-level measurement of the trade-off:

For MiniSharp, this is the central research point: if a switch-like datapath already contains parser state, VOQs, shared packet memory, arbitration, backpressure, and aggregation slots, then the open question is whether exact or near-exact floating-point accumulation can be integrated without destroying line-rate behavior.