Protocol¶

PRISM is a Layer-2 protocol for FPGA-based System Area Networks (SANs), carrying collective operations directly over Ethernet frames. Its target architecture combines normal forwarding with local collective processing and bounded hop-by-hop reliability. The current RTL implements one SINGLE collective node without forwarding or replay.

The implemented node supports two MPI-style semantics:

• REDUCE: several ranks contribute, but one selected rank receives the sum;
• ALLREDUCE: the same sum is returned to every participating rank.

A rank is one participating process or accelerator endpoint. In a reduction tree, leaf switches receive rank contributions, optional internal switches combine partial sums, and one aggregation root produces the final result.

The packet model has four local classes: in-band CONTROL, contributions and partials in DATA, results in RES, and reliability feedback in PROGRESS. The current node implements CONTROL and endpoint DATA reception plus generated RES; the wider registry is not itself an RTL support claim.

PRISM is a research FPGA datapath, not a complete production Ethernet switch. The protocol therefore prioritizes a clear packet model, reproducible tests, and implementable RTL over a broad management or security stack.

This page moves from frame ownership, identity, chunking, and setup to packet layouts. It then shows the implemented single-node flow and the complete multi-switch storyboard before collecting the Protocol 1.0 registries.

Frame ownership and forwarding¶

PRISM classification proceeds from the outer Ethernet frame inward:

1. Parse destination MAC, source MAC, and EtherType.
2. If EtherType is not 0x88B5, apply the normal Layer-2 forwarding policy.
3. If the frame is PRISM but its destination MAC is not local, forward it unchanged when policy permits.
4. Decode the PRISM opcode only for a locally addressed PRISM frame.

flowchart LR
    RX["Ethernet frame"] --> ET{"PRISM EtherType?"}
    ET -->|No| L2["Layer-2 forwarding"]
    ET -->|Yes| OWN{"Destination MAC local?"}
    OWN -->|No| L2
    OWN -->|Yes| OP{"Opcode"}
    OP -->|CONTROL| CTRL["Control path"]
    OP -->|COL_DATA| DATA["Collective engine"]
    OP -->|COL_RES| UNSUP["Unsupported receive class"]
    OP -->|COL_PROGRESS| UNSUP
    OP -->|Unknown| BAD["Invalid packet"]

The destination MAC and opcode answer different questions:

• the MAC selects the next node or local owner;
• the opcode selects what that owner does with the packet.

The current collective-node RTL drains both forwarding branches. They remain in the protocol and target architecture so that local collective handling can be integrated into a complete switch without changing the wire model.

Common frame format¶

The PRISM header is carried directly inside an Ethernet frame. The Ethernet header selects the next destination; the PRISM header identifies the packet class and the collective operation. The normative byte-order rule is stated with the packet layouts below.

For a standard PM_OP_COL_DATA or PM_OP_COL_RES frame, the first 512-bit AXI4-Stream beat is:

%%{init: {"packet": {"bitsPerRow": 128, "bitWidth": 8, "rowHeight": 78, "showBits": true}} }%%
packet
    +48: "Dst MAC (48b)"
    +48: "Src MAC (48b)"
    +16: "EtherType (16b)"
    +16: "PRISM Control (16b)"
    +192: "Operation Header (192b)"
    +192: "First 24 payload bytes (192b)"

Subsequent full beats can contain up to 64 bytes (512 bits) of numeric data.

TKEEP identifies valid bytes in the final beat and TLAST marks the physical end of the frame. They validate the packet boundary; the operation configuration still determines the exact logical vector length.

The common 16-bit control word is:

%%{init: {"packet": {"bitsPerRow": 16, "bitWidth": 28, "rowHeight": 78, "showBits": true}} }%%
packet
    +4: "Major (4b)"
    +4: "Minor (4b)"
    +8: "Opcode (8b)"

Protocol 1.0 uses Major=1 and Minor=0. A different major version is incompatible. The opcode tells the local parser how to interpret the following header. The host request and StartGrant are API events outside this wire ABI, so the host-controlled startup model does not change the Protocol 1.0 layout. This pre-release Protocol 1.0 document defines the current wire layout.

Collective identifiers¶

• Job / AM session: key (Job ID:64, switch-local Session Cookie:64); owns configuration and provides the recovery domain.
• Collective group: key (Job ID:64, Group ID:16, Group Generation:16); identifies one reusable contributor tree.
• Collective operation: key (group key, Operation ID:16, Operation Generation:32); identifies one execution of that collective.
• Contributor: key (operation key, Contributor ID:16); identifies one rank at a leaf or one child subtree at an internal/root node.
• Result path: key (operation key, Result Path ID:16); identifies one result replica on the current direct link.
• Chunk: key (direct stream, Chunk Index:24); identifies one DATA or RES packet position using that stream's wire format.
• Result-range route: key (operation key, chunk range); maps a disjoint output range to one Replication Group ID.
• Replication group / result destination set: key (group key, Replication Group ID:16); collects one or more direct result destinations.
• Result destination: key (replication-group key, Result Path ID:16); selects one egress port, destination MAC, and next-hop type.

• Contributor binding: Contributor IDs are dense and bound to an expected ingress port and source MAC; each switch stores only its local group records.

Node roles and the single-switch case¶

• LEAF: directly attached ranks; reduces local contributions and emits one partial DATA stream to its required parent.
• INTERNAL: child subtrees; reduces their partials and emits one new partial DATA stream to its required parent.
• ROOT: child subtrees and no parent; completes reduction and starts result distribution.
• SINGLE: directly attached ranks and no parent; completes reduction locally and starts result distribution.

Node Role belongs to the logical collective tree; it does not alter ordinary Ethernet forwarding.

Chunking model¶

A PRISM chunk is one maximally packed, whole-element DATA or RES packet. Chunk Index is local to a direct packet stream rather than a universal element interval shared by every stage of the tree. Endpoint DATA, switch partial DATA, and RES therefore derive independent geometries from the same Element Count.

The standard Ethernet payload is 1500 bytes. The 2-byte PRISM control word and 24-byte DATA/RES header leave:

For a wire format f with a positive integral byte width:

and

No element is split across packets. All chunks except the final one contain the maximum whole-element count for their wire format. The payload length is elements_in_chunk(i,f) × bytes_per_element(f); the unused one to three bytes that can remain after whole-element packing are not vector data.

DATA and RES carry no per-packet format tag. Their geometry and encoding are selected by the installed Operation descriptor:

• endpoint-to-leaf DATA uses Input Wire Format;
• switch-to-switch DATA uses Partial Wire Format;
• RES uses Output Wire Format.

For example, a 1000-element BF16 input stream contains chunks of 737 and 263 elements. An FP32 partial or result stream for the same operation contains 368, 368, and 264 elements. The switch maps both streams to the same global element positions; it does not preserve their numeric Chunk Index across conversion. Any future exact-partial wire format follows the same whole-element packing rule; its element width would be assigned by a later protocol version. Protocol 1.0 requires one tree-wide arithmetic configuration so every receiver can derive the correct geometry from its configured role and Contributor binding.

Chunks may arrive in a different order inside the admitted receive window. The tuple (packet class, peer, Chunk Index) identifies the direct transfer, while the derived First Element selects format-independent accumulator state.

In-band setup¶

The Host Coordinator gathers ranks and their request; the Aggregation Manager (AM) converts it into per-switch configuration using topology, capabilities, and available resources. AM commands are in band: they use PM_OP_CONTROL Ethernet packets on the collective ports, then follow a separate control path and never enter the numeric accumulator.

Host-controlled start¶

Ranks wait at a host-side barrier until every target switch reports ACTIVE. The AM then returns a StartGrant; the runtime's reliable host channel delivers it to every rank and later reports DONE or ERROR.

The rank-specific grant carries the operation identity, Contributor ID, destination MACs, formats, element count, payload limit, and admitted window. Ranks derive chunk geometry from those values; REDUCE also names the result rank. If setup fails anywhere, the AM releases partial state and no rank receives a grant.

The Host Coordinator and StartGrant are runtime concepts rather than PRISM opcodes. A coordinator co-located with a rank still contributes normally.

Target-specific delivery¶

The AM sends each target its own PM_OP_CONTROL transaction containing only that switch's records; a transaction may span several frames. A root consumes locally addressed commands and forwards commands for other switches unchanged; it never derives or rewrites their configuration.

From discovery to active operation¶

Protocol 1.0 separates reusable topology from one invocation:

Snapshots reserve nothing; GROUP_STAGE and OP_PREPARE do. The AM validates the complete tree and common arithmetic configuration before commit, aborts partial setup on failure, and returns a grant only after every group is INSTALLED and every operation is ACTIVE. Commands are idempotent; conflicting Transaction-ID reuse returns TXN_CONFLICT.

Failure, cancellation, and recovery¶

A changed Device Boot ID, unreachable target, or AM restart fails the collective. The AM best-effort releases surviving state, leases reclaim the rest, and later use installs a fresh tree. A runtime may retry from the original inputs with another algorithm; partial PRISM output is never successful output.

Packet layouts¶

The four opcodes are not four variants of the same operation. CONTROL configures and queries switches; COL_DATA carries contributions toward the root; COL_RES distributes results; and COL_PROGRESS returns acknowledgments and flow information to the adjacent sender. The common control word selects one of these headers. The diagrams show the encoding; the following tables explain the function of each field.

Unless a field says otherwise:

• fields appear in transmission order;
• Ethernet fields, PRISM headers, identifiers, lengths, flags, TLV headers, and scalar TLV integers use network byte order: the most significant byte is transmitted first;
• numeric DATA and RES elements use the operation's wire format and are little endian: the least significant byte of each encoded element is transmitted first;
• reserved fields and unused flag bits are transmitted as zero and rejected when received non-zero;
• widths are in bits and offsets are in bytes from the start of the containing header or TLV value.

An ACK is an acknowledgment: it tells the previous sender that a packet was accepted safely. A direct transfer, also called one network hop, is one link between two neighboring nodes; for example rank0 -> L0 or L0 -> S0.

PM_OP_CONTROL¶

PM_OP_CONTROL has this fixed 24-byte header after the common control word:

%%{init: {"packet": {"bitsPerRow": 64, "bitWidth": 12, "rowHeight": 78, "showBits": true}} }%%
packet
    +64: "Session Cookie (64b)"
    +64: "Transaction / Event ID (64b)"
    +16: "Control Command (16b)"
    +8: "Flags (8b)"
    +8: "Status (8b)"
    +16: "Fragment Index (16b)"
    +16: "Fragment Count (16b)"

CONTROL is the envelope of the in-band management protocol. The AM uses it to open sessions, install groups, prepare operations, and query state or resources; switches use it for responses and events. The header identifies the exchange and makes it reassemblable, while the TLV payload carries command-specific parameters.

%%{init: {"packet": {"bitsPerRow": 64, "bitWidth": 16, "rowHeight": 78, "showBits": true}} }%%
packet
    +16: "Type (16b)"
    +16: "Flags (16b)"
    +32: "Value Length (32b)"

DATA, RES, and PROGRESS belong to one already-configured operation. The first two carry numeric elements; PROGRESS carries no vector data and instead controls reliability on one direct link. Job, Group, and Operation fields occur in all three so that a valid packet cannot be applied to the wrong context.

PM_OP_COL_DATA¶

The 24-byte DATA header is followed by numeric vector elements in Little Endian format:

%%{init: {"packet": {"bitsPerRow": 96, "bitWidth": 9, "rowHeight": 78, "showBits": true}} }%%
packet
    +64: "Job ID (64b)"
    +16: "Group ID (16b)"
    +16: "Group Generation (16b)"
    +16: "Operation ID (16b)"
    +32: "Operation Generation (32b)"
    +16: "Contributor ID (16b)"
    +24: "Chunk Index (24b)"
    +8: "Flags (8b)"

The Contributor ID names a direct local source. At a leaf it normally names a rank; at an internal node it names a child subtree. When a switch sends its partial upward, it preserves the Job, Group, and Operation identity, replaces Contributor ID with the installed parent_contributor_id, uses its local PRISM egress MAC as source, and uses the Parent MAC from the upstream action as destination. It generates the outgoing Chunk Index from the Partial Wire Format geometry. Therefore a format conversion may combine input packet ranges or split them across several partial DATA packets; it does not blindly copy the incoming Chunk Index.

PM_OP_COL_RES¶

The 24-byte result header is followed by little-endian result elements:

%%{init: {"packet": {"bitsPerRow": 96, "bitWidth": 9, "rowHeight": 78, "showBits": true}} }%%
packet
    +64: "Job ID (64b)"
    +16: "Group ID (16b)"
    +16: "Group Generation (16b)"
    +16: "Operation ID (16b)"
    +32: "Operation Generation (32b)"
    +16: "Result Path ID (16b)"
    +24: "Chunk Index (24b)"
    +8: "Flags (8b)"

The Result Path ID identifies one configured result replica on the current direct path. It is replaced with the locally configured ID when another switch replicates the result. Each replica preserves the Job, Group, Operation, and Chunk Index because every RES hop uses the same Output Wire Format. It uses the local PRISM egress MAC as source and takes its destination MAC from the selected replication-group entry.

PM_OP_COL_PROGRESS¶

PM_OP_COL_PROGRESS has this fixed 40-byte header, followed by an optional selective-ACK bitmap:

%%{init: {"packet": {"bitsPerRow": 80, "bitWidth": 12, "rowHeight": 78, "showBits": true}} }%%
packet
    +64: "Job ID (64b)"
    +16: "Group ID (16b)"
    +16: "Group Generation (16b)"
    +16: "Operation ID (16b)"
    +32: "Operation Generation (32b)"
    +16: "Peer ID (16b)"
    +8: "Peer Kind (8b)"
    +8: "Flags (8b)"
    +24: "Window Base (24b)"
    +16: "Admitted Window (16b)"
    +24: "Selective ACK Base (24b)"
    +16: "Bitmap Bits (16b)"
    +8: "Status (8b)"
    +8: "Reason (8b)"
    +32: "Retry Delay us (32b)"
    +16: "Reserved (16b)"

PROGRESS travels in the opposite direction from the DATA or RES packet it controls. It does not mean that the whole collective has started or completed: it reports what the adjacent receiver accepted and how much more traffic it can receive. It is never the tree-wide start signal; that signal comes from the Host Coordinator only after the AM has observed every switch as ACTIVE. Each direct link can therefore retransmit missing chunks locally without restarting the end-to-end operation. An ACK terminates at the immediately previous sender; it is never forwarded through the tree as the same ACK.

Single-node collective flow¶

The implemented topology contains one SINGLE node with two directly attached endpoint ports. Both ports can carry CONTROL and DATA frames, and either port can receive generated RES frames according to the installed result routes. The storyboard follows REDUCE(SUM, root = rank1).

1. Configure the node¶

The AM opens a session, installs one group, then prepares and commits an operation. Contributor bindings identify each endpoint by ingress port and source MAC; result routes select the egress, destination MAC, and Result Path ID. The node accepts DATA only after OP_COMMIT makes the operation ACTIVE.

2. Release the endpoints from the barrier¶

After the SINGLE node reports the operation as ACTIVE, the AM returns a StartGrant to the Host Coordinator. The Coordinator then publishes the two rank-specific grants through the reliable host runtime channel. Only after this step may rank0 and rank1 begin transmitting DATA.

3. Submit endpoint contributions¶

Each endpoint sends the same logical chunk with its local Contributor ID. The ingress validates identity, source binding, generation, geometry, length, and frame boundary before emitting arithmetic tiles. A duplicate contribution for the same global range is ignored.

4. Reduce each range¶

The arithmetic path converts input elements and tracks contributors by global range. Once both contributors arrive, it applies the selected accumulation policy and converts the completed lanes to the output format. Ranges complete independently; a result chunk becomes visible only when all of its tiles are stored.

5. Deliver results¶

For this REDUCE example, the route sends one RES frame to rank1. The packetizer supplies the installed Result Path ID, exact Chunk Index, length, and frame boundary. ALLREDUCE instead creates one packet job per endpoint while retaining the shared result chunk until every replica finishes.

CONTROL responses and RES share each egress through a frame-locking arbiter, so their beats cannot interleave.

The current receive path does not consume switch partial DATA, RES, or PROGRESS frames. Multi-switch aggregation and hop-by-hop retry are described in Future work.

Spine-leaf collective flow¶

The following storyboard explains the complete protocol design with one spine S0, two leaves L0 and L1, and four ranks. It is the target multi-switch flow. The current RTL implements the SINGLE-node subset described above; switch-to-switch partials and hop-by-hop replay remain future work.

For chunk \(i\):

Steps 1 to 3 start one invocation, steps 4 to 7 can overlap across chunks, and steps 8 to 9 close and release the operation.

1. Ranks request the collective¶

The ranks enter the same runtime collective. rank0 also hosts the Host Coordinator, which waits for the participant set before asking the Aggregation Manager (AM) to begin. No PRISM DATA packet is sent yet.

2. Configure the switch tree¶

The AM installs the reusable Group when needed, then prepares and commits one Operation on each target switch.

Topology belongs to the Group. Vector length, arithmetic configuration, chunk geometry, and completion state belong to the Operation.

3. Release the ranks from the barrier¶

The AM returns a StartGrant only after all switches report ACTIVE. The Host Coordinator then publishes rank-specific grants through the runtime. A leaf alone cannot prove that the whole tree is ready.

4. Submit rank contributions¶

Each rank sends DATA chunk \(i\) to its configured leaf. The header identifies the Operation, local Contributor ID, and Chunk Index. This example uses FP32 for input, partial, and output, so their chunk geometry is equal. Other format combinations map the same global ranges to different chunk indices.

5. Complete the leaf reductions¶

The leaves reduce independently:

Each leaf retains its completed partial until the parent acknowledges it.

6. Aggregate at the root¶

The leaves send their partials as DATA. At S0, each child subtree is one local contributor. After both partials arrive:

S0 acknowledges the two direct transfers independently. A leaf may then reuse the replay slot for that partial without waiting for final result delivery.

7. Route the result to rank1¶

The route at S0 sends the result toward L0; the route at L0 selects rank1. Each direct transfer has its own Result Path ID and replay lifetime.

Accepting a result transfers responsibility to the next node. Duplicate packets are acknowledged again but do not update state twice.

8. Complete the operation¶

Nodes may become COMPLETE at different times. Per-chunk accumulator and replay resources can be reclaimed incrementally, while the Operation handle and retired-generation record remain until release.

9. Release the Operation state¶

After tree-wide completion, the Host Coordinator asks the AM to release the Operation on every switch. The reusable Group remains installed.

The target lifecycle is:

FREE -> PREPARED -> ACTIVE -> COMPLETE -> RELEASING -> FREE

PREPARED or ACTIVE may enter ABORTED; retained identity state prevents late packets from being accepted as part of a later generation.

How ALLREDUCE differs¶

Setup and upward aggregation remain the same. Only result delivery changes: S0 replicates toward both leaves and each leaf replicates toward its local participants. Every direct result link retains its own Result Path ID.

Collective semantics¶

The current RTL supports REDUCE and ALLREDUCE.

Reduction operator¶

SUM is the reduction operator implemented in the current RTL.

Wire formats and accumulator profiles¶

An input wire format describes endpoint contributions, a partial wire format describes DATA emitted from one switch to its parent, and an output wire format describes final RES elements. An accumulator profile defines the representation and width of the internal state together with the order in which contributors update it. The four explicit choices keep every conversion boundary selectable and queryable.

Protocol 1.0 assigns only FP32 for switch-to-switch partials:

This is deliberately stricter than the endpoint format registry: it prevents a tree from narrowing partials to INT8, FP16, or BF16 at every hop.

The accumulator-profile assignments are:

Assignment of a code does not claim RTL support: the complete (input, accumulator profile, partial, output) tuple must be advertised and accepted by OP_PREPARE.

KULISCH_FP32 uses the one accumulator width configured in the bitstream. During admission, a switch derives the exact width required by the Reduction Operator, selected input format, and Maximum Reduction Terms and compares it with that configured width. An insufficient width is reported through the sticky ACCUMULATION_INEXACT status; it does not select another datapath. Protocol 1.0 supports only scalar SUM; a future dot-product operator requires the wider product bound and enough descriptor information for both operands. Protocol 1.0 does not yet assign a lossless Kulisch Partial Wire Format. Using an FP32 partial would still round at every switch boundary, so tree-exact operation additionally requires a compatible partial encoding. See Floating-Point Reproducibility for the numerical derivation.

Normative registries and setup records¶

The packet path above is sufficient to understand the protocol. The following collapsed blocks are the exact implementation reference for parser, RTL, and host-software work.

Arithmetic support boundary¶

Wire-format and accumulator values are registry assignments, not support claims. INT8 payload elements are signed two's-complement bytes; FP16 and FP32 carry IEEE 754 binary16 and binary32 bit patterns; BF16 carries one sign bit, eight exponent bits, and seven fraction bits.

A node supports an arithmetic configuration when each selected input format, accumulator profile, partial format, and output format is present in its corresponding capability mask. OP_PREPARE selects one configuration and the AM requires the same configuration at every switch in the tree. Protocol 1.0 freezes one baseline configuration for the SUM operator:

An implementation advertising this configuration obeys the following normative semantics:

• each input and output is an IEEE 754 binary32 bit pattern;
• the FP32_ORDERED accumulator profile folds contributors in strictly ascending local Contributor ID order, starting with the value from the lowest ID; no zero seed is inserted;
• every binary addition is performed in binary32 and rounded immediately using round-to-nearest, ties-to-even; fused or wider intermediate arithmetic is not permitted;
• subnormal inputs and results use gradual underflow and are not flushed to zero;
• finite overflow produces the correctly signed infinity;
• infinity follows IEEE 754 addition, including an invalid +infinity + -infinity result;
• any NaN input, signaling NaN, or invalid addition produces the canonical quiet NaN bit pattern 0x7FC00000; no NaN payload is propagated;
• same-sign zero addition preserves that sign; an exact zero sum of opposite signs is +0 under the selected rounding mode;
• arithmetic exception flags and traps are not transported by Protocol 1.0.