Protocol¶

MiniSharp operates at Layer 2 of the OSI model and uses a custom Ethernet-based protocol to identify packets that must be processed by the MiniSharp datapath.

The protocol supports two main data-plane modes:

1. Transparent packet forwarding, where non-MiniSharp Ethernet frames may be forwarded through the switch-like forwarding pipeline.
2. Streaming collective accumulation, where MiniSharp collective packets are consumed by the collective engine and used to update an in-network reduction context.

MiniSharp is currently a research-oriented FPGA datapath, not a production Ethernet switch. The protocol therefore prioritizes simplicity, inspectability, hardware feasibility, and cycle-level RTL clarity over full production-grade management, security, and interoperability.

Common frame format¶

MiniSharp frames are carried inside Ethernet frames. The Ethernet header provides Layer-2 addressing, while the MiniSharp header identifies the protocol version, packet class, operation-specific metadata, and payload.

The full first-beat diagram intentionally groups small fields to avoid unreadable packet diagrams.

%%{init: {"packet": {"bitsPerRow": 128, "bitWidth": 8, "rowHeight": 78, "showBits": true}} }%%
packet
    +48: "Dst MAC"
    +48: "Src MAC"
    +16: "Ethertype"
    +16: "MS Control"
    +64: "Operation Header"
    +320: "First payload bytes"

The MiniSharp control word is defined as:

%%{init: {"packet": {"bitsPerRow": 16, "bitWidth": 28, "rowHeight": 78, "showBits": true}} }%%
packet
    +4: "Major"
    +4: "Minor"
    +8: "Opcode"

The initial experimental protocol version is:

Unsupported major versions are invalid. If the major version is supported but the minor version is newer than the local implementation, MiniSharp may process only the subset of fields defined by its supported minor version. Newer minor-version features are not guaranteed unless explicitly implemented. Reserved fields must be transmitted as zero.

MiniSharp EtherType and forwarding policy¶

MiniSharp uses a custom EtherType to identify frames belonging to the MiniSharp protocol family. In the current experimental design, 0x88B5 is used as the MiniSharp EtherType. This value belongs to the local experimental EtherType range and is suitable for laboratory and research use, but it is not a production protocol assignment.¹

Frame classification happens in two stages:

1. The Ethernet parser reads Dst MAC, Src MAC, and Ethertype.
2. If Ethertype == 0x88B5, the MiniSharp parser interprets the MiniSharp control word, opcode, operation header, and payload.
3. If Ethertype != 0x88B5, the frame is treated as a non-MiniSharp Ethernet frame.

MiniSharp may be configured with one of two outer forwarding policies:

The default prototype policy is expected to be MSP_TRANSPARENT, because MiniSharp is intended to behave as a switch-like datapath during hardware bring-up and should not unnecessarily discard ordinary Ethernet traffic.

Chunking model¶

MiniSharp collective vectors are split into chunks. A chunk is represented by one Ethernet/MiniSharp packet and may span multiple AXI-Stream beats.

MiniSharp does not carry a payload-length field in the common header. The Aggregation Manager configures the collective context, including vector size, input format, accumulator mode, output format, number of expected local contributors, and result egress range.

For the standard non-jumbo Ethernet case:

Therefore, the default MiniSharp chunk payload size for standard Ethernet frames is:

Jumbo-frame support is intentionally left outside the v0.1 protocol baseline and depends on the network and device configuration.²

The switch computes the expected number of chunks from the context installed by the AM:

For each chunk:

The last chunk may be shorter than MS_CHUNK_PAYLOAD_BYTES; its valid bytes are determined by the configured vector size and by the packet ending signaled through TLAST/TKEEP.

Chunks may arrive out of order if the collective engine uses Chunk Index as the placement key.

For collective data, the logical identity of a contribution is:

Each chunk is completed independently. A chunk result may be emitted as soon as the switch has received all expected local contributors for that chunk; it does not need to wait for other chunks of the same vector.

Data-plane packet classes¶

MiniSharp packets are classified by the parser. If the Ethernet frame carries the MiniSharp EtherType, the Opcode field determines how the operation header and payload are interpreted.

MS_OP_INVALID is not a normal transmitted packet format. It represents an internal classification outcome. The all-zero opcode is intentionally invalid so that reset/default state does not accidentally encode a valid MiniSharp operation.

The opcode values are part of the MiniSharp v0.1 ABI and should be defined in the RTL package, testbench, and host-side packet generator.

In-band setup¶

MiniSharp uses an in-band setup model for the initial FPGA prototype and calls the control entity that installs collective contexts the Aggregation Manager (AM). The name is intentionally aligned with SHARP-style terminology: the AM is the entity that prepares aggregation state before collective data packets arrive. In MiniSharp v0.1, the AM is not a full production control plane; it is the trusted software, testbench, or host-side component that sends MS_OP_COL_SETUP packets to configure the switch-local collective context.

sequenceDiagram
    participant AM as Aggregation Manager
    participant SW as MiniSharp setup path
    participant CTX as Collective context table
    participant CE as Collective engine

    AM->>SW: MS_OP_COL_SETUP
    SW->>CTX: install Collective ID context
    AM->>CE: MS_OP_COL_DATA packets arrive later
    CE->>CTX: lookup Collective ID
    CTX-->>CE: operation, formats, local contributors, egress range
    CE->>CE: accumulate chunks by Rank and Chunk Index

This choice is intentional. For an FPGA prototype connected through an OpenNIC-like environment, in-band setup is a practical way to configure the switch without requiring a separate management CPU, PCIe BAR interface, AXI-Lite register map, or external control-plane stack.

The logical AM interface is independent from the physical transport used to deliver setup commands. Future implementations may map the same operations to a sideband interface such as AXI-Lite, PCIe BAR registers, firmware, or a software control-plane API.

Operation headers¶

The common MiniSharp header contains a fixed 64-bit operation header. Its interpretation depends on the opcode.

flowchart LR
    ETH["Ethernet parser"] --> TYPE{"Ethertype == 0x88B5?"}
    TYPE -->|No| L2["Outer L2 policy"]
    TYPE -->|Yes| CTRL["Read MiniSharp control word"]
    CTRL --> OP{"Opcode"}
    OP -->|MS_OP_COL_SETUP| SETUP["Setup path"]
    OP -->|MS_OP_COL_DATA| DATA["Collective engine"]
    OP -->|MS_OP_COL_RES| RES["Result / egress path"]
    OP -->|unknown| INV["MS_OP_INVALID"]

Reserved fields must be transmitted as zero and ignored by receivers in v0.1.

MS_OP_COL_SETUP¶

MS_OP_COL_SETUP is sent by the Aggregation Manager to install or replace a switch-local collective context.

This packet is consumed by the setup path and must not be inserted into the normal forwarding VOQ datapath.

%%{init: {"packet": {"bitsPerRow": 64, "bitWidth": 12, "rowHeight": 78, "showBits": true}} }%%
packet
    +32: "Collective ID"
    +32: "Reserved"

The setup payload carries the collective context record. MiniSharp currently uses in-band setup packets as the practical control-plane mechanism for the FPGA prototype.

The v0.1 setup record is fixed-size and begins at the payload of MS_OP_COL_SETUP.

%%{init: {"packet": {"bitsPerRow": 128, "bitWidth": 8, "rowHeight": 78, "showBits": true}} }%%
packet
    +8: "Collective"
    +8: "Reduce Op"
    +8: "Input Fmt"
    +8: "Acc Mode"
    +8: "Output Fmt"
    +8: "Egr First"
    +8: "Egr Last"
    +8: "Reserved"
    +16: "Expected Local"
    +32: "Element Count"
    +16: "Reserved"

expected_local_contributors is local to this switch and this Collective ID. It is not the number of global ranks in the whole job. In a spine-leaf topology, each aggregation node waits only for the contributors directly assigned to that node by the AM.

The result of a completed chunk is emitted to every local output port \(O\) such that:

Examples:

This range-based model is simple and FPGA-friendly. Non-contiguous output sets are out of scope for v0.1 and may be added later through a bitmap or table-based destination model.

The setup packet does not carry a result destination MAC address. The AM configures each switch in the topology with the local egress range for the given Collective ID. The packet builder is responsible for generating valid Ethernet headers for emitted MS_OP_COL_RES frames according to local topology or testbench configuration.

A setup packet is invalid if:

• expected_local_contributors == 0;
• element_count == 0;
• egress_port_first > egress_port_last;
• egress_port_last >= NUM_PORTS;
• collective_semantic is unsupported;
• reduction_operator is unsupported;
• input_format, accumulator_mode, or output_format is unsupported;
• reserved fields are non-zero.

MiniSharp trusts setup writes to succeed in the current prototype. It does not define setup acknowledgments, setup responses, or status packets.

MS_OP_COL_DATA¶

MS_OP_COL_DATA carries a rank contribution for one vector chunk. The packet is consumed by the collective engine and is not forwarded directly through the normal VOQ datapath.

%%{init: {"packet": {"bitsPerRow": 64, "bitWidth": 12, "rowHeight": 78, "showBits": true}} }%%
packet
    +8: "Rank"
    +32: "Collective ID"
    +16: "Chunk Index"
    +8: "Reserved"

Rank is interpreted as a local contributor identifier within the collective context installed on this switch. It does not need to be the global MPI rank or the global process identifier.

Chunk Index identifies the vector region carried by this packet. It replaces the older Sequence name because its semantic role is placement within the configured vector, not generic packet sequencing.

The AM configures vector length, input format, accumulator mode, output format, and expected local contributors for the given Collective ID. Therefore, MS_OP_COL_DATA does not carry a payload-length field.

For each chunk, the collective engine waits until it has received expected_local_contributors unique local rank contributions. Once the chunk is complete, it can generate MS_OP_COL_RES for that chunk without waiting for other chunks of the same vector.

MS_OP_COL_RES¶

MS_OP_COL_RES is generated by MiniSharp when the collective engine has produced a result chunk and the configured egress range requires result emission.

%%{init: {"packet": {"bitsPerRow": 64, "bitWidth": 12, "rowHeight": 78, "showBits": true}} }%%
packet
    +8: "Reserved"
    +32: "Collective ID"
    +16: "Chunk Index"
    +8: "Reserved"

The result payload format is determined by the context configured by MS_OP_COL_SETUP. The output format may differ from both the input format and the internal accumulator mode.

Example:

input_format     = MS_FMT_FP32
accumulator_mode = MS_ACC_EXACT_FP32_MAG277
output_format    = MS_FMT_FP32

This means that input elements are received as MS_FMT_FP32, accumulated internally using an exact FP32 scalar-sum accumulator family, and emitted as MS_FMT_FP32.

MS_OP_INVALID¶

MS_OP_INVALID represents an internal parser or setup-path decision. It is used when a packet cannot be processed safely.

Examples include:

• unsupported MiniSharp major version;
• unknown opcode;
• malformed header;
• invalid setup payload;
• unknown Collective ID;
• unsupported wire format;
• unsupported accumulator mode;
• duplicate contribution for the same (Collective ID, Rank ID, Chunk Index);
• payload inconsistent with the configured collective context.

The current prototype drops invalid packets or streams and increments internal simulation-visible error counters. It does not generate error packets.

Collective semantic¶

The collective semantic describes the communication-level behavior.

Only MS_COL_REDUCE is in scope for the first MiniSharp thesis prototype.

Other collective semantics, such as allreduce, broadcast, and reduce-scatter, are future work. In particular, reduce-scatter requires a mapping from result chunks to different destinations and is intentionally left outside the v0.1 protocol.

Reduction operator¶

The reduction operator describes the mathematical operation applied to elements.

Only MS_RED_SUM is in scope for the first MiniSharp thesis prototype.

Wire formats and accumulator modes¶

MiniSharp distinguishes wire formats from accumulator modes.

A wire format describes how elements are encoded in MS_OP_COL_DATA and MS_OP_COL_RES payloads.

An accumulator mode describes how the collective engine internally combines contributions.

For exact finite scalar summation with \(R\) contributing ranks:

This follows the scalar-reduction sizing model, not the wider dot-product Kulisch bound.⁴

For the detailed derivation, see 1.2 Reproducibility.

Out-of-order chunks¶

MiniSharp allows collective chunks to arrive out of order if the collective engine uses Chunk Index as the placement key.

For each contribution, the collective engine identifies:

The Chunk Index determines the vector region updated by the packet payload. This allows the communication layer or packet generator to split a large vector into Ethernet-sized chunks and transmit them in an order independent of the final vector order.

Each chunk is treated as an independently completable reduction unit. A chunk result can be emitted once the switch has received all expected local contributors for that chunk.

Duplicate contributions for the same (Collective ID, Rank ID, Chunk Index) are ignored or dropped.

Reserved and invalid behavior¶

Reserved fields must be transmitted as zero and ignored by receivers in v0.1.

A packet is invalid if it uses unsupported major versions, unknown opcodes, unsupported setup values, non-zero reserved fields where zero is required, or context identifiers that are not installed.

MiniSharp v0.1 does not generate error packets. Invalid packets are dropped or ignored, and the implementation may increment simulation-visible or debug-visible error counters.