Skip to content

2.3 Alveo U280

The board build puts the collective core behind port 0 of an Alveo U280. It reuses the CMAC subsystem and the packet adapter of the OpenNIC shell at a pinned commit and has no PCIe endpoint: PRISM is configured in band, so the only connection the host needs is a 100 GbE interface on the same link. The top is described in prism_u280_top.

Build the bitstream

The build needs Vivado 2022.1 and Git. From the repository root:

make u280
make u280 RS_FEC=off

The OpenNIC shell is cloned into build/cache/open-nic-shell. Checkpoints, reports and prism_u280.bit are written to build/u280/rs-fec-on (or rs-fec-off). The run stops with an error, and writes no bitstream, when the routed design misses timing at 250 MHz. RS-FEC must match the far end of the link: the switch port or the NIC the card is cabled to.

A full build takes about seven hours on a 64 GB machine. Timing closes with little margin, so a small change of the design or of the flow can leave a full build short of it. An incremental build keeps most of the placement and routing of an earlier routed checkpoint that met timing (about 90 % for the two published bitstreams) and is the more reliable way after a small change:

python repo/prism-hdl/integrations/u280/build.py --incremental build/u280/rs-fec-on/routed.dcp

Simulate the board

make test SUITE=rtl-u280-board DOCKER=1

The suite simulates the board top without transceivers: the MAC is replaced by stream ports in its clock, and everything between those ports and the core is the design that is built, with behavioural models of the AMD primitives. It runs reductions with the library driver, with frames entering at 99 Gb/s.

Run a reduction on the card

The host plays the manager and two ranks. It needs Linux, Python 3.13, root for raw sockets, a 100 GbE interface on the same Ethernet segment as port 0 of the card, and JTAG access to the card with Vivado Lab.

  1. Take the card off the PCIe bus and prepare the interface: sudo repo/prism-hdl/integrations/u280/cloudlab/prepare_host.sh <interface>. The design has no PCIe endpoint, and a link that disappears while the card's drivers are loaded can be reported as a fatal error.
  2. Load the bitstream (volatile; a power cycle restores the design in the card's flash): vivado_lab -mode batch -source repo/prism-hdl/integrations/u280/cloudlab/program.tcl -tclargs prism_u280.bit.

  3. Run a reduction and check its result:

    uv tool install ./repo/prism-ccl
sudo ~/.local/bin/prism-single --interface <interface> --input FP32 --output FP32

    --input takes INT8, FP16, BF16 or FP32 and --output any result format, including AUTO and KULISCH. Each run is a new job, so runs can follow each other without reprogramming.

The node answers at 02:50:52:49:53:00, the manager is 02:50:52:49:53:ff and the ranks are 02:00:00:00:00:0a and 02:00:00:00:00:0b; the frames use EtherType 0x88B5. If no frame comes back, check with tcpdump -i <interface> -e ether proto 0x88b5 that the frames leave the host, and that the interface has a carrier (with no carrier, the FEC of the two ends usually differs).