FPGA/corundum
1
0
Fork 0
mirror of https://github.com/corundum/corundum.git synced 2025-01-16 08:12:53 +08:00
Code Issues Projects Releases Wiki Activity

Update readme

Browse Source
This commit is contained in:
Alex Forencich 2021-09-13 20:40:39 -07:00
parent 1bee717bc8
commit 52ba4c40e2
1 changed files with 122 additions and 89 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

211
README.md
View File

@ -4,32 +4,21 @@
GitHub repository: https://github.com/corundum/corundum
GitHub wiki: https://github.com/corundum/corundum/wiki
Google group: https://groups.google.com/d/forum/corundum-nic
Slack workspace: https://join.slack.com/t/corundumworkspace/shared_invite/zt-tj5azsbm-V9LV8L7ugSRDBpe2JiPKMA
## Introduction
Corundum is an open-source, high-performance FPGA-based NIC. Features include
a high performance datapath, 10G/25G/100G Ethernet, PCI express gen 3, a
custom, high performance, tightly-integrated PCIe DMA engine, many (1000+)
transmit, receive, completion, and event queues, scatter/gather DMA, MSI
interrupts, multiple interfaces, multiple ports per interface, per-port
transmit scheduling including high precision TDMA, flow hashing, RSS, checksum
offloading, and native IEEE 1588 PTP timestamping. A Linux driver is included
that integrates with the Linux networking stack. Development and debugging is
facilitated by an extensive simulation framework that covers the entire system
from a simulation model of the driver and PCI express interface on one side to
the Ethernet interfaces on the other side.
Corundum is an open-source, high-performance FPGA-based NIC. Features include a high performance datapath, 10G/25G/100G Ethernet, PCI express gen 3, a custom, high performance, tightly-integrated PCIe DMA engine, many (1000+) transmit, receive, completion, and event queues, scatter/gather DMA, MSI interrupts, multiple interfaces, multiple ports per interface, per-port transmit scheduling including high precision TDMA, flow hashing, RSS, checksum offloading, and native IEEE 1588 PTP timestamping. A Linux driver is included that integrates with the Linux networking stack. Development and debugging is facilitated by an extensive simulation framework that covers the entire system from a simulation model of the driver and PCI express interface on one side to the Ethernet interfaces on the other side.
Corundum has several unique architectural features. First, transmit, receive,
completion, and event queue states are stored efficiently in block RAM or
ultra RAM, enabling support for thousands of individually-controllable
queues. These queues are associated with interfaces, and each interface can
have multiple ports, each with its own independent scheduler. This enables
extremely fine-grained control over packet transmission. Coupled with PTP time
synchronization, this enables high precision TDMA.
Corundum has several unique architectural features. First, transmit, receive, completion, and event queue states are stored efficiently in block RAM or ultra RAM, enabling support for thousands of individually-controllable queues. These queues are associated with interfaces, and each interface can have multiple ports, each with its own independent scheduler. This enables extremely fine-grained control over packet transmission. Coupled with PTP time synchronization, this enables high precision TDMA.
Corundum currently supports Xilinx Virtex 7, UltraScale, and UltraScale+ series
devices. Designs are included for the following FPGA boards:
Corundum also provides an application section for implementing custom logic. The application section has a dedicated PCIe BAR for control and a number of interfaces that provide access to the core datapath and DMA infrastructure.
Corundum currently supports Xilinx Virtex 7, UltraScale, and UltraScale+ series devices. Designs are included for the following FPGA boards:
* Alpha Data ADM-PCIE-9V3 (Xilinx Virtex UltraScale+ XCVU3P)
* Exablaze ExaNIC X10 (Xilinx Kintex UltraScale XCKU035)
@ -45,11 +34,9 @@ devices. Designs are included for the following FPGA boards:
* Xilinx VCU1525 (Xilinx Virtex UltraScale+ XCVU9P)
* Xilinx ZCU106 (Xilinx Zynq UltraScale+ XCZU7EV)
For operation at 10G and 25G, Corundum uses the open source 10G/25G MAC and
PHY modules from the verilog-ethernet repository, no extra licenses are
required. However, it is possible to use other MAC and/or PHY modules.
Operation at 100G currently requires using the Xilinx CMAC core with RS-FEC
enabled, which is covered by the free CMAC license on Xilinx UltraScale+ parts.
For operation at 10G and 25G, Corundum uses the open source 10G/25G MAC and PHY modules from the verilog-ethernet repository, no extra licenses are required. However, it is possible to use other MAC and/or PHY modules.
Operation at 100G on Xilinx UltraScale+ devices currently requires using the Xilinx CMAC core with RS-FEC enabled, which is covered by the free CMAC license.
## Documentation
@ -63,132 +50,177 @@ Block diagram of the Corundum NIC. PCIe HIP: PCIe hard IP core; AXIL M: AXI lite
#### `cmac_pad` module
Frame pad module for 512 bit 100G CMAC TX interface. Zero pads transmit
frames to minimum 64 bytes.
Frame pad module for 512 bit 100G CMAC TX interface. Zero pads transmit frames to minimum 64 bytes.
#### `cpl_op_mux` module
Completion operation multiplexer module. Merges completion write operations
from different sources to enable sharing a single `cpl_write` module instance.
Completion operation multiplexer module. Merges completion write operations from different sources to enable sharing a single `cpl_write` module instance.
#### `cpl_queue_manager` module
Completion queue manager module. Stores device to host queue state in block
RAM or ultra RAM.
Completion queue manager module. Stores device to host queue state in block RAM or ultra RAM.
#### `cpl_write` module
Completion write module. Responsible for enqueuing completion and event
records into the completion queue managers and writing records into host
memory via DMA.
Completion write module. Responsible for enqueuing completion and event records into the completion queue managers and writing records into host memory via DMA.
#### `desc_fetch` module
Descriptor fetch module. Responsible for dequeuing descriptors from the queue
managers and reading descriptors from host memory via DMA.
Descriptor fetch module. Responsible for dequeuing descriptors from the queue managers and reading descriptors from host memory via DMA.
#### `desc_op_mux` module
Descriptor operation multiplexer module. Merges descriptor fetch operations
from different sources to enable sharing a single `desc_fetch` module instance.
Descriptor operation multiplexer module. Merges descriptor fetch operations from different sources to enable sharing a single `desc_fetch` module instance.
#### `event_mux` module
Event mux module. Enables multiple event sources to feed the same event queue.
#### `mqnic_core` module
Core module. Contains the interfaces, asynchronous FIFOs, PTP subsystem, statistics collection subsystem, and application block.
#### `mqnic_core_pcie` module
Core module for a PCIe host interface. Wraps `mqnic_core` along with generic PCIe interface components, including DMA engine and AXI lite masters.
#### `mqnic_core_pcie_us` module
Core module for a PCIe host interface on Xilinx 7-series, UltraScale, and UltraScale+. Wraps `mqnic_core_pcie` along with FPGA-specific interface logic.
#### `mqnic_interface` module
Interface module. Contains the event queues, interface queues, and ports.
#### `mqnic_port` module
Port module. Contains the transmit and receive datapath components, including
transmit and receive engines and checksum and hash offloading.
Port module. Contains the transmit and receive datapath components, including transmit and receive engines and checksum and hash offloading.
#### `mqnic_ptp` module
PTP subsystem. Contains one `mqnic_ptp_clock` instance and a parametrizable number of `mqnic_ptp_perout` instances.
#### `mqnic_ptp_clock` module
PTP clock module. Contains an instance of `ptp_clock` with a register interface.
#### `mqnic_ptp_perout` module
PTP period output module. Contains an instance of `ptp_perout` with a register interface.
#### `mqnic_tx_scheduler_block_rr` module
Transmit scheduler block with round-robin transmit scheduler and register interface.
#### `mqnic_tx_scheduler_block_rr_tdma` module
Transmit scheduler block with round-robin transmit scheduler, TDMA scheduler, TDMA scheduler controller, and register interface.
#### `queue_manager` module
Queue manager module. Stores host to device queue state in block RAM or ultra
RAM.
Queue manager module. Stores host to device queue state in block RAM or ultra RAM.
#### `rx_checksum` module
Receive checksum computation module. Computes 16 bit checksum of Ethernet
frame payload to aid in IP checksum offloading.
Receive checksum computation module. Computes 16 bit checksum of Ethernet frame payload to aid in IP checksum offloading.
#### `rx_engine` module
Receive engine. Manages receive datapath operations including descriptor
dequeue and fetch via DMA, packet reception, data writeback via DMA, and
completion enqueue and writeback via DMA. Handles PTP timestamps for
inclusion in completion records.
Receive engine. Manages receive datapath operations including descriptor dequeue and fetch via DMA, packet reception, data writeback via DMA, and completion enqueue and writeback via DMA. Handles PTP timestamps for inclusion in completion records.
#### `rx_hash` module
Receive hash computation module. Extracts IP addresses and ports from packet
headers and computes 32 bit Toeplitz flow hash.
Receive hash computation module. Extracts IP addresses and ports from packet headers and computes 32 bit Toeplitz flow hash.
#### `stats_collect` module
Statistics collector module. Parametrizable number of increment inputs, single AXI stream output for accumulated counts.
#### `stats_counter` module
Statistics counter module. Receives increments over AXI stream and accumulates them in block RAM, which is accessible via AXI lite.
#### `stats_dma_if_pcie` module
Collects DMA-related statistics for `dma_if_pcie` module, including operation latency.
#### `stats_dma_if_latency` module
DMA latency measurement module.
#### `stats_pcie_if` module
Collects TLP-level statistics for the generic PCIe interface.
#### `stats_pcie_tlp` module
Extracts TLP-level statistics for the generic PCIe interface (single channel).
#### `tdma_ber_ch` module
TDMA bit error ratio (BER) test channel module. Controls PRBS logic in
Ethernet PHY and accumulates bit errors. Can be configured to bin error
counts by TDMA timeslot.
TDMA bit error ratio (BER) test channel module. Controls PRBS logic in Ethernet PHY and accumulates bit errors. Can be configured to bin error counts by TDMA timeslot.
#### `tdma_ber` module
TDMA bit error ratio (BER) test module. Wrapper for a tdma_scheduler and
multiple instances of `tdma_ber_ch`.
TDMA bit error ratio (BER) test module. Wrapper for a tdma_scheduler and multiple instances of `tdma_ber_ch`.
#### `tdma_scheduler` module
TDMA scheduler module. Generates TDMA timeslot index and timing signals from
PTP time.
TDMA scheduler module. Generates TDMA timeslot index and timing signals from PTP time.
#### `tx_checksum` module
Transmit checksum computation and insertion module. Computes 16 bit checksum
of frame data with specified start offset, then inserts computed checksum at
the specified position.
Transmit checksum computation and insertion module. Computes 16 bit checksum of frame data with specified start offset, then inserts computed checksum at the specified position.
#### `tx_engine` module
Transmit engine. Manages transmit datapath operations including descriptor
dequeue and fetch via DMA, packet data fetch via DMA, packet transmission, and
completion enqueue and writeback via DMA. Handles PTP timestamps for
inclusion in completion records.
Transmit engine. Manages transmit datapath operations including descriptor dequeue and fetch via DMA, packet data fetch via DMA, packet transmission, and completion enqueue and writeback via DMA. Handles PTP timestamps for inclusion in completion records.
#### `tx_scheduler_ctrl_tdma` module
TDMA transmit scheduler control module. Controls queues in a transmit
scheduler based on PTP time, via a `tdma_scheduler` instance.
TDMA transmit scheduler control module. Controls queues in a transmit scheduler based on PTP time, via a `tdma_scheduler` instance.
#### `tx_scheduler_rr` module
Round-robin transmit scheduler. Determines which queues from which to send
packets.
Round-robin transmit scheduler. Determines which queues from which to send packets.
### Source Files
cmac_pad.v : Pad frames to 64 bytes for CMAC TX
cpl_op_mux.v : Completion operation mux
cpl_queue_manager.v : Completion queue manager
cpl_write.v : Completion write module
desc_fetch.v : Descriptor fetch module
desc_op_mux.v : Descriptor operation mux
event_mux.v : Event mux
event_queue.v : Event queue
mqnic_interface.v : Interface
mqnic_port.v : Port
queue_manager.v : Queue manager
rx_checksum.v : Receive checksum offload
rx_engine.v : Receive engine
rx_hash.v : Receive hashing module
tdma_ber_ch.v : TDMA BER channel
tdma_ber.v : TDMA BER
tdma_scheduler.v : TDMA scheduler
tx_checksum.v : Transmit checksum offload
tx_engine.v : Transmit engine
tx_scheduler_ctrl_tdma.v : TDMA transmit scheduler controller
tx_scheduler_rr.v : Round robin transmit scheduler
cmac_pad.v : Pad frames to 64 bytes for CMAC TX
cpl_op_mux.v : Completion operation mux
cpl_queue_manager.v : Completion queue manager
cpl_write.v : Completion write module
desc_fetch.v : Descriptor fetch module
desc_op_mux.v : Descriptor operation mux
event_mux.v : Event mux
event_queue.v : Event queue
mqnic_core.v : Core logic
mqnic_core_pcie.v : Core logic for PCIe
mqnic_core_pcie_us.v : Core logic for PCIe (UltraScale)
mqnic_interface.v : Interface
mqnic_port.v : Port
mqnic_ptp.v : PTP subsystem
mqnic_ptp_clock.v : PTP clock wrapper
mqnic_ptp_perout.v : PTP period output wrapper
mqnic_tx_scheduler_block_rr.v : Scheduler block (round-robin)
mqnic_tx_scheduler_block_rr_tdma.v : Scheduler block (round-robin TDMA)
queue_manager.v : Queue manager
rx_checksum.v : Receive checksum offload
rx_engine.v : Receive engine
rx_hash.v : Receive hashing module
stats_collect.v : Statistics collector
stats_counter.v : Statistics counter
stats_dma_if_pcie.v : DMA interface statistics
stats_dma_latency.v : DMA latency measurement
stats_pcie_if.v : PCIe interface statistics
stats_pcie_tlp.v : PCIe TLP statistics
tdma_ber_ch.v : TDMA BER channel
tdma_ber.v : TDMA BER
tdma_scheduler.v : TDMA scheduler
tx_checksum.v : Transmit checksum offload
tx_engine.v : Transmit engine
tx_scheduler_ctrl_tdma.v : TDMA transmit scheduler controller
tx_scheduler_rr.v : Round robin transmit scheduler
## Testing
@ -201,6 +233,7 @@ Running the included testbenches requires [cocotb](https://github.com/cocotb/coc
- J. A. Forencich, *System-Level Considerations for Optical Switching in Data Center Networks*, [Paper](https://escholarship.org/uc/item/3mc9070t)
## Citation
If you use Corundum in your project please cite one of the following papers
and/or link to the github project: