Update readme

README.md

@@ -6,8 +6,11 @@ GitHub repository: https://github.com/alexforencich/verilog-pcie
 
 ## Introduction
 
-Collection of PCI express related components.  Includes full MyHDL testbench
+Collection of PCI express related components.  Includes PCIe to AXI and AXI
-with intelligent bus cosimulation endpoints.
+lite bridges, a simple PCIe AXI DMA engine, and a flexible, high-performance
+DMA subsystem.  Currently supports operation with the Xilinx Ultrascale and
+Ultrascale Plus PCIe hard IP cores with interfaces between 64 and 512 bits.
+Includes full MyHDL testbench with intelligent bus cosimulation endpoints.
 
 ## Documentation
 
@@ -26,21 +29,129 @@ cosimulated design, enabling testing of not only isolated components and
 host-device communication but also communication between multiple components
 such as device-to-device DMA and message passing.
 
-A MyHDL model of the Xilinx Ultrascale PCIe hard core is included in
+MyHDL models of the Xilinx Ultrascale and Ultrascale Plus PCIe hard cores are
-pcie_us.py.  This module can be used in combination with the PCIe BFM to test
+included in pcie_us.py and pcie_usp.py.  These modules can be used in
-a MyHDL or Verilog design that targets a Xilinx Ultrascale FPGA.  The model
+combination with the PCIe BFM to test a MyHDL or Verilog design that targets a
-currently only supports operation as a device, not as a root port.
+Xilinx Ultrascale or Ultrascale Plus FPGA.  The models currently only support
+operation as a device, not as a root port.
+
+### PCIe AXI and AXI lite master
+
+The pcie_us_axi_master and pcie_us_axil_master modules provide a bridge
+between PCIe and AXI.  These can be used to implement PCIe BARs.  The
+pcie_us_axil_master module is a very simple module for providing register
+access, supporting only 32 bit operations.  The pcie_us_axi_master module is
+more complex, converting PCIe operations to AXI bursts.  It can be used to
+terminate device-to-device DMA operations with reasonable performance.  The
+pcie_us_axis_cq_demux module can be used to demultiplex PCIe operations based
+on the target BAR.
+
+### PCIe AXI DMA
+
+The pcie_us_axi_dma module provides a DMA engine with an internal AXI
+interface.  The AXI interface width must match the PCIe interface width.  The
+module directly translates AXI operations into PCIe operations.  As a result,
+it is relatively simple, but the performance is limited due to the constraints
+of the AXI interface.  Backpressure on the AXI interface is also passed
+through to the PCIe interface.
+
+The pcie_axi_dma_desc_mux module can be used to share the AXI DMA module
+between multiple request sources.
+
+### Flexible DMA subsystem
+
+The split DMA interface/DMA client modules support highly flexible, highly
+performant DMA operations.  The DMA interface and DMA client modules are
+connected by dual port RAMs with a high performance segmented memory
+interface.  The segmented memory interface is a better 'impedance match' to
+the PCIe hard core interface - data realignment can be done in the same clock
+cycle; no bursts, address decoding, arbitration, or reordering simplifies
+implementation and provides much higher performance than AXI.  The architecture
+is also quite flexible as it decouples the DMA interface from the clients with
+dual port RAMs, enabling mixing different client interface types and widths
+and even supporting clients running in different clock domains without
+datapath FIFOs.
+
+The dma_if_pcie_us module connects the PCIe interface to the segmented memory
+interface.  Currently, it does not support TLP straddling, but it should be
+possible to support this with the segmented interface.
+
+The dma_psdpram module is a dual clock, parallel simple dual port RAM module
+with a segmented interface.  The depth is independently adjustable from the
+address width, simplifying use of the segmented interface.  The module also contains a parametrizable output pipeline register to improve timing.
+
+The dma_if_mux module enables sharing the DMA interface across several DMA
+clients.  This module handles the tags and select lines appropriately on both
+the descriptor and segmented memory interface for plug-and-play operation
+without address assignment - routing is completely determined by component
+connections.  The module also contains a FIFO to maintain read data ordering
+across multiple clients.  Make sure to equalize pipeline delay across all
+paths for maximum performance.
+
+DMA client modules connect the segmented memory interface to different
+internal interfaces.
+
+The dma_client_axis_source and dma_client_axis_sink modules provide support
+for streaming DMA over AXI stream.  The AXI stream width can be any power of
+two fraction of the segmented memory interface width.
 
 ### arbiter module
 
 General-purpose parametrizable arbiter.  Supports priority and round-robin
-arbitration.  Supports blocking until request release or acknowledge.  
+arbitration.  Supports blocking until request release or acknowledge.
 
 ### axis_arb_mux module
 
 Frame-aware AXI stream arbitrated muliplexer with parametrizable data width
 and port count.  Supports priority and round-robin arbitration.
 
+### dma_client_axis_sink module
+
+AXI stream sink DMA client module.  Uses a segmented memory interface.
+
+### dma_client_axis_source module
+
+AXI stream source DMA client module.  Uses a segmented memory interface.
+
+### dma_if_mux module
+
+DMA interface mux module.  Enables sharing a DMA interface module between
+multiple DMA client modules.  Wrapper for dma_if_mux_rd and dma_if_mux_wr.
+
+### dma_if_mux_rd module
+
+DMA interface mux module.  Enables sharing a DMA interface module between
+multiple DMA client modules.  Muxes descriptors and demuxes memory writes.
+
+### dma_if_mux_wr module
+
+DMA interface mux module.  Enables sharing a DMA interface module between
+multiple DMA client modules.  Muxes descriptors, demuxes memory read commands,
+and muxes read data.
+
+### dma_if_pcie_us module
+
+PCIe DMA interface module for Xilinx Ultrascale series FPGAs.  Supports 64,
+128, 256, and 512 bit datapaths.  Uses a double width segmented memory
+interface.  Wrapper for dma_if_pcie_us_rd and dma_if_pcie_us_wr.
+
+### dma_if_pcie_us_rd module
+
+PCIe DMA interface module for Xilinx Ultrascale series FPGAs.  Supports 64,
+128, 256, and 512 bit datapaths.  Uses a double width segmented memory
+interface.
+
+### dma_if_pcie_us_wr module
+
+PCIe DMA interface module for Xilinx Ultrascale series FPGAs.  Supports 64,
+128, 256, and 512 bit datapaths.  Uses a double width segmented memory
+interface.
+
+### dma_psdpram module
+
+DMA RAM module.  Segmented simple dual port RAM to connect a DMA interface
+module to a DMA client.
+
 ### pcie_axi_dma_desc_mux module
 
 Descriptor multiplexer/demultiplexer for PCIe AXI DMA module.  Enables sharing
@@ -53,43 +164,52 @@ PCIe in-flight tag manager.
 
 ### pcie_us_axi_dma module
 
-PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.  Wrapper for pcie_us_axi_dma_rd and pcie_us_axi_dma_wr
+PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, 256,
+and 512 bit datapaths.  Parametrizable AXI burst length.  Wrapper for
+pcie_us_axi_dma_rd and pcie_us_axi_dma_wr.
 
 ### pcie_us_axi_dma_rd module
 
-PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.
+PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, 256,
+and 512 bit datapaths.  Parametrizable AXI burst length.
 
 ### pcie_us_axi_dma_wr module
 
-PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.
+PCIe AXI DMA module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, 256,
+and 512 bit datapaths.  Parametrizable AXI burst length.
 
 ### pcie_us_axi_master module
 
-PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.  Wrapper for pcie_us_axi_master_rd and pcie_us_axi_master_wr.  
+PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128,
+256, and 512 bit datapaths.  Parametrizable AXI burst length.  Wrapper for
+pcie_us_axi_master_rd and pcie_us_axi_master_wr.
 
 ### pcie_us_axi_master_rd module
 
-PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.
+PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128,
+256, and 512 bit datapaths.  Parametrizable AXI burst length.
 
 ### pcie_us_axi_master_wr module
 
-PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit datapaths.  Parametrizable AXI burst length.
+PCIe AXI master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128,
+256, and 512 bit datapaths.  Parametrizable AXI burst length.
 
 ### pcie_us_axil_master module
 
-PCIe AXI lite master module for Xilinx Ultrascale series FPGAs.  Supports 64, 128, and 256 bit PCIe interfaces.  
+PCIe AXI lite master module for Xilinx Ultrascale series FPGAs.  Supports 64,
+128, 256, and 512 bit PCIe interfaces.
 
 ### pcie_us_axis_cq_demux module
 
 Demux module for Xilinx Ultrascale CQ interface.  Can be used to route
 incoming requests based on function, BAR, and other fields.  Supports 64, 128,
-and 256 bit datapaths.
+256, and 512 bit datapaths.
 
 ### pcie_us_axis_rc_demux module
 
 Demux module for Xilinx Ultrascale RC interface.  Can be used to route
 incoming completions based on the requester ID (function).  Supports 64, 128,
-and 256 bit datapaths.
+256, and 512 bit datapaths.
 
 ### pcie_us_cfg module
 
@@ -114,23 +234,32 @@ signals together.
 
 ### Source Files
 
-    arbiter.v               : Parametrizable arbiter
+    arbiter.v                : Parametrizable arbiter
-    axis_arb_mux.v          : Parametrizable AXI stream mux
+    axis_arb_mux.v           : Parametrizable AXI stream mux
-    pcie_axi_dma_desc_mux.v : Descriptor mux for DMA engine
+    dma_client_axis_sink.v   : AXI stream sink DMA client
-    pcie_tag_manager.v      : PCIe in-flight tag manager
+    dma_client_axis_source.v : AXI stream source DMA client
-    pcie_us_axi_dma.v       : PCIe AXI DMA module with Xilinx Ultrascale interface
+    dma_if_mux.v             : DMA interface mux
-    pcie_us_axi_dma_rd.v    : PCIe AXI DMA read module with Xilinx Ultrascale interface
+    dma_if_mux_rd.v          : DMA interface mux (read)
-    pcie_us_axi_dma_wr.v    : PCIe AXI DMA write module with Xilinx Ultrascale interface
+    dma_if_mux_wr.v          : DMA interface mux (write)
-    pcie_us_axi_master.v    : AXI Master module with Xilinx Ultrascale interface
+    dma_if_pcie_us.v         ; DMA interface for Xilinx Ultrascale PCIe
-    pcie_us_axi_master_rd.v : AXI Master read module with Xilinx Ultrascale interface
+    dma_if_pcie_us_rd.v      ; DMA interface for Xilinx Ultrascale PCIe (read)
-    pcie_us_axi_master_wr.v : AXI Master write module with Xilinx Ultrascale interface
+    dma_if_pcie_us_wr.v      ; DMA interface for Xilinx Ultrascale PCIe (write)
-    pcie_us_axil_master.v   : AXI Lite Master module with Xilinx Ultrascale interface
+    dma_psdpram.v            : DMA RAM (segmented simple dual port RAM)
-    pcie_us_axis_cq_demux.v : Parametrizable AXI stream CQ demux
+    pcie_axi_dma_desc_mux.v  : Descriptor mux for DMA engine
-    pcie_us_axis_rc_demux.v : Parametrizable AXI stream RC demux
+    pcie_tag_manager.v       : PCIe in-flight tag manager
-    pcie_us_cfg.v           : Configuration shim for Xilinx Ultrascale devices
+    pcie_us_axi_dma.v        : PCIe AXI DMA module (Xilinx Ultrascale)
-    pcie_us_msi.v           : MSI shim for Xilinx Ultrascale devices
+    pcie_us_axi_dma_rd.v     : PCIe AXI DMA read module (Xilinx Ultrascale)
-    priority_encoder.v      : Parametrizable priority encoder
+    pcie_us_axi_dma_wr.v     : PCIe AXI DMA write module (Xilinx Ultrascale)
-    pulse_merge             : Parametrizable pulse merge module
+    pcie_us_axi_master.v     : PCIe AXI Master module (Xilinx Ultrascale)
+    pcie_us_axi_master_rd.v  : PCIe AXI Master read module (Xilinx Ultrascale)
+    pcie_us_axi_master_wr.v  : PCIe AXI Master write module (Xilinx Ultrascale)
+    pcie_us_axil_master.v    : PCIe AXI Lite Master module (Xilinx Ultrascale)
+    pcie_us_axis_cq_demux.v  : Parametrizable AXI stream CQ demux
+    pcie_us_axis_rc_demux.v  : Parametrizable AXI stream RC demux
+    pcie_us_cfg.v            : Configuration shim for Xilinx Ultrascale devices
+    pcie_us_msi.v            : MSI shim for Xilinx Ultrascale devices
+    priority_encoder.v       : Parametrizable priority encoder
+    pulse_merge              : Parametrizable pulse merge module
 
 ## Testing
 
@@ -144,3 +273,4 @@ individual test scripts can be run with python directly.
     tb/axis_ep.py        : MyHDL AXI Stream endpoints
     tb/pcie.py           : MyHDL PCI Express BFM
     tb/pcie_us.py        : MyHDL Xilinx Ultrascale PCIe core model
+    tb/pcie_usp.py       : MyHDL Xilinx Ultrascale Plus PCIe core model