For this tutorial, we will use two boards: the small PYNQ-Z2 and the bigger ZCU104. Both are MPSoC (MultiProcessor System-on-Chip), which means that they integrates several fixed blocks (we call them IP for Intellectual Properties [of their design]) such as Arm cores, memory controller, Ethernet controllers, … These fixed IPs forms the PS (Processing Subsystem), while the FPGA is called PL (Programmable Logic). While communication channels exist between PS and PL, PS IPs are not directly accessible from the FPGA. It is for example impossible to use the (PS-wired) Ethernet controller on the PL without a transmitting program running on the Arm CPU to forward packets from the network interface to a shared PS-PL bus.
Therefore, we will first boot the Arm cores of the boards, and from it, install and boot our softcore on the PL. Two Linux with different ISA will cohabit on the board, each one linked to its own storage, peripherals and RAM region.
The PYNQ-Z2 is an entry-level (~200€) board manufactured by TUL. It features a ZYNQ XC7Z020-1CLG400C MPSoC and sufficient IO for small embedded projects (HDMI in/out, mic, GPIO, Ethernet, 2 dual PMOD connectors). More precisely, the PYNQ-Z2 integrates two Cortex-A9 cores running at 650 MHz, a FPGA with 85K logic cells, 630 KB of block RAM and 220 DSP slices; and a total of 512 MiB of shared DDR3.
The reference manual of the PYNQ-Z2 is available here. Download it, as it contains important information for I/O mapping on the FPGA.
The PYNQ-Z2 is big enough to fit one trimmed-down CVA6 64-bit core without debug module, but with sufficient peripherals to boot Linux. This will be our base platform to customize our CVA6 SoC.
(Already done in this case): To share DRAM between the Arm cores and the CVA6 softcore, we used this tutorial to modify the device tree and add a kernel boot parameter to only allow Linux to use the first 256 MiB of RAM, leaving the other 256 MiB for the CVA6.
Note: Boards files (included in the provided Vivado images) are available here.
While both boards include a number of I/O components, none are sufficient as it to boot our softcore. Therefore, we will use PMOD (Peripheral MODule interface) and FMC modules to extend the capabilities of our FPGA and add disk, serial and ethernet communication capabilities.
Pmod interface (Peripheral Module interface) is an open standard defined by Digilent in the Pmod Interface Specification for connecting peripheral modules to FPGA and microcontroller development boards using 6 pins. (Wikipedia)
Pmod is a general purpose interface allowing extensions of boards in a form factor similar to Raspberry Pi’s GPIO. Out of the 6 pins of each Pmod connectors, 4 are used for signals and 2 for power delivery (3.3 V / ground). On our boards, Pmod is available on dual-bundled connectors consisting of 12 pins (2 stacked interfaces).
We will use Pmod extension modules for low-bandwidth peripherals: disk and serial (i.e. console).
Secure Digital (SD) is a proprietary, non-volatile, flash memory card format developed by the SD Association (SDA). Owing to their compact size, SD cards have been widely adopted in a variety of portable consumer electronics, including digital cameras, camcorders, video game consoles, mobile phones, action cameras, and camera drones. (Wikipedia)
SD cards are small flash memory storage devices dating back 1999. Several format and read mechanisms exist to satisfy various needs and speed requirements, from Low-Speed (12.5 MB/s) to SD-Express (3,940 MB/s). In this tutorial we will use the smallest form factor, MicroSD card, as base drive using a Pmod adapter due to their easiness of integration with our boards. The reference manual of the Pmod microSD adapter can be found here. Keep it bookmarked, as we will use it later.
More precisely, we will use the slowest operating mode of an SD card: Serial Peripheral Interface (SPI), which is both simple to implement at the hardware level, widely supported, not subject to royalties and fast enough for the low frequency of our softcore.
A universal asynchronous receiver-transmitter (UART) is a peripheral device for asynchronous serial communication in which the data format and transmission speeds are configurable. (Wikipedia)
While every machine nowadays is equipped with Ethernet and/or WiFi connection to be reachable over network, our softcore is not. The simplest way to interact is over a serial connection provided by a UART. As computers are not equipped anymore with serial ports, we will use a Pmod to USB-UART adapter to implement serial communication with the softcore. The reference manual of the Pmod microSD adapter can be found here. Once again, keep it bookmarked, as we will use it later.
The module relies on a FTDI FT232RQ to expose the module’s UART to a host computer over USB. The module must be wired on another PL-side UART IP to communicate. It also integrates two red LEDs that blinks on transmission/reception of data, and a jumper that select the power source (USB or Pmod).
The following board and extension module is used only in the bonus part of the tutorial. You may safely skip it for now.
The ZCU104 is an mid-range (~2000€) board manufactured by AMD (formerly Xilinx). It features a ZYNQ Ultrascale+ XCZU7EV-2FFVC1156 MPSoC and sufficient IO for various projects (HDMI in/out, ethernet, M.2 socket, DDR4 SO-DIMM socket, 3 dual PMOD connectors, FMC connector). The ZCU104 integrates four Cortex-A53 cores running at 1,2 GHz, a FPGA of 504K logic cells, 38 MiB of block RAM and 1,728 DSP slices; and a total of 2 GiB MiB of PS-side DDR4 (shareable with the PL, and extensible on PL-side using the SO-DIMM socket).
The reference manual of the ZCU104 is available here. Once again, download it as we will use it later.
The ZCU104 is big enough to fit one 64-bit CVA6 core with its debug module as well as plenty of peripherals. We will use use it for Ethernet expansion as well as frequency modification.
FPGA Mezzanine Card (FMC) is an ANSI/VITA (VMEbus International Trade Association) 57.1 standard that defines I/O mezzanine modules with connection to an FPGA or other device with re-configurable I/O capability. (Wikipedia)
The FMC interface is design for more bandwidth-hungry expansions of FPGAs, such as networking cards. Our ZCU104 board integrates the LPC (Low Pin Count) variant of the FMC module. It is physically compatible with the EVAL-CN0506-FMCZ add-in card providing dual-PHY Ethernet interface. However, in practice, one should be careful about the supplied voltage - 1.2 V only - when the ZCU104 FMC connector can supply 0.8 V, 1.0 V and 1.2 V. Recompiling the FSBL (First Stage BootLoader) to support the correct voltage is necessary in order to have the module initializing, which is (thankfully) already done in our case.
References designs as well as documentation is available here, with design and integration files downloadable here. Once again, keep these files as we will use them later.
In the bonus section of this tutorial, we will use the EVAL-CN0506-FMCZ to add networking capabilities to our CVA6 softcore through the use of one Ethernet port.