SIFIVE FREEDOM ON VC707

Hoang Trong Thuc - UEC, Tokyo, Japan

SIFIVE FREEDOM ON VC707

I. Hardware

Original repo. Modified repo

I. a) Build

Git clone:

$ git clone https://github.com/thuchoang90/freedom.git	
$ cd freedom/
$ git checkout keystone-fpga		#commit 76dd152a on 5-Nov-2019

this will 'submodule update' without download the toolchain again, and also patch the fpga-shells submodule afterward
$ . update.sh

(i) Make

Before ‘make’, remember to set the environment (see I. c) Other Utilities) simply by: $ . setenv.sh

The format for ‘make’ command is:

This is for creating .v files:
$ make [CONFIG] [MODEL] [BOOTROM_DIR] [BUILD_DIR] -f Makefile.vc707-u500devkit verilog -j`nproc`

This is for Vivado build:
$ make [CONFIG] [MODEL] [BOOTROM_DIR] [BUILD_DIR] -f Makefile.vc707-u500devkit mcs -j`nproc`

The [CONFIG] option is for selecting the frequency:

CONFIG=DevKitU500FPGADesign_WithDevKit50MHz (default value)
       DevKitU500FPGADesign_WithDevKit100MHz
       DevKitU500FPGADesign_WithDevKit125MHz
       DevKitU500FPGADesign_WithDevKit150MHz

The [MODEL] option is for selecting the PCIe option:

with PCIe build (default value):
MODEL=VC707PCIeShell

without PCIe build:
MODEL=VC707BaseShell

The [BOOTROM_DIR] option is to specify the bootrom directory that you want to use:

using the sifive's sdboot (default value):
BOOTROM_DIR=`pwd`/bootrom/sdboot

using the keystone's zsfl+fsbl boot:
BOOTROM_DIR=`pwd`/bootrom/freedom-u540-c000-bootloader

The [BUILD_DIR] option is to specify the build directory:

BUILD_DIR=`pwd`/builds/<name>
default value: BUILD_DIR=`pwd`/builds/vc707-u500devkit

Example:

To build with PCIe, 125MHz, using sifive's sdboot:
$ make CONFIG=DevKitU500FPGADesign_WithDevKit125MHz BUILD_DIR=`pwd`/builds/vc707-u500devkit-withpcie-sdboot -f Makefile.vc707-u500devkit verilog -j`nproc`
$ make CONFIG=DevKitU500FPGADesign_WithDevKit125MHz BUILD_DIR=`pwd`/builds/vc707-u500devkit-withpcie-sdboot -f Makefile.vc707-u500devkit mcs -j`nproc`

To build without PCIe, 150MHz, using sifive's sdboot:
$ make CONFIG=DevKitU500FPGADesign_WithDevKit150MHz MODEL=VC707BaseShell BUILD_DIR=`pwd`/builds/vc707-u500devkit-nopcie-sdboot -f Makefile.vc707-u500devkit verilog -j`nproc`
$ make CONFIG=DevKitU500FPGADesign_WithDevKit150MHz MODEL=VC707BaseShell BUILD_DIR=`pwd`/builds/vc707-u500devkit-nopcie-sdboot -f Makefile.vc707-u500devkit mcs -j`nproc`

To build with PCIe, 125MHz, using keystone's zsbl-fsbl boot:
$ make CONFIG=DevKitU500FPGADesign_WithDevKit125MHz BOOTROM_DIR=`pwd`/bootrom/freedom-u540-c000-bootloader BUILD_DIR=`pwd`/builds/vc707-u500devkit-withpcie-keystoneboot -f Makefile.vc707-u500devkit verilog -j`nproc`
$ make CONFIG=DevKitU500FPGADesign_WithDevKit125MHz BOOTROM_DIR=`pwd`/bootrom/freedom-u540-c000-bootloader BUILD_DIR=`pwd`/builds/vc707-u500devkit-withpcie-keystoneboot -f Makefile.vc707-u500devkit mcs -j`nproc`

To build without PCIe, 150MHz, using keystone's zsbl-fsbl boot:
$ make CONFIG=DevKitU500FPGADesign_WithDevKit150MHz MODEL=VC707BaseShell BOOTROM_DIR=`pwd`/bootrom/freedom-u540-c000-bootloader BUILD_DIR=`pwd`/builds/vc707-u500devkit-nopcie-keystoneboot -f Makefile.vc707-u500devkit verilog -j`nproc`
$ make CONFIG=DevKitU500FPGADesign_WithDevKit150MHz MODEL=VC707BaseShell BOOTROM_DIR=`pwd`/bootrom/freedom-u540-c000-bootloader BUILD_DIR=`pwd`/builds/vc707-u500devkit-nopcie-keystoneboot -f Makefile.vc707-u500devkit mcs -j`nproc`

To clean and build again:

(ii) Notes

1: The maximum frequency for the VC707 board with PCIE is 125MHz, and without PCIE is 150MHz.

2: Sometime the make mcs end with timing error and did not continue to generate the final mcs files for the flash programming, but still, it did generate the .bit file. Then, we can manually generate the .mcs from the .bit:

cd to the build folder
$ cd builds/<name>/obj/

for VC707PCIeShell:
$ vivado -nojournal -mode batch -source ../../../fpga-shells/xilinx/common/tcl/write_cfgmem.tcl -tclargs vc707 VC707PCIeShell.mcs VC707PCIeShell.bit

for VC707BaseShell:
$ vivado -nojournal -mode batch -source ../../../fpga-shells/xilinx/common/tcl/write_cfgmem.tcl -tclargs vc707 VC707BaseShell.mcs VC707BaseShell.bit

3: Built files are under **builds//obj/**. The important built files are:

VC707Shell.v							the verilog source code
VC707Shell.mcs and VC707Shell.prm		the two files for flash programming
VC707Shell.bit							the bitstream file for direct programming

I. b) Program the board

Remember to switch the switches above the LCD to UP-UP-DOWN-UP-DOWN, then open vivado, open hardware manager, open target board, auto connect.

About the cable driver:

$ cd to [installation path]/Vivado/2016.4/data/xicom/cable_drivers/lin64/install_script/install_drivers
then run the script "$ sudo ./install_drivers"

Vivado direct programming:

a. right-click on the xc7vx485t_0
b. Program Device ...
c. select the .bit file in the builds/vc707-u500devkit/obj/VC707Shell.bit
d. Program

Vivado flash programming:

a. right-click on the xc7vx485t_0
b. Add Configuration Memory Device ...
c. select the one with the Alias of 28f00ag18f ---> OK
d. OK to continue to program the device
e. select the Configuration file (.mcs) and PRM file (.prm) in the builds/vc707-u500devkit/obj/
f. select the RS pins: 25:24
g. OK to write data to the flash
h. finally, right-click again on the xc7vx485t_0 and select Boot from Configuration Memory Device

I. c) Debug with GDB via JTAG

Using the Olimex JTAG debugger. Ref link.

(i) Driver

To install the driver for the debugger:

$ sudo apt-get install libftdi-dev libftdi1
$ vi /etc/udev/rules.d/olimex-arm-usb-tiny-h.rules

Then add this single line in the olimex-arm-usb-tiny-h.rules file:

SUBSYSTEM=="usb", ACTION=="add", ATTRS{idProduct}=="002a", ATTRS{idVendor}=="15ba", MODE="664", GROUP="plugdev"

(ii) Connection

Connect your Olimex JTAG debugger to the VC707 FPGA board by the XADC (J19) header as shown as follows:

Branching

The four data pins TDI (pin 5), TMS (pin 7), TCLK (pin 9), and TDO (pin 13) are connected to the XADC_GPIO 0 to 3 (pin 17 to 20). The VCC (pin 1) is connected to the XADC_VCC_HEADER (pin 14). And any of the GND pin on the JTAG side is connected to the GND pin 16 of the XADC header.

Branching

(iii) Run

Copy this file to your riscv-openocd/ folder. Then:

$ cd to your riscv-openocd/ folder
$ openocd -f openocd.cfg

If succeed, it will print something like this:

$ openocd -f openocd.cfg 
Open On-Chip Debugger 0.10.0+dev-00824-gf93ede540 (2019-11-05-10:20)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
Info : clock speed 10000 kHz
Info : JTAG tap: riscv.cpu tap/device found: 0x20000913 (mfg: 0x489 (SiFive Inc), part: 0x0000, ver: 0x2)
Info : datacount=2 progbufsize=16
Info : Disabling abstract command reads from CSRs.
Info : Examined RISC-V core; found 4 harts
Info :  hart 0: XLEN=64, misa=0x800000000014112d
Info :  hart 1: XLEN=64, misa=0x800000000014112d
Info :  hart 2: XLEN=64, misa=0x800000000014112d
Info :  hart 3: XLEN=64, misa=0x800000000014112d
Info : Listening on port 3333 for gdb connections
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections

Then, open another terminal to start a GDB session: (remember to export the riscv toolchain to your PATH)

$ riscv64-unknown-elf-gdb
The GDB terminal will appear after this.

From the GDB terminal, type:
$ target extended-remote localhost:3333
If succeed, the CPU will enter the debugging state after this command,
and on the GDB terminal you can see the current instruction address

Some useful tips for debugging the RISC-V CPU:

Run next line:			$ si
Run continue:			$ c
Set breakpoint:		$ hb 0x...
Delete breakpoint:		$ delete
Print cores info:		$ info threads
Select core:			$ thread i			# i is the core number
Set program counter:	$ set $pc=0x...
Print registers table:	$ info registers
Write to address:		$ set *0x...=0x...
Read from address:		$ print/x *0x...
Reset CPU:			$ monitor reset halt

I. d) Other utilities

To quickly set build environment:

$ vi setenv.sh
and change the correct corresponding paths in your machine

Then from this point forward, you can auto set your environment by simply:
$ . setenv.sh

If you want to change the ROM size:

BootRom size is declared in the file:
	rocket-chip/src/main/scala/devices/tilelink/MaskROM.scala

On the line 12:
	case class MaskROMParams(address: BigInt, name: String, depth: Int = 2048, width: Int = 32)

Change the <depth> value to change the size.

If you want to change the modules’ addresses, number of CPU cores, etc:

The declarations are in the file:
	src/main/scala/unleashed/DevKitConfigs.scala

But remember to change the addresses on the software as well. These files:
	bootrom/sdboot/include/platform.h
	bootrom/sdboot/linker/memory.lds
	bootrom/freedom-u540-c000-bootloader/sifive/platform.h
	bootrom/freedom-u540-c000-bootloader/memory_vc707.lds

TODO: script to auto update ROM size, modules’ addresses, and number of CPU to software

I. e) Use with Idea IntelliJ

Idea IntelliJ is a tool for compiling scala codes. It much more like a GUI for the SBT tool.

After download the Idea IntelliJ from the website, extract it and run it by ./idea.sh

To import the freedom folder to the Idea IntelliJ tool:

java -jar /home/ubuntu/project/freedom/rocket-chip/sbt-launch.jar ++2.12.4 "runMain freechips.rocketchip.system.Generator /home/ubuntu/project/freedom/builds/chip uec.nedo.chip ChipShell uec.nedo.chip ChipDesignTop"
(the content may be differ on your machine)

In the first time 'make', it usually appears near after this line:
	[success] Total time: xxx s, completed xxx, xxx, xx:xx:xx
In the NOT first time 'make', it usually appears right away

To debug with the Idea IntelliJ tool:

II. Software

II. a) Without KeyStone

Original repo. Modified repo.

TODO: currently, this way uses the prebuilt toolchain of gcc 7. In the future, this need to be upgraded to the current mainstream toolchain of sifive-freedom, gcc 9.2. Then after that, we can avoid to download the prebuilt toolchain again.

(i) Git clone

$ git clone https://github.com/mcd500/freedom-u-sdk.git
$ cd freedom-u-sdk
$ git checkout linux_u500vc707devkit_config		#commit 29fe529f on 12-Apr-2019
$ git submodule update --init --recursive

Note: sometimes the riscv-gnu-toolchain submodule fails to clone. If that happened, we have to git clone it manually:

$ rm -rf riscv-gnu-toolchain		#first, remove the corrupted folder
$ git clone https://github.com/riscv/riscv-gnu-toolchain.git	#then clone a new
$ cd riscv-gnu-toolchain/
$ git checkout b4dae89			#switch to the Bump GCC version
$ git submodule update --init --recursive
$ cd ../						#get back to the top folder of freedom-u-sdk

(ii) Make the bbl.bin

$ unset RISCV		#make sure that there is nothing on the RISCV variable

export your toolchains to the PATH (both the elf & linux toolchains)
$ export PATH=<your riscv64 elf toolchain>/bin/:$PATH
$ export PATH=<your riscv64 linux toolchain>/bin/:$PATH

then 'make' with PCIE:
$ make -j`nproc` BOARD=vc707devkit

or 'make' without PCIE:
$ make -j`nproc` BOARD=vc707devkit_nopci

(iii) Write SD card & boot on

$ sudo dd if=work/bbl.bin of=/dev/sdX bs=4096 conv=fsync
where X is the number of the USB device

Put in the SD card to the board, program the board, then wait for the board to boot on. Communicate with the board via UART:

$ sudo minicom -b 115200 -D /dev/ttyUSBx
where x is the number of connected USB-UART
Login by the id of 'root' and the password of 'sifive'.

II. b) SD card with KeyStone

Ref link for the KeyStone project.

(i) Partition the SD card

Use the gptfdisk tool to create 4 partitions in the SD card.

If you don’t have the gptfdisk tool, then do the followings to install it:

$ git clone https://github.com/tmagik/gptfdisk.git	#branch master commit 3d6a1587 on 9-Dec-2018
$ cd gptfdisk/
$ make -j`nproc`

Copy the mksd-sifive.sh file to your gptfdisk directory, then do this to partition your SD card:

$ ./mksd-sifive.sh /dev/sdX 8G
where X is the number of the USB device, and you can change the 8G to 16G if your SD card is 16G

The partitions after created:

Number	Start (sector)	End (sector)	Size			Code	Name
1		2048		67583		32.0 MiB		5202	SiFive bare-metal (...
2		264192		15759326	7.4 GiB		8300	Linux filesystem
4		67584		67839		128.0 KiB		5201	SiFive FSBL (first-...

(ii) Prepare BBL & FSBL

For the bbl.bin:

Follow the instruction to make the bbl.bin of the Keystone & Keystone demo.

The bbl.bin is for the 1st partition of the SD card:

$ cd <keystone folder>		#cd to your keystone folder
$ sudo dd if=hifive-work/bbl.bin of=/dev/sdX1 bs=4096 conv=fsync
where the X1 is the 1st partition of the USB device

For the fsbl.bin:

After the hardware make (section I. a) above), there is a vc707fsbl.bin file inside the folder bootrom/freedom-u540-c000-bootloader/. That file is for the 4th partition of the SD card:

$ cd <your freedom folder>
$ cd bootrom/freedom-u540-c000-bootloader/
$ sudo dd if=vc707fsbl.bin of=/dev/sdX4 bs=4096 conv=fsync
where the X4 is the 4th partition of the USB device

(iii) Boot on & run the test

Finally, put in the SD card to the board, program the board, then wait for the board to boot on. Communicate with the board via UART:

$ sudo minicom -b 115200 -D /dev/ttyUSBx
where x is the number of connected USB-UART
Login by the id of 'root' and the password of 'sifive'.

To run the keystone test:

$ insmod keystone-driver.ko		#install driver
$ time ./tests/tests.ke 			#okay if the 'Attestation report SIGNATURE is valid' is printed
$ cd keystone-demo/			#go to the keystone-demo test
$ ./enclave-host.riscv &			#run host in localhost
$ ./trusted_client.riscv localhost	#connect to localhost and test
okay if the 'Attestation signature and enclave hash are valid' is printed

BOTTOM PAGE

Back Next
Keystone Guide Scala Study Notes