bluefield-bfb-install

Posted on In

bluefield bfb install

ref: https://docs.nvidia.com/doca/sdk/installation-guide/index.html

1. install rshim and pv

install rshim:

1
dpkg -i rshim_2.0.6-3.ge329c69_amd64.deb

output:

1
2
3
4
5
6
7
Selecting previously unselected package rshim.
(Reading database ... 71752 files and directories currently installed.)
Preparing to unpack rshim_2.0.6-3.ge329c69_amd64.deb ...
Unpacking rshim (2.0.6-3.ge329c69) ...
Setting up rshim (2.0.6-3.ge329c69) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rshim.service → /lib/systemd/system/rshim.service.
Processing triggers for man-db (2.9.1-1) ...

install pv:

1
apt install -y pv

2 start and check rshim service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
systemctl is-enabled rshim
enabled

systemctl start rshim

systemctl status rshim
● rshim.service - rshim driver for BlueField SoC
     Loaded: loaded (/lib/systemd/system/rshim.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-12-17 07:01:48 UTC; 14min ago
       Docs: man:rshim(8)
    Process: 10663 ExecStart=/usr/sbin/rshim $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 10666 (rshim)
      Tasks: 6 (limit: 629145)
     Memory: 2.2M
     CGroup: /system.slice/rshim.service
             └─10666 /usr/sbin/rshim

Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Starting rshim driver for BlueField SoC...
Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Started rshim driver for BlueField SoC.
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2
Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached
Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open
Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close

2.1 check rshim

1
2
3
4
5
6
7
# ls -lh /dev/rshim0/
total 0
crw------- 1 root root 235, 0 Dec 17 07:01 boot
crw------- 1 root root 234, 0 Dec 17 07:01 console
crw------- 1 root root 511, 0 Dec 17 07:01 misc
crw------- 1 root root 236, 0 Dec 17 07:01 rshim

1
2
3
4
5
6
7
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2
Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached
Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open
Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close

pcie-0000:3b:00.2  --> rshim0

3 install bfb to bluefield NIC

3.1 create login password

The password is lab123 

1
echo "ubuntu_PASSWORD='$(openssl passwd -1 lab123)'" | tee bf.cfg

3.2 install bfb to NIC

1
bfb-install --rshim <rshimN> --bfb <image_path.bfb> --config bf.cfg

example:

1
bfb-install --rshim rshim0 --bfb DOCA_v1.2.0_BlueField_OS_Ubuntu_20.04-5.4.0-1022-bluefield-5.5-1.0.3.2-3.8.0.11969-1.signed-aarch64.bfb --config bf.cfg

output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Pushing bfb + cfg
 623MiB 0:00:54 [11.4MiB/s] [                                       <=>                                               ]
Collecting BlueField booting status. Press Ctrl+C to stop…
 INFO[BL2]: start
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: runtime
 INFO[UEFI]: UPVS valid
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: PMI: updates started
 INFO[UEFI]: PMI: boot image update
 INFO[UEFI]: PMI: updates completed, status 0
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[MISC]: Found bf.cfg
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installing OS image
 INFO[MISC]: Changing the default password for user ubuntu
 INFO[MISC]: Installation finished

4 connect bluefield nic by console port (open another session)

1
screen /dev/rshim0/console

console output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
INFO: Changing the default password for user ubuntu
Boot0000* focal
INFO: Installation finished
INFO: Rebooting...




Mellanox BlueField-2 A1 BL1 V1.1
NOTICE:  BL2R: v2.2(release):3.8.0-25-ge146e67
NOTICE:  BL2R: Built : 15:12:50, Nov 30 2021
NOTICE:  BL2R built for hw (ver 1)
NOTICE:  BL2R: Booting BL2
NOTICE:  BL2: v2.2(release):3.8.0-25-ge146e67
NOTICE:  BL2: Built : 15:12:50, Nov 30 2021
NOTICE:  BL2 built for hw (ver 1)
NOTICE:  Running as MBF2H332A-AECO system
NOTICE:  No SPD detected on MSS0 DIMM0
NOTICE:  No SPD detected on MSS0 DIMM1
NOTICE:  Finished initializing DDR
NOTICE:  DDR POST passed.
NOTICE:  BL31: v2.2(release):3.8.0-25-ge146e67
NOTICE:  BL31: Built : 15:12:50, Nov 30 2021
NOTICE:  BL31 built for hw (ver 1)
UEFI firmware (version BlueField:3.8.0-34-gb1e3ae0 built at 15:15:10 on Nov 30 2021)
Press <ESC> twice to enter UEFI menu
3 seconds remaining
2 seconds remaining
1 seconds remaining
0
[    0.608703] rtc-efi rtc-efi: hctosys: unable to read the hardware clock
[   10.715782] mlx5_core 0000:03:00.0: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): hash based LAG is not supported by current device
[   12.861453] mlx5_core 0000:03:00.1: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): hash based LAG is not supported by current device

Ubuntu 20.04.3 LTS localhost hvc0

localhost login:

The login username is ubuntu. The password is lab123

5 connect bluefield use by ssh

5.1 config tmfifo_net0

1
ip add add 192.168.100.1/30 dev tmfifo_net0

check interface

1
2
3
4
5
6
7
8
9
10
11
root@ubuntu2004-lab1:~# ip address show tmfifo_net0
16: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/30 scope global tmfifo_net0
       valid_lft forever preferred_lft forever
    inet6 fe80::21a:caff:feff:ff02/64 scope link
       valid_lft forever preferred_lft forever

root@ubuntu2004-lab1:~# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=2.46 ms

connect bluefield by ssh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@ubuntu2004-lab1:~# ssh -l ubuntu 192.168.100.2
The authenticity of host '192.168.100.2 (192.168.100.2)' can't be established.
ECDSA key fingerprint is SHA256:QqO4i9YSGBI6pp1N8JJKqnTFokEEn1JVh+sdxcYPIb4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.2' (ECDSA) to the list of known hosts.
ubuntu@192.168.100.2's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-1022-bluefield aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Tue Sep  7 18:44:49 UTC 2021

  System load:  0.11              Processes:                    231
  Usage of /:   5.1% of 57.10GB   Users logged in:              1
  Memory usage: 5%                IPv4 address for tmfifo_net0: 192.168.100.2
  Swap usage:   0%


0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings