0%

For GL.inet box

Requirements

You need the following tools to compile OpenWrt, the package names vary between distributions. A complete list with distribution specific packages is found in the Build System Setup documentation

OS

  • Ubuntu >= 18.04 (16.04 python3 is 3.5, need > python 3.6)
  • Debian >= 10

packages install

1
2
3
4
5
6
sudo apt update -y
sudo apt install build-essential ccache ecj fastjar file g++ gawk \
gettext git java-propose-classpath libelf-dev libncurses5-dev \
libncursesw5-dev libssl-dev python python2.7-dev python3 unzip wget \
python3-distutils python3-setuptools python3-dev rsync subversion \
swig time xsltproc zlib1g-dev -y

Quickstart

Clone repository

1
git clone https://github.com/gl-inet/gl-infra-builder.git
1
cd gl-infra-builder

Download OpenWRT 21.02

Setup, the command will auto download openwrt-21.02 by default and auto config, and then patch all the the GL product Patches

1
python3 setup.py -c config-21.02.0.yml
1
2
3
4
5
6
7
8
9
$ more config-21.02.0.yml
repo: https://github.com/openwrt/openwrt.git
branch: v21.02.0
git_clone_dir: openwrt-21.02/openwrt-21.02.0
openwrt_root_dir: openwrt-21.02/openwrt-21.02.0
revision: b2ae4233149dfd78f2ac00bb5327695bcacdc255

patch_folders:
- patches-21.02.0/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
~/gl-infra-builder/patches-21.02.0$ ls -l
total 172
-rw-r--r-- 1 lab lab 15930 Jan 29 16:47 0001-ath79-add-support-for-gl-xe300.patch
-rw-r--r-- 1 lab lab 49979 Jan 29 16:47 0002-ath79-add-support-for-gl-mifi.patch
-rw-r--r-- 1 lab lab 14045 Jan 29 16:47 0003-add-modem-support.patch
-rw-r--r-- 1 lab lab 3514 Jan 29 16:47 0004-add-RS485-support.patch
-rw-r--r-- 1 lab lab 3481 Jan 29 16:47 0005-add-spinand-support.patch
-rw-r--r-- 1 lab lab 2223 Jan 29 16:47 0006-led-netdev-trigger-by-wwanx.patch
-rw-r--r-- 1 lab lab 7408 Jan 29 16:47 0007-ath79-add-support-for-gl-ar300m.patch
-rw-r--r-- 1 lab lab 8408 Jan 29 16:47 0008-ath79-add-support-for-gl-x300b.patch
-rw-r--r-- 1 lab lab 6330 Jan 29 16:47 0009-ath79-add-support-for-gl-ar150.patch
-rw-r--r-- 1 lab lab 8705 Jan 29 16:47 0010-ath79-add-support-for-gl-usb150.patch
-rw-r--r-- 1 lab lab 7017 Jan 29 16:47 0011-add-support-gl-ar750-ar750s.patch
-rw-r--r-- 1 lab lab 13400 Jan 29 16:47 0012-ath79-add-support-for-gl-x750.patch
-rw-r--r-- 1 lab lab 11790 Jan 29 16:47 0013-support-fast-forward.patch

Update feeds and install frr vxlan feeds

1
./scripts/feeds update -a
1
./scripts/feeds install frr vxlan libpam liblzma libnetsnmp

Make config file

create the default config file

1
make defconfig

make config file. select which kernel model and package you need to add to OpenWRT

1
make menuconfig

Select target system

1644454983595.png

Kernel model

1
2
3
4
Kernel modules ---> Network Support --->
enable kmod-vlxan
enable kmod-veth
enable kmod-tun

1644455181686.png

Add frr

1
Network ---> Routing and Redirection ---> frr

1644455358487.png

add some tools

1
Network ---> Routing and Redirection
1
2
3
4
enable ip-bridge ## MUST vxlan forwording plane will use this tool
enable ip-full ## MUST create vxlan interface will use this tool
enable ss
enable tc-full

add vxlan

1
Network --->
1
enable vxlan ## MUST vxlan forwording plane will use this script

1644455697606.png

Make the image

1
make -j$(nproc) V=s

1. Topology

1644369044061.png

GL.inet MT1300 is MT7621 chip

1644372818557.png

2. device config

2.1 QFX5100 config

2.1.1 interface config

1
2
3
4
5
6
7
8
9
10
11
# L2 interface 
set xe-0/0/0 unit 0 family ethernet-switching interface-mode access
set xe-0/0/0 unit 0 family ethernet-switching vlan members vlan-vxlan10

# connect L2 switch
set xe-0/0/37 unit 0 family ethernet-switching interface-mode trunk
set xe-0/0/37 unit 0 family ethernet-switching vlan members vlan81

# vtep source interface
set irb unit 81 family inet address 11.81.1.1/24
set lo0 unit 0 family inet address 11.1.1.1/32

vlan-vxlan10 is vxlan vlan

2.1.2 protocol config

1
2
3
4
5
6
7
8
9
10
11
set routing-options router-id 11.1.1.1
set routing-options autonomous-system 65000
set protocols bgp group evpn type internal
set protocols bgp group evpn local-address 11.1.1.1
set protocols bgp group evpn family evpn signaling
set protocols bgp group evpn allow 11.81.1.0/24
set protocols bgp group evpn allow 11.82.1.0/24
set protocols evpn encapsulation vxlan
set protocols evpn vni-options vni 10 vrf-target target:65000:10
set protocols evpn extended-vni-list 10

2.1.3 switch and vlan config

1
2
3
4
5
6
7
8
9
set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 11.1.1.1:10
set switch-options vrf-target target:65000:10
set vlans vlan-vxlan10 vlan-id 10
set vlans vlan-vxlan10 vxlan vni 10
set vlans vlan-vxlan10 vxlan ingress-node-replication
set vlans vlan81 description "openwrt test"
set vlans vlan81 vlan-id 81
set vlans vlan81 l3-interface irb.81

2.2 OpenWRT config

2.2.1 FRR config

1
2
3
4
5
6
7
8
9
10
11
12
13
router bgp 65000
bgp router-id 11.81.1.2
neighbor evpn peer-group
neighbor evpn remote-as internal
neighbor evpn update-source wan
neighbor 11.1.1.1 peer-group evpn
!
address-family l2vpn evpn
neighbor evpn activate
advertise-all-vni
advertise ipv4 unicast
exit-address-family

2.2.2 vxlan interface config

The OpenWRT config (/etc/config/network) must need vxlan peeraddr.

ref: https://openwrt.org/docs/guide-user/network/tunneling_interface_protocols

1644369624624.png

so, use hotplug to add the vxlan interface.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@OpenWrt:~# cat /etc/hotplug.d/iface/50-vxlan
#!/bin/sh

[ "$ACTION" == "ifup" ] && [ "$INTERFACE" == "wan" ] && {

WAN_IFNAME='wan@eth0'
WAN_IP="$(ip -br a | grep "$WAN_IFNAME" | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')"

ip link del vxlan1
ip link add vxlan1 type vxlan id 10 local ${WAN_IP} dstport 4789
ip link set vxlan1 master br-lan
ip link set vxlan1 up

}

2.2.3 firewall config

The OpenWRT default firewall is disable the VxLAN traffic. so we need add the firewall rule. JUNOS will use 4789 to connect the VxLAN.

1
2
3
4
5
6
7
# … in the /etc/config/firewall
config rule
option name Allow-VxLAN
option src wan
option dest_port 4789
option proto udp
option target ACCEPT

if we need another BGP speaker to connect active. we need to add the BGP 179(TCP) port in the firewall rule. In my use case, the JUNOS BGP speaker is passive mode.

1
2
3
4
5
6
7
8
# … in the /etc/config/firewall
config rule
option name Allow-BGP
option src wan
option dest_port 179
option proto tcp
option target ACCEPT

1644370121708.png

3. vxlan evpn status

3.1 QFX5100

3.1.1 BGP (control plane)

BGP peer is evpn.

1
2
3
4
5
6
7
8
9
10
11
12
lab@qfx5100-48s-6q> show bgp summary
Threading mode: BGP I/O
Groups: 1 Peers: 1 Down peers: 0
Unconfigured peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
2 2 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
11.81.1.2 65000 80 70 0 0 28:40 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 2/2/2/0
default-switch.evpn.0: 2/2/2/0

BGP evpn NLRI

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
lab@qfx5100-48s-6q> show route table bgp.evpn.0

bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2:11.1.1.1:10::10::e4:43:4b:f6:c5:56/304 MAC/IP
*[EVPN/170] 00:39:07
Indirect
2:11.81.1.2:2::0::e4:43:4b:f6:c5:75/304 MAC/IP
*[BGP/170] 00:45:24, localpref 100
AS path: I, validation-state: unverified
> to 11.81.1.2 via irb.81
2:11.1.1.1:10::10::e4:43:4b:f6:c5:56::192.168.8.100/304 MAC/IP
*[EVPN/170] 00:19:13
Indirect
2:11.81.1.2:2::0::e4:43:4b:f6:c5:75::192.168.8.10/304 MAC/IP
*[BGP/170] 00:00:19, localpref 100
AS path: I, validation-state: unverified
> to 11.81.1.2 via irb.81
3:11.1.1.1:10::10::11.1.1.1/248 IM
*[EVPN/170] 00:46:26
Indirect
3:11.81.1.2:2::0::11.81.1.2/248 IM
*[BGP/170] 00:45:24, localpref 100
AS path: I, validation-state: unverified
> to 11.81.1.2 via irb.81

One evpn BGP NLRI extensive information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
lab@qfx5100-48s-6q> show route table bgp.evpn.0 extensive evpn-mac-address e4:43:4b:f6:c5:75

bgp.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
2:11.81.1.2:2::0::e4:43:4b:f6:c5:75/304 MAC/IP (1 entry, 0 announced)
*BGP Preference: 170/-101
Route Distinguisher: 11.81.1.2:2
Next hop type: Indirect, Next hop index: 0
Address: 0xccd60b0
Next-hop reference count: 4
Source: 11.81.1.2
Protocol next hop: 11.81.1.2
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
State: <Active Int Ext>
Local AS: 65000 Peer AS: 65000
Age: 17:50:15 Metric2: 0
Validation State: unverified
Task: BGP_65000.11.81.1.2+43211
AS path: I
Communities: target:65000:10 encapsulation:vxlan(0x8)
Import Accepted
Route Label: 10
ESI: 00:00:00:00:00:00:00:00:00:00
Localpref: 100
Router ID: 11.81.1.2
Secondary Tables: default-switch.evpn.0
Indirect next hops: 1
Protocol next hop: 11.81.1.2
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: 11.81.1.2 via irb.81
Session Id: 0x0
11.81.1.0/24 Originating RIB: inet.0
Node path count: 1
Forwarding nexthops: 1
Next hop type: Interface
Nexthop: via irb.81

3.1.2 forwarding plane

mac-ip table

1
2
3
4
5
6
7
8
9
10
11
12
13
{master:0}
lab@qfx5100-48s-6q> show ethernet-switching mac-ip-table

MAC IP flags (S - Static, D - Dynamic, L - Local , R - Remote, Lp - Local Proxy,
Rp - Remote Proxy, K - Kernel, RT - Dest Route, AD - Advt to remote,
RE - Re-ARP/ND, RO - Router, OV - Override)
Routing instance : default-switch
Bridging domain : vlan-vxlan10
IP MAC Flags Logical Active
address address Interface source
192.168.8.100 e4:43:4b:f6:c5:56 DL,K,AD xe-0/0/0.0
192.168.8.10 e4:43:4b:f6:c5:75 DR,K vtep.32769 11.81.1.2

MAC table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{master:0}
lab@qfx5100-48s-6q> show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 3 entries, 3 learned
Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
vlan-vxlan10 e4:43:4b:f6:c5:56 D xe-0/0/0.0
vlan-vxlan10 e4:43:4b:f6:c5:75 D vtep.32769 11.81.1.2
vlan81 94:83:c4:16:51:44 D xe-0/0/37.0

{master:0}

forwarding-table detail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{master:0}
lab@qfx5100-48s-6q> show route forwarding-table family ethernet-switching matching e4:43:4b:f6:c5:75/48 extensive
Routing table: default-switch.evpn-vxlan [Index 7]
Bridging domain: vlan-vxlan10.evpn-vxlan [Index 3]
VPLS:
Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN,

Destination: e4:43:4b:f6:c5:75/48
Learn VLAN: 0 Route type: user
Route reference: 0 Route interface-index: 566
Multicast RPF nh index: 0
P2mpidx: 0
IFL generation: 131 Epoch: 0
Sequence Number: 0 Learn Mask: 0x4000000000000000000000000000000000000000
L2 Flags: control_dyn
Flags: sent to PFE
Nexthop:
Next-hop type: composite Index: 1742 Reference: 5
Next-hop type: indirect Index: 524286 Reference: 3
Nexthop: 11.81.1.2
Next-hop type: unicast Index: 1741 Reference: 4
Next-hop interface: xe-0/0/37.0

3.2 OpenWRT

3.2.1 BGP

BGP peer

1
2
3
4
5
6
7
8
9
10
11
12
13
# frr command
OpenWrt# show bgp l2vpn evpn summ
BGP router identifier 11.81.1.2, local AS number 65000 vrf-id 0
BGP table version 0
RIB entries 3, using 384 bytes of memory
Peers 1, using 10896 bytes of memory
Peer groups 1, using 32 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
11.1.1.1 4 65000 3090 3048 0 0 0 17:27:52 3 3

Total number of neighbors 1

evpn BGP NLRI

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# frr command
OpenWrt# show bgp l2vpn evpn
BGP table version is 15, local router ID is 11.81.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 11.1.1.1:10
*>i[2]:[10]:[48]:[e4:43:4b:f6:c5:56]
11.1.1.1 100 0 i
RT:65000:10 ET:8
*>i[2]:[10]:[48]:[e4:43:4b:f6:c5:56]:[32]:[192.168.8.100]
11.1.1.1 100 0 i
RT:65000:10 ET:8
*>i[3]:[10]:[32]:[11.1.1.1]
11.1.1.1 100 0 i
RT:65000:10 ET:8
Route Distinguisher: 11.81.1.2:2
*> [2]:[0]:[48]:[e4:43:4b:f6:c5:75]
11.81.1.2 32768 i
ET:8 RT:65000:10
*> [2]:[0]:[48]:[e4:43:4b:f6:c5:75]:[32]:[192.168.8.10]
11.81.1.2 32768 i
ET:8 RT:65000:10
*> [3]:[0]:[32]:[11.81.1.2]
11.81.1.2 32768 i
ET:8 RT:65000:10

Displayed 6 out of 6 total prefixes


evpn BGP advertised-routes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# frr command
OpenWrt# show bgp l2vpn evpn neighbors 11.1.1.1 advertised-routes
BGP table version is 0, local router ID is 11.81.1.2
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 11.81.1.2:2
*> [2]:[0]:[48]:[e4:43:4b:f6:c5:75]
100 32768 i
*> [2]:[0]:[48]:[e4:43:4b:f6:c5:75]:[32]:[192.168.8.10]
100 32768 i
*> [3]:[0]:[32]:[11.81.1.2]
100 32768 i

Total number of prefixes 3

3.2.2 Linux forwarding plane

bridge fdb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# linux command
root@OpenWrt:~# bridge fdb | grep -E 'vxlan|wan|lan1'
e4:43:4b:f6:c5:75 dev lan1 master br-lan
94:83:c4:16:51:45 dev lan1 vlan 1 master br-lan permanent
94:83:c4:16:51:45 dev lan1 master br-lan permanent
e4:43:4b:f6:c5:75 dev lan1 vlan 1 self
c0:42:d0:08:9a:80 dev wan vlan 1 self
1c:9c:8c:67:c3:44 dev wan vlan 1 self
1c:9c:8c:68:e4:c0 dev wan vlan 1 self
e4:43:4b:f6:c5:56 dev vxlan1 vlan 1 extern_learn master br-lan
e4:43:4b:f6:c5:56 dev vxlan1 extern_learn master br-lan
3e:dd:f3:a4:65:2b dev vxlan1 vlan 1 master br-lan permanent
3e:dd:f3:a4:65:2b dev vxlan1 master br-lan permanent
00:00:00:00:00:00 dev vxlan1 dst 11.1.1.1 self permanent
e4:43:4b:f6:c5:56 dev vxlan1 dst 11.1.1.1 self extern_learn
33:33:00:00:00:01 dev wlan1 self permanent
33:33:00:00:00:02 dev wlan1 self permanent

00:00:00:00:00:00 dev vxlan1 dst 11.1.1.1 self permanent. is add by /lib/netifd/proto/vxlan.sh

vxlan interface:

1
2
3
4
5
root@OpenWrt:~# ip -d link show vxlan1
8: vxlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 3e:dd:f3:a4:65:2b brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535
vxlan id 10 local 11.81.1.2 srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8003 port_no 0x3 designated_port 32771 designated_cost 0 designated_bridge 7fff.94:83:C4:16:51:45 designated_root 7fff.94:83:C4:16:51:45 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

linux bridge:

1
2
3
4
5
root@OpenWrt:~# brctl show
bridge name bridge id STP enabled interfaces
br-lan 7fff.9483c4165145 no lan2
vxlan1
lan1

4. Throughput

4.1 Bidirection(1450 sizes)

1644372431377.png

4.2 One direction(1450 sizes)

1644372481487.png

4.3 Bidirection(256 sizes)

1644372647021.png

4.4 One direction(256 sizes)

1644372910007.png

prepare

use console cable to connect the switch

if we don’t have the ZTP environment. we need use console port to install the switch os. and we need a network cable to connect the mgmt interface.

switch the next boot to ONIE

check the grub menuentry and switch to onie

1
2
3
4
5
6
7
grep -i "menuentry '" /boot/grub/grub.cfg|sed -r "s|--class .*$||g"|nl -v 0


0 menuentry 'Cumulus-Linux GNU/Linux'
1 menuentry 'Cumulus-Linux GNU/Linux, with Linux 4.19.0-cl-1-amd64'
2 menuentry 'Cumulus-Linux GNU/Linux, with Linux 4.19.0-cl-1-amd64 (recovery mode)'
3 menuentry ONIE {
1
2
3
grub-reboot ONIE

reboot

setup the onie network

stop onie ztp

1
2
3
onie-stop


add ip address on eth0(mgmt interface)

1
2
ip address add 100.64.31.101/24 dev eth0
ip route add default via 100.64.31.1

install the cumulus system

start a web serivce with python3

1
2
python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

install cumulus os

1
2
ONIE:/ # onie-nos-install http://100.64.21.13:8000/cumulus-linux-5.0.0-mlx-amd64
.bin

proxy ARP golang

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package main

import (
"flag"
"log"
"net"

"github.com/mdlayher/arp"
)

type cliFlag struct {
Ifname string
Debug bool
}

func main() {

var cli cliFlag
flag.StringVar(&cli.Ifname, "i", "vlan1000", "Proxy Arp interface name")
flag.BoolVar(&cli.Debug, "v", false, "show verbose informations")
flag.Parse()

ifname, err := net.InterfaceByName(cli.Ifname)

if err != nil {
log.Fatalln("get interface by name: ", err)
}
client, _ := arp.Dial(ifname)
for {
arp_recv, _, err := client.Read()

if err != nil {
log.Fatalln("read packets error: ", err)

}
if cli.Debug {
log.Println("recv arp: ", arp_recv)
}
arp_replay, err := arp.NewPacket(arp.OperationReply, ifname.HardwareAddr, arp_recv.TargetIP, arp_recv.SenderHardwareAddr, arp_recv.SenderIP)
if err != nil {
log.Fatalln("create packets error: ", err)
}
destinationMAC := arp_recv.SenderHardwareAddr
err = client.WriteTo(arp_replay, destinationMAC)
if err != nil {

log.Fatalln("sent packet error: ", err)
}
if cli.Debug {
log.Println("sent arp: ", arp_replay)
}

}

}


go.mod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
module proxy_arp

go 1.17

require github.com/google/gopacket v1.1.19

require (
github.com/mdlayher/arp v0.0.0-20191213142603-f72070a231fc // indirect
github.com/mdlayher/ethernet v0.0.0-20190606142754-0394541c37b7 // indirect
github.com/mdlayher/raw v0.0.0-20211126142749-4eae47f3d54b // indirect
golang.org/x/net v0.0.0-20211216030914-fe4d6282115f // indirect
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect
)

makefile

1
2
3
4
5
linux-arm64:	
CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o ./bin/proxy_arp.arm64 src/main.go
linux:
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -o ./bin/proxy_arp src/main.go

Cloud init guide

ref: https://cloudinit.readthedocs.io/en/latest/topics/examples.html

1. Install tools on the hypervisor machine

1.1 Install kvm qemu virt

1
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst

1.2 Install cloud init tools

1
sudo apt install -y cloud-utils

download a cloud image

1
wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img

create a vm image

1
2
qemu-img create -f qcow2 -b ubuntu-20.04-server-cloudimg-amd64.img client.qcow2
qemu-img resize client.qcow2 +10G

create a cloud init file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cat > cloud-init.yaml << EOF
#cloud-config
hostname: cloudimg.local
user: root
password: lab123
chpasswd: { expire: False }
ssh_pwauth: True
users:
- name: lab
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
groups: root, sudo
plain_text_passwd: 'lab123'
lock_passwd: false
growpart:
mode: auto
devices: ['/']
locale: en_US.UTF-8
timezone: Asia/Shanghai
EOF

create network config file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > network.yaml << EOF  
ethernets:
enp1s0:
dhcp4: false
addresses:
- 11.1.10.12/24
gateway4: 11.1.10.1
enp2s0:
addresses: [ 192.168.122.110/24 ]
#gateway4: 192.168.122.1
#nameservers:
# addresses: [114.114.114.114]
version: 2
EOF

create the cloud-init iso file

1
2
cloud-localds -m local -N network.yaml cloud-init.iso cloud-init.yaml

boot a vm by virt-install

1
2
3
4
5
6
7
8
9
10
11
12
virt-install \
--name vm1 \
--ram=8192 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu20.04 \
--graphics none \
--disk /home/lab/cloud/client.qcow2,device=disk,bus=virtio \
--disk /home/lab/cloud/cloud-init.iso,device=cdrom \
--network bridge=br10,model=virtio \
--network bridge=virbr0,model=virtio \
--import

guestfish

open a image file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
guestfish --rw -a ./ubuntu-20.04-server-cloudimg-amd64.img
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘helpfor help on commands
‘man’ to read the manual
‘quit’ to quit the shell

><fs>
><fs> run
><fs> list-filesystems
><fs> mount /dev/sda1 /
><fs> cat /etc/shadow
><fs> quit

virt-customize tools

ref: https://libguestfs.org/virt-customize.1.html

1
sudo apt install -y libguestfs-tools

command help

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
virt-customize --help
virt-customize: customize a virtual machine

virt-customize [--options] -d domname

virt-customize [--options] -a disk.img [-a disk.img ...]

A short summary of the options is given below. For detailed help please
read the man page virt-customize(1).

Options:
-a, --add <file> Add disk image file
--append-line <FILE:LINE> Append line(s) to the file
--attach <iso> Attach data disk/ISO during install
--attach-format <format> Set attach disk format
-c, --connect <uri> Set libvirt URI
--chmod <PERMISSIONS:FILE> Change the permissions of a file
--color, --colors, --colour, --colours
Use ANSI colour sequences even if not tty
--commands-from-file <FILENAME> Read customize commands from file
--copy <SOURCE:DEST> Copy files in disk image
--copy-in <LOCALPATH:REMOTEDIR> Copy local files or directories into image
-d, --domain <domain> Set libvirt guest name
--delete <PATH> Delete a file or directory
-n, --dryrun, --dry-run Perform a dry run
--echo-keys Don’t turn off echo for passphrases
--edit <FILE:EXPR> Edit file using Perl expression
--firstboot <SCRIPT> Run script at first guest boot
--firstboot-command <'CMD+ARGS'> Run command at first guest boot
--firstboot-install <PKG,PKG..> Add package(s) to install at first boot
--format <format> Set format (default: auto)
--help Display brief help
--hostname <HOSTNAME> Set the hostname
--install <PKG,PKG..> Add package(s) to install
--key <SELECTOR> Specify a LUKS key
--keys-from-stdin Read passphrases from stdin
--link <TARGET:LINK[:LINK..]> Create symbolic links
-m, --memsize <mb> Set memory size
--mkdir <DIR> Create a directory
--move <SOURCE:DEST> Move files in disk image
--network Enable appliance network (default)
--no-logfile Scrub build log file
--no-network Disable appliance network
--password <USER:SELECTOR> Set user password
--password-crypto <md5|sha256|sha512>
Set password crypto
-q, --quiet Don’t print progress messages
--root-password <SELECTOR> Set root password
--run <SCRIPT> Run script in disk image
--run-command <'CMD+ARGS'> Run command in disk image
--scrub <FILE> Scrub a file
--selinux-relabel Relabel files with correct SELinux labels
--sm-attach <SELECTOR> Attach to a subscription-manager pool
--sm-credentials <SELECTOR> Credentials for subscription-manager
--sm-register Register using subscription-manager
--sm-remove Remove all the subscriptions
--sm-unregister Unregister using subscription-manager
--smp <vcpus> Set number of vCPUs
--ssh-inject <USER[:SELECTOR]> Inject a public key into the guest
--timezone <TIMEZONE> Set the default timezone
--touch <FILE> Run touch on a file
--truncate <FILE> Truncate a file to zero size
--truncate-recursive <PATH> Recursively truncate all files in directory
--uninstall <PKG,PKG..> Uninstall package(s)
--update Update packages
--upload <FILE:DEST> Upload local file to destination
-V, --version Display version and exit
-v, --verbose Enable libguestfs debugging messages
--write <FILE:CONTENT> Write file
-x Enable tracing of libguestfs calls

1 设置root密码

1
2
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \
--root-password password:lab123

output:

1
2
3
4
5
6
7
[   0.0] Examining the guest ...
[ 4.3] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 4.4] Setting the machine ID in /etc/machine-id
[ 4.4] Setting passwords
[ 5.8] Finishing off

2 在Image中安装软件包

1
2
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \
--install net-tools,unzip

output:

1
2
3
4
5
6
[   0.0] Examining the guest ...
[ 2.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 2.3] Installing packages: net-tools unzip
[ 9.2] Finishing off

3 copy 文件到镜像中

1
2
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \
--upload ../http_test:/http_test

output:

1
2
3
4
5
6
[   0.0] Examining the guest ...
[ 2.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 2.2] Uploading: ../http_test to /http_test
[ 2.3] Finishing off

4 修改时区

1
2
3
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \
--timezone "Asia/Shanghai"

5 上传SSH公钥

1
2
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \
--ssh-inject lab:file:./id_rsa.pub

6 在文件中添加内容

1
2
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \
--append-line '/etc/hosts:10.0.0.1 foo'

7 create a user in image

1
2
3
4
5
6
virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \
--run-command 'useradd test -s /bin/bash -m'

virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \
--password test:password:lab123

output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[   0.0] Examining the guest ...
[ 2.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 2.3] Running: useradd test -s /bin/bash -m
[ 2.4] Finishing off


[ 0.0] Examining the guest ...
[ 2.2] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 2.3] Setting passwords
[ 3.6] Finishing off

create vm example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
#!/bin/bash
set -e
set -x
ID=$1

BACKFILE_IMG="./ubuntu-20.04-server-cloudimg-amd64.img"
TARGET_IMG="client-${ID}.qcow2"
TARGET_IMG_SIZE="20G"


ADDFILES="http_test"

HOSTNAME="client-${ID}"
IF1_NAME="enp1s0"
IF1_IPADDRESS="11.1.11.$(( 20 + ${ID} ))/24"
IF1_GATEWAY="11.1.11.1"

IF2_NAME="enp2s0"
IF2_IPADDRESS="192.168.122.$(( 30 + ${ID} ))/24"



## create a image from backfile
qemu-img create -f qcow2 -b $BACKFILE_IMG $TARGET_IMG
## resize target image size
qemu-img resize $TARGET_IMG $TARGET_IMG_SIZE

## add user to image
#virt-customize -a ${TARGET_IMG} --run-command 'useradd lab -s /bin/bash -m -p "" -G sudo ; chage -d 99999 lab'
## set user password
#virt-customize -a ${TARGET_IMG} --password test:password:lab123
## set root password
virt-customize -a ${TARGET_IMG} --root-password password:lab123


## upload http_test and script
http_test_script=$(mktemp /tmp/httpXXXXXXXX.sh)
cat > ${http_test_script} << EOF
#!/bin/bash
set -x
while true
do
/http_test -c 11.1.10.10 -s 90000000
sleep \$(( \$RANDOM % 1 ))
done
EOF
virt-customize -a ${TARGET_IMG} --upload http_test:/http_test
virt-customize -a ${TARGET_IMG} --upload ${http_test_script}:/http.sh
virt-customize -a ${TARGET_IMG} --run-command 'chmod +x /http.sh'

## run script on image on firstboot
script=$(mktemp /tmp/scriptXXXXXXXXXX.sh)
cat > ${script} << EOF
#!/bin/bash
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N ""
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N ""
chmod 0600 /etc/ssh/ssh_host*
echo PermitRootLogin yes >> /etc/ssh/sshd_config
sed -i "s/PasswordAuthentication no/PasswordAuthentication yes/g" /etc/ssh/sshd_config
hostnamectl set-hostname ${HOSTNAME}
EOF

virt-customize -a ${TARGET_IMG} --firstboot ${script}
rm -rf ${script}

## create network config file
network_cfg_tmp=$(mktemp /tmp/networkXXXXXXXX.yaml)
cat > ${network_cfg_tmp} << EOF
network:
ethernets:
${IF1_NAME}:
dhcp4: false
addresses:
- ${IF1_IPADDRESS}
gateway4: ${IF1_GATEWAY}
${IF2_NAME}:
addresses: [ ${IF2_IPADDRESS} ]
version: 2
EOF

## add network config
virt-customize -a ${TARGET_IMG} --run-command 'rm -rf /etc/netplan/*'
virt-customize -a ${TARGET_IMG} --upload ${network_cfg_tmp}:/etc/netplan/00-installer-config.yaml
rm -rf ${network_cfg_tmp}

### add start script
start_script=$(mktemp /tmp/startXXXXXXXX)

cat > ${start_script} << EOF

[Unit]
Description=start http test
After=network.target

[Service]
ExecStart=bash -c /http.sh
[Install]
WantedBy=multi-user.target
EOF

virt-customize -a ${TARGET_IMG} --upload ${start_script}:/lib/systemd/system/http_test.service
virt-customize -a ${TARGET_IMG} --run-command 'systemctl daemon-reload; systemctl enable http_test'
rm -rf ${start_script}


## boot vm
virt-install \
--name ${HOSTNAME} \
--ram=8192 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu20.04 \
--graphics none \
--disk /home/lab/cloud/${TARGET_IMG},device=disk,bus=virtio \
--network bridge=br11,model=virtio \
--network bridge=virbr0,model=virtio \
--noautoconsole \
--import

resize the disk size

1
2
3
growpart /dev/vda 1
resize2fs /dev/vda1

bluefield bfb install

ref: https://docs.nvidia.com/doca/sdk/installation-guide/index.html

1. install rshim and pv

install rshim:

1
dpkg -i rshim_2.0.6-3.ge329c69_amd64.deb

output:

1
2
3
4
5
6
7
Selecting previously unselected package rshim.
(Reading database ... 71752 files and directories currently installed.)
Preparing to unpack rshim_2.0.6-3.ge329c69_amd64.deb ...
Unpacking rshim (2.0.6-3.ge329c69) ...
Setting up rshim (2.0.6-3.ge329c69) ...
Created symlink /etc/systemd/system/multi-user.target.wants/rshim.service → /lib/systemd/system/rshim.service.
Processing triggers for man-db (2.9.1-1) ...

install pv:

1
apt install -y pv

2 start and check rshim service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
systemctl is-enabled rshim
enabled

systemctl start rshim

systemctl status rshim
● rshim.service - rshim driver for BlueField SoC
Loaded: loaded (/lib/systemd/system/rshim.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-12-17 07:01:48 UTC; 14min ago
Docs: man:rshim(8)
Process: 10663 ExecStart=/usr/sbin/rshim $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 10666 (rshim)
Tasks: 6 (limit: 629145)
Memory: 2.2M
CGroup: /system.slice/rshim.service
└─10666 /usr/sbin/rshim

Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Starting rshim driver for BlueField SoC...
Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Started rshim driver for BlueField SoC.
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2
Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached
Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open
Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close

2.1 check rshim

1
2
3
4
5
6
7
# ls -lh /dev/rshim0/
total 0
crw------- 1 root root 235, 0 Dec 17 07:01 boot
crw------- 1 root root 234, 0 Dec 17 07:01 console
crw------- 1 root root 511, 0 Dec 17 07:01 misc
crw------- 1 root root 236, 0 Dec 17 07:01 rshim

1
2
3
4
5
6
7
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2
Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2
Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached
Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open
Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close

pcie-0000:3b:00.2 --> rshim0

3 install bfb to bluefield NIC

3.1 create login password

The password is lab123

1
echo "ubuntu_PASSWORD='$(openssl passwd -1 lab123)'" | tee bf.cfg

3.2 install bfb to NIC

1
bfb-install --rshim <rshimN> --bfb <image_path.bfb> --config bf.cfg

example:

1
bfb-install --rshim rshim0 --bfb DOCA_v1.2.0_BlueField_OS_Ubuntu_20.04-5.4.0-1022-bluefield-5.5-1.0.3.2-3.8.0.11969-1.signed-aarch64.bfb --config bf.cfg

output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Pushing bfb + cfg
623MiB 0:00:54 [11.4MiB/s] [ <=> ]
Collecting BlueField booting status. Press Ctrl+C to stop…
INFO[BL2]: start
INFO[BL2]: DDR POST passed
INFO[BL2]: UEFI loaded
INFO[BL31]: start
INFO[BL31]: runtime
INFO[UEFI]: UPVS valid
INFO[UEFI]: eMMC init
INFO[UEFI]: eMMC probed
INFO[UEFI]: PMI: updates started
INFO[UEFI]: PMI: boot image update
INFO[UEFI]: PMI: updates completed, status 0
INFO[UEFI]: PCIe enum start
INFO[UEFI]: PCIe enum end
INFO[MISC]: Found bf.cfg
INFO[MISC]: Ubuntu installation started
INFO[MISC]: Installing OS image
INFO[MISC]: Changing the default password for user ubuntu
INFO[MISC]: Installation finished

4 connect bluefield nic by console port (open another session)

1
screen /dev/rshim0/console

console output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
write counter to semaphore: Operation not permitted
INFO: Changing the default password for user ubuntu
Boot0000* focal
INFO: Installation finished
INFO: Rebooting...




Mellanox BlueField-2 A1 BL1 V1.1
NOTICE: BL2R: v2.2(release):3.8.0-25-ge146e67
NOTICE: BL2R: Built : 15:12:50, Nov 30 2021
NOTICE: BL2R built for hw (ver 1)
NOTICE: BL2R: Booting BL2
NOTICE: BL2: v2.2(release):3.8.0-25-ge146e67
NOTICE: BL2: Built : 15:12:50, Nov 30 2021
NOTICE: BL2 built for hw (ver 1)
NOTICE: Running as MBF2H332A-AECO system
NOTICE: No SPD detected on MSS0 DIMM0
NOTICE: No SPD detected on MSS0 DIMM1
NOTICE: Finished initializing DDR
NOTICE: DDR POST passed.
NOTICE: BL31: v2.2(release):3.8.0-25-ge146e67
NOTICE: BL31: Built : 15:12:50, Nov 30 2021
NOTICE: BL31 built for hw (ver 1)
UEFI firmware (version BlueField:3.8.0-34-gb1e3ae0 built at 15:15:10 on Nov 30 2021)
Press <ESC> twice to enter UEFI menu
3 seconds remaining
2 seconds remaining
1 seconds remaining
0
[ 0.608703] rtc-efi rtc-efi: hctosys: unable to read the hardware clock
[ 10.715782] mlx5_core 0000:03:00.0: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): hash based LAG is not supported by current device
[ 12.861453] mlx5_core 0000:03:00.1: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): hash based LAG is not supported by current device

Ubuntu 20.04.3 LTS localhost hvc0

localhost login:

The login username is ubuntu. The password is lab123

5 connect bluefield use by ssh

5.1 config tmfifo_net0

1
ip add add 192.168.100.1/30 dev tmfifo_net0

check interface

1
2
3
4
5
6
7
8
9
10
11
root@ubuntu2004-lab1:~# ip address show tmfifo_net0
16: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.1/30 scope global tmfifo_net0
valid_lft forever preferred_lft forever
inet6 fe80::21a:caff:feff:ff02/64 scope link
valid_lft forever preferred_lft forever

root@ubuntu2004-lab1:~# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=2.46 ms

connect bluefield by ssh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@ubuntu2004-lab1:~# ssh -l ubuntu 192.168.100.2
The authenticity of host '192.168.100.2 (192.168.100.2)' can't be established.
ECDSA key fingerprint is SHA256:QqO4i9YSGBI6pp1N8JJKqnTFokEEn1JVh+sdxcYPIb4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.100.2' (ECDSA) to the list of known hosts.
ubuntu@192.168.100.2's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-1022-bluefield aarch64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

System information as of Tue Sep 7 18:44:49 UTC 2021

System load: 0.11 Processes: 231
Usage of /: 5.1% of 57.10GB Users logged in: 1
Memory usage: 5% IPv4 address for tmfifo_net0: 192.168.100.2
Swap usage: 0%


0 updates can be applied immediately.

Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings