<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Xin Yu&#39;s Notes</title>
  
  
  <link href="https://nfshanq.github.io/atom.xml" rel="self"/>
  
  <link href="https://nfshanq.github.io/"/>
  <updated>2022-02-10T01:17:38.052Z</updated>
  <id>https://nfshanq.github.io/</id>
  
  <author>
    <name>Xin Yu</name>
    
  </author>
  
  <generator uri="https://hexo.io/">Hexo</generator>
  
  <entry>
    <title>openwrt compile</title>
    <link href="https://nfshanq.github.io/2022/02/08/Linux/openwrt/openwrt_compile/"/>
    <id>https://nfshanq.github.io/2022/02/08/Linux/openwrt/openwrt_compile/</id>
    <published>2022-02-08T23:55:10.000Z</published>
    <updated>2022-02-10T01:17:38.052Z</updated>
    
    <content type="html"><![CDATA[<p>For GL.inet box</p><h1 id="Requirements"><a href="#Requirements" class="headerlink" title="Requirements"></a>Requirements</h1><p>You need the following tools to compile OpenWrt, the package names vary between distributions. A complete list with distribution specific packages is found in the <a href="https://openwrt.org/docs/guide-developer/build-system/install-buildsystem">Build System Setup</a> documentation</p><h2 id="OS"><a href="#OS" class="headerlink" title="OS"></a>OS</h2><ul><li>Ubuntu &gt;= 18.04 (16.04 python3 is 3.5, need &gt; python 3.6)</li><li>Debian &gt;= 10</li></ul><h2 id="packages-install"><a href="#packages-install" class="headerlink" title="packages install"></a>packages install</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">sudo apt update -y</span><br><span class="line">sudo apt install build-essential ccache ecj fastjar file g++ gawk \</span><br><span class="line">gettext git java-propose-classpath libelf-dev libncurses5-dev \</span><br><span class="line">libncursesw5-dev libssl-dev python python2.7-dev python3 unzip wget \</span><br><span class="line">python3-distutils python3-setuptools python3-dev rsync subversion \</span><br><span class="line">swig time xsltproc zlib1g-dev -y</span><br></pre></td></tr></table></figure><h1 id="Quickstart"><a href="#Quickstart" class="headerlink" title="Quickstart"></a>Quickstart</h1><h2 id="Clone-repository"><a href="#Clone-repository" class="headerlink" title="Clone repository"></a>Clone repository</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">git <span class="built_in">clone</span> https://github.com/gl-inet/gl-infra-builder.git</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cd</span> gl-infra-builder</span><br></pre></td></tr></table></figure><h2 id="Download-OpenWRT-21-02"><a href="#Download-OpenWRT-21-02" class="headerlink" title="Download OpenWRT 21.02"></a>Download OpenWRT 21.02</h2><p>Setup, the command will auto download openwrt-21.02 by default and auto config, and then patch all the the GL product Patches</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python3 setup.py -c config-21.02.0.yml</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">$ more config-21.02.0.yml</span><br><span class="line">repo: https://github.com/openwrt/openwrt.git</span><br><span class="line">branch: v21.02.0</span><br><span class="line">git_clone_dir: openwrt-21.02/openwrt-21.02.0</span><br><span class="line">openwrt_root_dir: openwrt-21.02/openwrt-21.02.0</span><br><span class="line">revision: b2ae4233149dfd78f2ac00bb5327695bcacdc255</span><br><span class="line"></span><br><span class="line">patch_folders:</span><br><span class="line">  - patches-21.02.0/</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">~/gl-infra-builder/patches-21.02.0$ ls -l</span><br><span class="line">total 172</span><br><span class="line">-rw-r--r-- 1 lab lab 15930 Jan 29 16:47 0001-ath79-add-support-for-gl-xe300.patch</span><br><span class="line">-rw-r--r-- 1 lab lab 49979 Jan 29 16:47 0002-ath79-add-support-for-gl-mifi.patch</span><br><span class="line">-rw-r--r-- 1 lab lab 14045 Jan 29 16:47 0003-add-modem-support.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  3514 Jan 29 16:47 0004-add-RS485-support.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  3481 Jan 29 16:47 0005-add-spinand-support.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  2223 Jan 29 16:47 0006-led-netdev-trigger-by-wwanx.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  7408 Jan 29 16:47 0007-ath79-add-support-for-gl-ar300m.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  8408 Jan 29 16:47 0008-ath79-add-support-for-gl-x300b.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  6330 Jan 29 16:47 0009-ath79-add-support-for-gl-ar150.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  8705 Jan 29 16:47 0010-ath79-add-support-for-gl-usb150.patch</span><br><span class="line">-rw-r--r-- 1 lab lab  7017 Jan 29 16:47 0011-add-support-gl-ar750-ar750s.patch</span><br><span class="line">-rw-r--r-- 1 lab lab 13400 Jan 29 16:47 0012-ath79-add-support-for-gl-x750.patch</span><br><span class="line">-rw-r--r-- 1 lab lab 11790 Jan 29 16:47 0013-support-fast-forward.patch</span><br></pre></td></tr></table></figure><h2 id="Update-feeds-and-install-frr-vxlan-feeds"><a href="#Update-feeds-and-install-frr-vxlan-feeds" class="headerlink" title="Update feeds and install frr vxlan feeds"></a>Update feeds and install frr vxlan feeds</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">./scripts/feeds update -a</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">./scripts/feeds install frr vxlan libpam liblzma libnetsnmp</span><br></pre></td></tr></table></figure><h2 id="Make-config-file"><a href="#Make-config-file" class="headerlink" title="Make config file"></a>Make config file</h2><p>create the default config file</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">make defconfig</span><br></pre></td></tr></table></figure><p>make config file. select which kernel model and package you need to add to OpenWRT</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">make menuconfig</span><br></pre></td></tr></table></figure><h3 id="Select-target-system"><a href="#Select-target-system" class="headerlink" title="Select target system"></a>Select target system</h3><p><img src="/2022/02/08/Linux/openwrt/openwrt_compile/1644454983595.png" alt="1644454983595.png"></p><h3 id="Kernel-model"><a href="#Kernel-model" class="headerlink" title="Kernel model"></a>Kernel model</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Kernel modules ---&gt; Network Support ---&gt;</span><br><span class="line"><span class="built_in">enable</span> kmod-vlxan</span><br><span class="line"><span class="built_in">enable</span> kmod-veth</span><br><span class="line"><span class="built_in">enable</span> kmod-tun</span><br></pre></td></tr></table></figure><p><img src="/2022/02/08/Linux/openwrt/openwrt_compile/1644455181686.png" alt="1644455181686.png"></p><h3 id="Add-frr"><a href="#Add-frr" class="headerlink" title="Add frr"></a>Add frr</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Network ---&gt; Routing and Redirection ---&gt; frr</span><br></pre></td></tr></table></figure><p><img src="/2022/02/08/Linux/openwrt/openwrt_compile/1644455358487.png" alt="1644455358487.png"></p><h3 id="add-some-tools"><a href="#add-some-tools" class="headerlink" title="add some tools"></a>add some tools</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Network ---&gt; Routing and Redirection</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">enable</span> ip-bridge <span class="comment">## MUST vxlan forwording plane will use this tool</span></span><br><span class="line"><span class="built_in">enable</span> ip-full  <span class="comment">## MUST create vxlan interface will use this tool</span></span><br><span class="line"><span class="built_in">enable</span> ss</span><br><span class="line"><span class="built_in">enable</span> tc-full</span><br></pre></td></tr></table></figure><h3 id="add-vxlan"><a href="#add-vxlan" class="headerlink" title="add vxlan"></a>add vxlan</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Network ---&gt;</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">enable</span> vxlan <span class="comment">## MUST vxlan forwording plane will use this script</span></span><br></pre></td></tr></table></figure><p><img src="/2022/02/08/Linux/openwrt/openwrt_compile/1644455697606.png" alt="1644455697606.png"></p><h2 id="Make-the-image"><a href="#Make-the-image" class="headerlink" title="Make the image"></a>Make the image</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">make -j$(nproc) V=s</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;p&gt;For GL.inet box&lt;/p&gt;
&lt;h1 id=&quot;Requirements&quot;&gt;&lt;a href=&quot;#Requirements&quot; class=&quot;headerlink&quot; title=&quot;Requirements&quot;&gt;&lt;/a&gt;Requirements&lt;/h1&gt;&lt;p&gt;You nee</summary>
      
    
    
    
    
  </entry>
  
  <entry>
    <title>OpenWRT VxLAN</title>
    <link href="https://nfshanq.github.io/2022/02/07/Linux/openwrt/vxlan/"/>
    <id>https://nfshanq.github.io/2022/02/07/Linux/openwrt/vxlan/</id>
    <published>2022-02-07T21:07:50.000Z</published>
    <updated>2022-02-09T02:30:51.697Z</updated>
    
    <content type="html"><![CDATA[<h1 id="1-Topology"><a href="#1-Topology" class="headerlink" title="1. Topology"></a>1. Topology</h1><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644369044061.png" alt="1644369044061.png"></p><p>GL.inet MT1300 is MT7621 chip</p><p><img src="/2022/02/07/Linux/openwrt/vxlan//1644372818557.png" alt="1644372818557.png"></p><h1 id="2-device-config"><a href="#2-device-config" class="headerlink" title="2. device config"></a>2. device config</h1><h2 id="2-1-QFX5100-config"><a href="#2-1-QFX5100-config" class="headerlink" title="2.1 QFX5100 config"></a>2.1 QFX5100 config</h2><h3 id="2-1-1-interface-config"><a href="#2-1-1-interface-config" class="headerlink" title="2.1.1 interface config"></a>2.1.1 interface config</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># L2 interface </span></span><br><span class="line"><span class="built_in">set</span> xe-0/0/0 unit 0 family ethernet-switching interface-mode access</span><br><span class="line"><span class="built_in">set</span> xe-0/0/0 unit 0 family ethernet-switching vlan members vlan-vxlan10</span><br><span class="line"></span><br><span class="line"><span class="comment"># connect L2 switch</span></span><br><span class="line"><span class="built_in">set</span> xe-0/0/37 unit 0 family ethernet-switching interface-mode trunk</span><br><span class="line"><span class="built_in">set</span> xe-0/0/37 unit 0 family ethernet-switching vlan members vlan81</span><br><span class="line"></span><br><span class="line"><span class="comment"># vtep source interface</span></span><br><span class="line"><span class="built_in">set</span> irb unit 81 family inet address 11.81.1.1/24</span><br><span class="line"><span class="built_in">set</span> lo0 unit 0 family inet address 11.1.1.1/32</span><br></pre></td></tr></table></figure><p>vlan-vxlan10 is vxlan vlan</p><h3 id="2-1-2-protocol-config"><a href="#2-1-2-protocol-config" class="headerlink" title="2.1.2 protocol config"></a>2.1.2 protocol config</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">set</span> routing-options router-id 11.1.1.1</span><br><span class="line"><span class="built_in">set</span> routing-options autonomous-system 65000</span><br><span class="line"><span class="built_in">set</span> protocols bgp group evpn <span class="built_in">type</span> internal</span><br><span class="line"><span class="built_in">set</span> protocols bgp group evpn local-address 11.1.1.1</span><br><span class="line"><span class="built_in">set</span> protocols bgp group evpn family evpn signaling</span><br><span class="line"><span class="built_in">set</span> protocols bgp group evpn allow 11.81.1.0/24</span><br><span class="line"><span class="built_in">set</span> protocols bgp group evpn allow 11.82.1.0/24</span><br><span class="line"><span class="built_in">set</span> protocols evpn encapsulation vxlan</span><br><span class="line"><span class="built_in">set</span> protocols evpn vni-options vni 10 vrf-target target:65000:10</span><br><span class="line"><span class="built_in">set</span> protocols evpn extended-vni-list 10</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="2-1-3-switch-and-vlan-config"><a href="#2-1-3-switch-and-vlan-config" class="headerlink" title="2.1.3 switch and vlan config"></a>2.1.3 switch and vlan config</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">set</span> switch-options vtep-source-interface lo0.0</span><br><span class="line"><span class="built_in">set</span> switch-options route-distinguisher 11.1.1.1:10</span><br><span class="line"><span class="built_in">set</span> switch-options vrf-target target:65000:10</span><br><span class="line"><span class="built_in">set</span> vlans vlan-vxlan10 vlan-id 10</span><br><span class="line"><span class="built_in">set</span> vlans vlan-vxlan10 vxlan vni 10</span><br><span class="line"><span class="built_in">set</span> vlans vlan-vxlan10 vxlan ingress-node-replication</span><br><span class="line"><span class="built_in">set</span> vlans vlan81 description <span class="string">&quot;openwrt test&quot;</span></span><br><span class="line"><span class="built_in">set</span> vlans vlan81 vlan-id 81</span><br><span class="line"><span class="built_in">set</span> vlans vlan81 l3-interface irb.81</span><br></pre></td></tr></table></figure><h2 id="2-2-OpenWRT-config"><a href="#2-2-OpenWRT-config" class="headerlink" title="2.2 OpenWRT config"></a>2.2 OpenWRT config</h2><h3 id="2-2-1-FRR-config"><a href="#2-2-1-FRR-config" class="headerlink" title="2.2.1 FRR config"></a>2.2.1 FRR config</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">router bgp 65000</span><br><span class="line"> bgp router-id 11.81.1.2</span><br><span class="line"> neighbor evpn peer-group</span><br><span class="line"> neighbor evpn remote-as internal</span><br><span class="line"> neighbor evpn update-source wan</span><br><span class="line"> neighbor 11.1.1.1 peer-group evpn</span><br><span class="line"> !</span><br><span class="line"> address-family l2vpn evpn</span><br><span class="line">  neighbor evpn activate</span><br><span class="line">  advertise-all-vni</span><br><span class="line">  advertise ipv4 unicast</span><br><span class="line"> exit-address-family</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="2-2-2-vxlan-interface-config"><a href="#2-2-2-vxlan-interface-config" class="headerlink" title="2.2.2 vxlan interface config"></a>2.2.2 vxlan interface config</h3><p>The OpenWRT config (/etc/config/network) must need  vxlan peeraddr.</p><p>ref: <a href="https://openwrt.org/docs/guide-user/network/tunneling_interface_protocols">https://openwrt.org/docs/guide-user/network/tunneling_interface_protocols</a></p><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644369624624.png" alt="1644369624624.png"></p><p>so, use hotplug to add the vxlan interface.</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">root@OpenWrt:~<span class="comment"># cat /etc/hotplug.d/iface/50-vxlan</span></span><br><span class="line"><span class="comment">#!/bin/sh</span></span><br><span class="line"></span><br><span class="line">[ <span class="string">&quot;<span class="variable">$ACTION</span>&quot;</span> == <span class="string">&quot;ifup&quot;</span> ] &amp;&amp; [ <span class="string">&quot;<span class="variable">$INTERFACE</span>&quot;</span> == <span class="string">&quot;wan&quot;</span> ] &amp;&amp; &#123;</span><br><span class="line"></span><br><span class="line">WAN_IFNAME=<span class="string">&#x27;wan@eth0&#x27;</span></span><br><span class="line">WAN_IP=<span class="string">&quot;<span class="subst">$(ip -br a | grep <span class="string">&quot;<span class="variable">$WAN_IFNAME</span>&quot;</span> | grep -Eo &#x27;[0-9]&#123;1,3&#125;\.[0-9]&#123;1,3&#125;\.[0-9]&#123;1,3&#125;\.[0-9]&#123;1,3&#125;&#x27;)</span>&quot;</span></span><br><span class="line"></span><br><span class="line">ip link del vxlan1</span><br><span class="line">ip link add vxlan1 <span class="built_in">type</span> vxlan id 10 <span class="built_in">local</span> <span class="variable">$&#123;WAN_IP&#125;</span> dstport 4789</span><br><span class="line">ip link <span class="built_in">set</span> vxlan1 master br-lan</span><br><span class="line">ip link <span class="built_in">set</span> vxlan1 up</span><br><span class="line"></span><br><span class="line">&#125;</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="2-2-3-firewall-config"><a href="#2-2-3-firewall-config" class="headerlink" title="2.2.3 firewall config"></a>2.2.3 firewall config</h3><p>The OpenWRT default firewall is disable the VxLAN traffic.  so we need add the firewall rule. JUNOS will use 4789 to connect the VxLAN. </p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># … in the /etc/config/firewall</span></span><br><span class="line">config rule</span><br><span class="line">        option name             Allow-VxLAN</span><br><span class="line">        option src              wan</span><br><span class="line">        option dest_port        4789</span><br><span class="line">        option proto            udp</span><br><span class="line">        option target           ACCEPT</span><br></pre></td></tr></table></figure><p>if we need another BGP speaker to connect active.  we need to add the BGP 179(TCP) port in the firewall rule. In my use case, the JUNOS BGP speaker is passive mode.</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># … in the /etc/config/firewall</span></span><br><span class="line">config rule</span><br><span class="line">        option name             Allow-BGP</span><br><span class="line">        option src              wan</span><br><span class="line">        option dest_port        179</span><br><span class="line">        option proto            tcp</span><br><span class="line">        option target           ACCEPT</span><br><span class="line"></span><br></pre></td></tr></table></figure><p><img src="./1644370121708.png" alt="1644370121708.png"></p><h1 id="3-vxlan-evpn-status"><a href="#3-vxlan-evpn-status" class="headerlink" title="3. vxlan evpn status"></a>3. vxlan evpn status</h1><h2 id="3-1-QFX5100"><a href="#3-1-QFX5100" class="headerlink" title="3.1 QFX5100"></a>3.1 QFX5100</h2><h3 id="3-1-1-BGP-control-plane"><a href="#3-1-1-BGP-control-plane" class="headerlink" title="3.1.1 BGP (control plane)"></a>3.1.1 BGP (control plane)</h3><p>BGP peer is evpn.</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">lab@qfx5100-48s-6q&gt; show bgp summary</span><br><span class="line">Threading mode: BGP I/O</span><br><span class="line">Groups: 1 Peers: 1 Down peers: 0</span><br><span class="line">Unconfigured peers: 1</span><br><span class="line">Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending</span><br><span class="line">bgp.evpn.0</span><br><span class="line">                       2          2          0          0          0          0</span><br><span class="line">Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|<span class="comment">#Active/Received/Accepted/Damped...</span></span><br><span class="line">11.81.1.2             65000         80         70       0       0       28:40 Establ</span><br><span class="line">  __default_evpn__.evpn.0: 0/0/0/0</span><br><span class="line">  bgp.evpn.0: 2/2/2/0</span><br><span class="line">  default-switch.evpn.0: 2/2/2/0</span><br></pre></td></tr></table></figure><p>BGP evpn NLRI</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line">lab@qfx5100-48s-6q&gt; show route table bgp.evpn.0</span><br><span class="line"></span><br><span class="line">bgp.evpn.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)</span><br><span class="line">+ = Active Route, - = Last Active, * = Both</span><br><span class="line"></span><br><span class="line">2:11.1.1.1:10::10::e4:43:4b:f6:c5:56/304 MAC/IP</span><br><span class="line">                   *[EVPN/170] 00:39:07</span><br><span class="line">                       Indirect</span><br><span class="line">2:11.81.1.2:2::0::e4:43:4b:f6:c5:75/304 MAC/IP</span><br><span class="line">                   *[BGP/170] 00:45:24, localpref 100</span><br><span class="line">                      AS path: I, validation-state: unverified</span><br><span class="line">                    &gt;  to 11.81.1.2 via irb.81</span><br><span class="line">2:11.1.1.1:10::10::e4:43:4b:f6:c5:56::192.168.8.100/304 MAC/IP</span><br><span class="line">                   *[EVPN/170] 00:19:13</span><br><span class="line">                       Indirect</span><br><span class="line">2:11.81.1.2:2::0::e4:43:4b:f6:c5:75::192.168.8.10/304 MAC/IP</span><br><span class="line">                   *[BGP/170] 00:00:19, localpref 100</span><br><span class="line">                      AS path: I, validation-state: unverified</span><br><span class="line">                    &gt;  to 11.81.1.2 via irb.81</span><br><span class="line">3:11.1.1.1:10::10::11.1.1.1/248 IM</span><br><span class="line">                   *[EVPN/170] 00:46:26</span><br><span class="line">                       Indirect</span><br><span class="line">3:11.81.1.2:2::0::11.81.1.2/248 IM</span><br><span class="line">                   *[BGP/170] 00:45:24, localpref 100</span><br><span class="line">                      AS path: I, validation-state: unverified</span><br><span class="line">                    &gt;  to 11.81.1.2 via irb.81</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>One evpn BGP NLRI extensive information.</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">lab@qfx5100-48s-6q&gt; show route table bgp.evpn.0 extensive evpn-mac-address e4:43:4b:f6:c5:75</span><br><span class="line"></span><br><span class="line">bgp.evpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)</span><br><span class="line">2:11.81.1.2:2::0::e4:43:4b:f6:c5:75/304 MAC/IP (1 entry, 0 announced)</span><br><span class="line">        *BGP    Preference: 170/-101</span><br><span class="line">                Route Distinguisher: 11.81.1.2:2</span><br><span class="line">                Next hop <span class="built_in">type</span>: Indirect, Next hop index: 0</span><br><span class="line">                Address: 0xccd60b0</span><br><span class="line">                Next-hop reference count: 4</span><br><span class="line">                Source: 11.81.1.2</span><br><span class="line">                Protocol next hop: 11.81.1.2</span><br><span class="line">                Indirect next hop: 0x2 no-forward INH Session ID: 0x0</span><br><span class="line">                State: &lt;Active Int Ext&gt;</span><br><span class="line">                Local AS: 65000 Peer AS: 65000</span><br><span class="line">                Age: 17:50:15 Metric2: 0</span><br><span class="line">                Validation State: unverified</span><br><span class="line">                Task: BGP_65000.11.81.1.2+43211</span><br><span class="line">                AS path: I</span><br><span class="line">                Communities: target:65000:10 encapsulation:vxlan(0x8)</span><br><span class="line">                Import Accepted</span><br><span class="line">                Route Label: 10</span><br><span class="line">                ESI: 00:00:00:00:00:00:00:00:00:00</span><br><span class="line">                Localpref: 100</span><br><span class="line">                Router ID: 11.81.1.2</span><br><span class="line">                Secondary Tables: default-switch.evpn.0</span><br><span class="line">                Indirect next hops: 1</span><br><span class="line">                        Protocol next hop: 11.81.1.2</span><br><span class="line">                        Indirect next hop: 0x2 no-forward INH Session ID: 0x0</span><br><span class="line">                        Indirect path forwarding next hops: 1</span><br><span class="line">                                Next hop <span class="built_in">type</span>: Router</span><br><span class="line">                                Next hop: 11.81.1.2 via irb.81</span><br><span class="line">                                Session Id: 0x0</span><br><span class="line">11.81.1.0/24 Originating RIB: inet.0</span><br><span class="line">  Node path count: 1</span><br><span class="line">  Forwarding nexthops: 1</span><br><span class="line">Next hop <span class="built_in">type</span>: Interface</span><br><span class="line">Nexthop: via irb.81</span><br></pre></td></tr></table></figure><h3 id="3-1-2-forwarding-plane"><a href="#3-1-2-forwarding-plane" class="headerlink" title="3.1.2 forwarding plane"></a>3.1.2 forwarding plane</h3><p>mac-ip table</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">&#123;master:0&#125;</span><br><span class="line">lab@qfx5100-48s-6q&gt; show ethernet-switching mac-ip-table</span><br><span class="line"></span><br><span class="line">MAC IP flags  (S - Static, D - Dynamic, L - Local , R - Remote, Lp - Local Proxy,</span><br><span class="line">               Rp - Remote Proxy, K - Kernel, RT - Dest Route, AD - Advt to remote,</span><br><span class="line">               RE - Re-ARP/ND, RO - Router, OV - Override)</span><br><span class="line"> Routing instance : default-switch</span><br><span class="line"> Bridging domain : vlan-vxlan10</span><br><span class="line">   IP                           MAC                  Flags             Logical            Active</span><br><span class="line">   address                      address                                Interface          <span class="built_in">source</span></span><br><span class="line">   192.168.8.100                e4:43:4b:f6:c5:56    DL,K,AD           xe-0/0/0.0</span><br><span class="line">   192.168.8.10                 e4:43:4b:f6:c5:75    DR,K              vtep.32769         11.81.1.2</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>MAC table:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">&#123;master:0&#125;</span><br><span class="line">lab@qfx5100-48s-6q&gt; show ethernet-switching table</span><br><span class="line"></span><br><span class="line">MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static</span><br><span class="line">           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Ethernet switching table : 3 entries, 3 learned</span><br><span class="line">Routing instance : default-switch</span><br><span class="line">   Vlan                MAC                 MAC      Logical                Active</span><br><span class="line">   name                address             flags    interface              <span class="built_in">source</span></span><br><span class="line">   vlan-vxlan10        e4:43:4b:f6:c5:56   D        xe-0/0/0.0</span><br><span class="line">   vlan-vxlan10        e4:43:4b:f6:c5:75   D        vtep.32769             11.81.1.2</span><br><span class="line">   vlan81              94:83:c4:16:51:44   D        xe-0/0/37.0</span><br><span class="line"></span><br><span class="line">&#123;master:0&#125;</span><br></pre></td></tr></table></figure><p>forwarding-table detail:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">&#123;master:0&#125;</span><br><span class="line">lab@qfx5100-48s-6q&gt; show route forwarding-table family ethernet-switching matching e4:43:4b:f6:c5:75/48 extensive</span><br><span class="line">Routing table: default-switch.evpn-vxlan [Index 7]</span><br><span class="line">Bridging domain: vlan-vxlan10.evpn-vxlan [Index 3]</span><br><span class="line">VPLS:</span><br><span class="line">Enabled protocols: Bridging, ACKed by all peers, EVPN VXLAN,</span><br><span class="line"></span><br><span class="line">Destination:  e4:43:4b:f6:c5:75/48</span><br><span class="line">  Learn VLAN: 0                        Route <span class="built_in">type</span>: user</span><br><span class="line">  Route reference: 0                   Route interface-index: 566</span><br><span class="line">  Multicast RPF nh index: 0</span><br><span class="line">  P2mpidx: 0</span><br><span class="line">  IFL generation: 131                  Epoch: 0</span><br><span class="line">  Sequence Number: 0                   Learn Mask: 0x4000000000000000000000000000000000000000</span><br><span class="line">  L2 Flags: control_dyn</span><br><span class="line">  Flags: sent to PFE</span><br><span class="line">  Nexthop:</span><br><span class="line">  Next-hop <span class="built_in">type</span>: composite             Index: 1742     Reference: 5</span><br><span class="line">  Next-hop <span class="built_in">type</span>: indirect              Index: 524286   Reference: 3</span><br><span class="line">  Nexthop: 11.81.1.2</span><br><span class="line">  Next-hop <span class="built_in">type</span>: unicast               Index: 1741     Reference: 4</span><br><span class="line">  Next-hop interface: xe-0/0/37.0</span><br></pre></td></tr></table></figure><h2 id="3-2-OpenWRT"><a href="#3-2-OpenWRT" class="headerlink" title="3.2 OpenWRT"></a>3.2 OpenWRT</h2><h3 id="3-2-1-BGP"><a href="#3-2-1-BGP" class="headerlink" title="3.2.1 BGP"></a>3.2.1 BGP</h3><p>BGP peer</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># frr command</span></span><br><span class="line">OpenWrt<span class="comment"># show bgp l2vpn evpn summ</span></span><br><span class="line">BGP router identifier 11.81.1.2, <span class="built_in">local</span> AS number 65000 vrf-id 0</span><br><span class="line">BGP table version 0</span><br><span class="line">RIB entries 3, using 384 bytes of memory</span><br><span class="line">Peers 1, using 10896 bytes of memory</span><br><span class="line">Peer groups 1, using 32 bytes of memory</span><br><span class="line"></span><br><span class="line">Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt</span><br><span class="line">11.1.1.1        4      65000      3090      3048        0    0    0 17:27:52            3        3</span><br><span class="line"></span><br><span class="line">Total number of neighbors 1</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>evpn BGP NLRI</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># frr command</span></span><br><span class="line">OpenWrt<span class="comment"># show bgp l2vpn evpn</span></span><br><span class="line">BGP table version is 15, <span class="built_in">local</span> router ID is 11.81.1.2</span><br><span class="line">Status codes: s suppressed, d damped, h <span class="built_in">history</span>, * valid, &gt; best, i - internal</span><br><span class="line">Origin codes: i - IGP, e - EGP, ? - incomplete</span><br><span class="line">EVPN type-1 prefix: [1]:[ESI]:[EthTag]:[IPlen]:[VTEP-IP]</span><br><span class="line">EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]</span><br><span class="line">EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]</span><br><span class="line">EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]</span><br><span class="line">EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]</span><br><span class="line"></span><br><span class="line">   Network          Next Hop            Metric LocPrf Weight Path</span><br><span class="line">Route Distinguisher: 11.1.1.1:10</span><br><span class="line">*&gt;i[2]:[10]:[48]:[e4:43:4b:f6:c5:56]</span><br><span class="line">                    11.1.1.1                      100      0 i</span><br><span class="line">                    RT:65000:10 ET:8</span><br><span class="line">*&gt;i[2]:[10]:[48]:[e4:43:4b:f6:c5:56]:[32]:[192.168.8.100]</span><br><span class="line">                    11.1.1.1                      100      0 i</span><br><span class="line">                    RT:65000:10 ET:8</span><br><span class="line">*&gt;i[3]:[10]:[32]:[11.1.1.1]</span><br><span class="line">                    11.1.1.1                      100      0 i</span><br><span class="line">                    RT:65000:10 ET:8</span><br><span class="line">Route Distinguisher: 11.81.1.2:2</span><br><span class="line">*&gt; [2]:[0]:[48]:[e4:43:4b:f6:c5:75]</span><br><span class="line">                    11.81.1.2                          32768 i</span><br><span class="line">                    ET:8 RT:65000:10</span><br><span class="line">*&gt; [2]:[0]:[48]:[e4:43:4b:f6:c5:75]:[32]:[192.168.8.10]</span><br><span class="line">                    11.81.1.2                          32768 i</span><br><span class="line">                    ET:8 RT:65000:10</span><br><span class="line">*&gt; [3]:[0]:[32]:[11.81.1.2]</span><br><span class="line">                    11.81.1.2                          32768 i</span><br><span class="line">                    ET:8 RT:65000:10</span><br><span class="line"></span><br><span class="line">Displayed 6 out of 6 total prefixes</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure><p>evpn BGP advertised-routes</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># frr command</span></span><br><span class="line">OpenWrt<span class="comment"># show bgp l2vpn evpn neighbors 11.1.1.1 advertised-routes</span></span><br><span class="line">BGP table version is 0, <span class="built_in">local</span> router ID is 11.81.1.2</span><br><span class="line">Default <span class="built_in">local</span> pref 100, <span class="built_in">local</span> AS 65000</span><br><span class="line">Status codes: s suppressed, d damped, h <span class="built_in">history</span>, * valid, &gt; best, i - internal</span><br><span class="line">Origin codes: i - IGP, e - EGP, ? - incomplete</span><br><span class="line"></span><br><span class="line">   Network          Next Hop            Metric LocPrf Weight Path</span><br><span class="line">Route Distinguisher: 11.81.1.2:2</span><br><span class="line">*&gt; [2]:[0]:[48]:[e4:43:4b:f6:c5:75]</span><br><span class="line">                                  100  32768 i</span><br><span class="line">*&gt; [2]:[0]:[48]:[e4:43:4b:f6:c5:75]:[32]:[192.168.8.10]</span><br><span class="line">                                  100  32768 i</span><br><span class="line">*&gt; [3]:[0]:[32]:[11.81.1.2]</span><br><span class="line">                                  100  32768 i</span><br><span class="line"></span><br><span class="line">Total number of prefixes 3</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="3-2-2-Linux-forwarding-plane"><a href="#3-2-2-Linux-forwarding-plane" class="headerlink" title="3.2.2 Linux forwarding plane"></a>3.2.2 Linux forwarding plane</h3><p>bridge fdb</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># linux command</span></span><br><span class="line">root@OpenWrt:~<span class="comment"># bridge fdb | grep -E &#x27;vxlan|wan|lan1&#x27;</span></span><br><span class="line">e4:43:4b:f6:c5:75 dev lan1 master br-lan</span><br><span class="line">94:83:c4:16:51:45 dev lan1 vlan 1 master br-lan permanent</span><br><span class="line">94:83:c4:16:51:45 dev lan1 master br-lan permanent</span><br><span class="line">e4:43:4b:f6:c5:75 dev lan1 vlan 1 self</span><br><span class="line">c0:42:d0:08:9a:80 dev wan vlan 1 self</span><br><span class="line">1c:9c:8c:67:c3:44 dev wan vlan 1 self</span><br><span class="line">1c:9c:8c:68:e4:c0 dev wan vlan 1 self</span><br><span class="line">e4:43:4b:f6:c5:56 dev vxlan1 vlan 1 extern_learn master br-lan</span><br><span class="line">e4:43:4b:f6:c5:56 dev vxlan1 extern_learn master br-lan</span><br><span class="line">3e:dd:f3:a4:65:2b dev vxlan1 vlan 1 master br-lan permanent</span><br><span class="line">3e:dd:f3:a4:65:2b dev vxlan1 master br-lan permanent</span><br><span class="line">00:00:00:00:00:00 dev vxlan1 dst 11.1.1.1 self permanent</span><br><span class="line">e4:43:4b:f6:c5:56 dev vxlan1 dst 11.1.1.1 self extern_learn</span><br><span class="line">33:33:00:00:00:01 dev wlan1 self permanent</span><br><span class="line">33:33:00:00:00:02 dev wlan1 self permanent</span><br></pre></td></tr></table></figure><p>00:00:00:00:00:00 dev vxlan1 dst 11.1.1.1 self permanent. is add by /lib/netifd/proto/vxlan.sh</p><p>vxlan interface:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">root@OpenWrt:~<span class="comment"># ip -d link show vxlan1</span></span><br><span class="line">8: vxlan1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue master br-lan state UNKNOWN mode DEFAULT group default qlen 1000</span><br><span class="line">    link/ether 3e:dd:f3:a4:65:2b brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535</span><br><span class="line">    vxlan id 10 <span class="built_in">local</span> 11.81.1.2 srcport 0 0 dstport 4789 ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx</span><br><span class="line">    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8003 port_no 0x3 designated_port 32771 designated_cost 0 designated_bridge 7fff.94:83:C4:16:51:45 designated_root 7fff.94:83:C4:16:51:45 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535</span><br></pre></td></tr></table></figure><p>linux bridge:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">root@OpenWrt:~<span class="comment"># brctl show</span></span><br><span class="line">bridge namebridge idSTP enabledinterfaces</span><br><span class="line">br-lan7fff.9483c4165145nolan2</span><br><span class="line">vxlan1</span><br><span class="line">lan1</span><br></pre></td></tr></table></figure><h1 id="4-Throughput"><a href="#4-Throughput" class="headerlink" title="4. Throughput"></a>4. Throughput</h1><h2 id="4-1-Bidirection-1450-sizes"><a href="#4-1-Bidirection-1450-sizes" class="headerlink" title="4.1 Bidirection(1450 sizes)"></a>4.1 Bidirection(1450 sizes)</h2><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644372431377.png" alt="1644372431377.png"></p><h2 id="4-2-One-direction-1450-sizes"><a href="#4-2-One-direction-1450-sizes" class="headerlink" title="4.2 One direction(1450 sizes)"></a>4.2 One direction(1450 sizes)</h2><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644372481487.png" alt="1644372481487.png"></p><h2 id="4-3-Bidirection-256-sizes"><a href="#4-3-Bidirection-256-sizes" class="headerlink" title="4.3 Bidirection(256 sizes)"></a>4.3 Bidirection(256 sizes)</h2><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644372647021.png" alt="1644372647021.png"></p><h2 id="4-4-One-direction-256-sizes"><a href="#4-4-One-direction-256-sizes" class="headerlink" title="4.4 One direction(256 sizes)"></a>4.4 One direction(256 sizes)</h2><p><img src="/2022/02/07/Linux/openwrt/vxlan/1644372910007.png" alt="1644372910007.png"></p>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;1-Topology&quot;&gt;&lt;a href=&quot;#1-Topology&quot; class=&quot;headerlink&quot; title=&quot;1. Topology&quot;&gt;&lt;/a&gt;1. Topology&lt;/h1&gt;&lt;p&gt;&lt;img src=&quot;/2022/02/07/Linux/openwrt/</summary>
      
    
    
    
    
    <category term="OpenWRT" scheme="https://nfshanq.github.io/tags/OpenWRT/"/>
    
  </entry>
  
  <entry>
    <title>switch os upgrade with onie</title>
    <link href="https://nfshanq.github.io/2022/01/13/mellanox/switch/os-upgrade/"/>
    <id>https://nfshanq.github.io/2022/01/13/mellanox/switch/os-upgrade/</id>
    <published>2022-01-13T03:21:03.000Z</published>
    <updated>2022-01-13T01:39:39.809Z</updated>
    
    <content type="html"><![CDATA[<h1 id="prepare"><a href="#prepare" class="headerlink" title="prepare"></a>prepare</h1><h2 id="use-console-cable-to-connect-the-switch"><a href="#use-console-cable-to-connect-the-switch" class="headerlink" title="use console cable to connect the switch"></a>use console cable to connect the switch</h2><p>if we don’t have the ZTP environment. we need use console port to install the switch os. and we need a network cable to connect the mgmt interface.</p><h1 id="switch-the-next-boot-to-ONIE"><a href="#switch-the-next-boot-to-ONIE" class="headerlink" title="switch the next boot to ONIE"></a>switch the next boot to ONIE</h1><h2 id="check-the-grub-menuentry-and-switch-to-onie"><a href="#check-the-grub-menuentry-and-switch-to-onie" class="headerlink" title="check the grub menuentry and switch to onie"></a>check the grub menuentry and switch to onie</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">grep -i <span class="string">&quot;menuentry &#x27;&quot;</span> /boot/grub/grub.cfg|sed -r <span class="string">&quot;s|--class .*$||g&quot;</span>|nl -v 0</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> 0  menuentry <span class="string">&#x27;Cumulus-Linux GNU/Linux&#x27;</span></span><br><span class="line"> 1          menuentry <span class="string">&#x27;Cumulus-Linux GNU/Linux, with Linux 4.19.0-cl-1-amd64&#x27;</span></span><br><span class="line"> 2          menuentry <span class="string">&#x27;Cumulus-Linux GNU/Linux, with Linux 4.19.0-cl-1-amd64 (recovery mode)&#x27;</span></span><br><span class="line"> 3  menuentry ONIE &#123;</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">grub-reboot ONIE</span><br><span class="line"></span><br><span class="line">reboot</span><br></pre></td></tr></table></figure><h1 id="setup-the-onie-network"><a href="#setup-the-onie-network" class="headerlink" title="setup the onie network"></a>setup the onie network</h1><p>stop onie ztp</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">onie-stop</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure><p>add ip address on eth0(mgmt interface)</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">ip address add 100.64.31.101/24 dev eth0</span><br><span class="line">ip route add default via 100.64.31.1</span><br></pre></td></tr></table></figure><h1 id="install-the-cumulus-system"><a href="#install-the-cumulus-system" class="headerlink" title="install the cumulus system"></a>install the cumulus system</h1><h2 id="start-a-web-serivce-with-python3"><a href="#start-a-web-serivce-with-python3" class="headerlink" title="start a web serivce with python3"></a>start a web serivce with python3</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">python3 -m http.server</span><br><span class="line">Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...</span><br></pre></td></tr></table></figure><p>install cumulus os</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">ONIE:/ <span class="comment"># onie-nos-install http://100.64.21.13:8000/cumulus-linux-5.0.0-mlx-amd64</span></span><br><span class="line">.bin</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;prepare&quot;&gt;&lt;a href=&quot;#prepare&quot; class=&quot;headerlink&quot; title=&quot;prepare&quot;&gt;&lt;/a&gt;prepare&lt;/h1&gt;&lt;h2 id=&quot;use-console-cable-to-connect-the-switch&quot;&gt;&lt;a h</summary>
      
    
    
    
    <category term="mellanox" scheme="https://nfshanq.github.io/categories/mellanox/"/>
    
    
    <category term="mellanox" scheme="https://nfshanq.github.io/tags/mellanox/"/>
    
  </entry>
  
  <entry>
    <title>proxy arp</title>
    <link href="https://nfshanq.github.io/2021/12/27/golang/proxy_arp/"/>
    <id>https://nfshanq.github.io/2021/12/27/golang/proxy_arp/</id>
    <published>2021-12-27T11:27:40.000Z</published>
    <updated>2021-12-28T01:37:23.662Z</updated>
    
    <content type="html"><![CDATA[<h1 id="proxy-ARP-golang"><a href="#proxy-ARP-golang" class="headerlink" title="proxy ARP golang"></a>proxy ARP golang</h1><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">&quot;flag&quot;</span></span><br><span class="line"><span class="string">&quot;log&quot;</span></span><br><span class="line"><span class="string">&quot;net&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="string">&quot;github.com/mdlayher/arp&quot;</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> cliFlag <span class="keyword">struct</span> &#123;</span><br><span class="line">Ifname <span class="keyword">string</span></span><br><span class="line">Debug  <span class="keyword">bool</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> cli cliFlag</span><br><span class="line">flag.StringVar(&amp;cli.Ifname, <span class="string">&quot;i&quot;</span>, <span class="string">&quot;vlan1000&quot;</span>, <span class="string">&quot;Proxy Arp interface name&quot;</span>)</span><br><span class="line">flag.BoolVar(&amp;cli.Debug, <span class="string">&quot;v&quot;</span>, <span class="literal">false</span>, <span class="string">&quot;show verbose informations&quot;</span>)</span><br><span class="line">flag.Parse()</span><br><span class="line"></span><br><span class="line">ifname, err := net.InterfaceByName(cli.Ifname)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalln(<span class="string">&quot;get interface by name: &quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line">client, _ := arp.Dial(ifname)</span><br><span class="line"><span class="keyword">for</span> &#123;</span><br><span class="line">arp_recv, _, err := client.Read()</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalln(<span class="string">&quot;read packets error: &quot;</span>, err)</span><br><span class="line"></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">if</span> cli.Debug &#123;</span><br><span class="line">log.Println(<span class="string">&quot;recv arp: &quot;</span>, arp_recv)</span><br><span class="line">&#125;</span><br><span class="line">arp_replay, err := arp.NewPacket(arp.OperationReply, ifname.HardwareAddr, arp_recv.TargetIP, arp_recv.SenderHardwareAddr, arp_recv.SenderIP)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalln(<span class="string">&quot;create packets error: &quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line">destinationMAC := arp_recv.SenderHardwareAddr</span><br><span class="line">err = client.WriteTo(arp_replay, destinationMAC)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line"></span><br><span class="line">log.Fatalln(<span class="string">&quot;sent packet error: &quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">if</span> cli.Debug &#123;</span><br><span class="line">log.Println(<span class="string">&quot;sent arp: &quot;</span>, arp_replay)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure><p>go.mod</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">module proxy_arp</span><br><span class="line"></span><br><span class="line"><span class="keyword">go</span> <span class="number">1.17</span></span><br><span class="line"></span><br><span class="line">require github.com/google/gopacket v1<span class="number">.1</span><span class="number">.19</span></span><br><span class="line"></span><br><span class="line">require (</span><br><span class="line">github.com/mdlayher/arp v0<span class="number">.0</span><span class="number">.0</span><span class="number">-20191213142603</span>-f72070a231fc <span class="comment">// indirect</span></span><br><span class="line">github.com/mdlayher/ethernet v0<span class="number">.0</span><span class="number">.0</span><span class="number">-20190606142754</span><span class="number">-0394541</span>c37b7 <span class="comment">// indirect</span></span><br><span class="line">github.com/mdlayher/raw v0<span class="number">.0</span><span class="number">.0</span><span class="number">-20211126142749</span><span class="number">-4</span>eae47f3d54b <span class="comment">// indirect</span></span><br><span class="line">golang.org/x/net v0<span class="number">.0</span><span class="number">.0</span><span class="number">-20211216030914</span>-fe4d6282115f <span class="comment">// indirect</span></span><br><span class="line">golang.org/x/sys v0<span class="number">.0</span><span class="number">.0</span><span class="number">-20211216021012</span><span class="number">-1</span>d35b9e2eb4e <span class="comment">// indirect</span></span><br><span class="line">)</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>makefile</p><figure class="highlight makefile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">linux-arm64:</span></span><br><span class="line">CGO_ENABLED=0 GOOS=linux  GOARCH=arm64 go build -o ./bin/proxy_arp.arm64 src/main.go </span><br><span class="line"><span class="section">linux:</span></span><br><span class="line">CGO_ENABLED=1 GOOS=linux  GOARCH=amd64 go build -o ./bin/proxy_arp src/main.go </span><br><span class="line"> </span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;proxy-ARP-golang&quot;&gt;&lt;a href=&quot;#proxy-ARP-golang&quot; class=&quot;headerlink&quot; title=&quot;proxy ARP golang&quot;&gt;&lt;/a&gt;proxy ARP golang&lt;/h1&gt;&lt;figure class=&quot;hi</summary>
      
    
    
    
    <category term="golang" scheme="https://nfshanq.github.io/categories/golang/"/>
    
    
    <category term="proxy_arp" scheme="https://nfshanq.github.io/tags/proxy-arp/"/>
    
  </entry>
  
  <entry>
    <title>Linux Common Command</title>
    <link href="https://nfshanq.github.io/2021/12/25/Linux/common_cmd/Linux-common-cmd/"/>
    <id>https://nfshanq.github.io/2021/12/25/Linux/common_cmd/Linux-common-cmd/</id>
    <published>2021-12-25T03:27:40.000Z</published>
    <updated>2021-12-25T13:02:38.060Z</updated>
    
    <content type="html"><![CDATA[<h1 id="Linux-Common-Command-notes"><a href="#Linux-Common-Command-notes" class="headerlink" title="Linux Common Command notes"></a>Linux Common Command notes</h1>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;Linux-Common-Command-notes&quot;&gt;&lt;a href=&quot;#Linux-Common-Command-notes&quot; class=&quot;headerlink&quot; title=&quot;Linux Common Command notes&quot;&gt;&lt;/a&gt;Linux Co</summary>
      
    
    
    
    <category term="Linux" scheme="https://nfshanq.github.io/categories/Linux/"/>
    
    <category term="Command" scheme="https://nfshanq.github.io/categories/Linux/Command/"/>
    
    
    <category term="Linux Common Command" scheme="https://nfshanq.github.io/tags/Linux-Common-Command/"/>
    
  </entry>
  
  <entry>
    <title>cloud-init</title>
    <link href="https://nfshanq.github.io/2021/12/19/Linux/virt/cloud-init/"/>
    <id>https://nfshanq.github.io/2021/12/19/Linux/virt/cloud-init/</id>
    <published>2021-12-19T03:27:40.000Z</published>
    <updated>2021-12-19T11:38:05.008Z</updated>
    
    <content type="html"><![CDATA[<h1 id="Cloud-init-guide"><a href="#Cloud-init-guide" class="headerlink" title="Cloud init guide"></a>Cloud init guide</h1><p>ref: <a href="https://cloudinit.readthedocs.io/en/latest/topics/examples.html">https://cloudinit.readthedocs.io/en/latest/topics/examples.html</a></p><h2 id="1-Install-tools-on-the-hypervisor-machine"><a href="#1-Install-tools-on-the-hypervisor-machine" class="headerlink" title="1. Install tools on the hypervisor machine"></a>1. Install tools on the hypervisor machine</h2><h3 id="1-1-Install-kvm-qemu-virt"><a href="#1-1-Install-kvm-qemu-virt" class="headerlink" title="1.1 Install kvm qemu virt"></a>1.1 Install kvm qemu virt</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst</span><br></pre></td></tr></table></figure><h3 id="1-2-Install-cloud-init-tools"><a href="#1-2-Install-cloud-init-tools" class="headerlink" title="1.2 Install cloud init tools"></a>1.2 Install cloud init tools</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt install -y cloud-utils</span><br></pre></td></tr></table></figure><h2 id="download-a-cloud-image"><a href="#download-a-cloud-image" class="headerlink" title="download a cloud image"></a>download a cloud image</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">wget https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img</span><br></pre></td></tr></table></figure><h2 id="create-a-vm-image"><a href="#create-a-vm-image" class="headerlink" title="create a vm image"></a>create a vm image</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">qemu-img create -f qcow2 -b ubuntu-20.04-server-cloudimg-amd64.img client.qcow2</span><br><span class="line">qemu-img resize client.qcow2 +10G</span><br></pre></td></tr></table></figure><h2 id="create-a-cloud-init-file"><a href="#create-a-cloud-init-file" class="headerlink" title="create a cloud init file"></a>create a cloud init file</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; cloud-init.yaml &lt;&lt; <span class="string">EOF</span></span><br><span class="line"><span class="string">#cloud-config</span></span><br><span class="line"><span class="string">hostname: cloudimg.local</span></span><br><span class="line"><span class="string">user: root</span></span><br><span class="line"><span class="string">password: lab123</span></span><br><span class="line"><span class="string">chpasswd: &#123; expire: False &#125;</span></span><br><span class="line"><span class="string">ssh_pwauth: True</span></span><br><span class="line"><span class="string">users:</span></span><br><span class="line"><span class="string">  - name: lab</span></span><br><span class="line"><span class="string">    shell: /bin/bash</span></span><br><span class="line"><span class="string">    sudo: ALL=(ALL) NOPASSWD:ALL</span></span><br><span class="line"><span class="string">    groups: root, sudo</span></span><br><span class="line"><span class="string">    plain_text_passwd: &#x27;lab123&#x27;</span></span><br><span class="line"><span class="string">    lock_passwd: false</span></span><br><span class="line"><span class="string">growpart:</span></span><br><span class="line"><span class="string">  mode: auto</span></span><br><span class="line"><span class="string">  devices: [&#x27;/&#x27;]</span></span><br><span class="line"><span class="string">locale: en_US.UTF-8</span></span><br><span class="line"><span class="string">timezone: Asia/Shanghai</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure><h2 id="create-network-config-file"><a href="#create-network-config-file" class="headerlink" title="create network config file"></a>create network config file</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">cat &gt; network.yaml &lt;&lt; <span class="string">EOF  </span></span><br><span class="line"><span class="string">ethernets:</span></span><br><span class="line"><span class="string">  enp1s0:</span></span><br><span class="line"><span class="string">    dhcp4: false</span></span><br><span class="line"><span class="string">    addresses:</span></span><br><span class="line"><span class="string">    - 11.1.10.12/24</span></span><br><span class="line"><span class="string">    gateway4: 11.1.10.1</span></span><br><span class="line"><span class="string">  enp2s0:</span></span><br><span class="line"><span class="string">    addresses: [ 192.168.122.110/24 ]</span></span><br><span class="line"><span class="string">    #gateway4: 192.168.122.1</span></span><br><span class="line"><span class="string">    #nameservers:</span></span><br><span class="line"><span class="string">    #  addresses: [114.114.114.114]</span></span><br><span class="line"><span class="string">version: 2</span></span><br><span class="line"><span class="string">EOF</span></span><br></pre></td></tr></table></figure><h2 id="create-the-cloud-init-iso-file"><a href="#create-the-cloud-init-iso-file" class="headerlink" title="create the cloud-init iso file"></a>create the cloud-init iso file</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cloud-localds -m <span class="built_in">local</span> -N network.yaml cloud-init.iso cloud-init.yaml</span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="boot-a-vm-by-virt-install"><a href="#boot-a-vm-by-virt-install" class="headerlink" title="boot a vm by virt-install"></a>boot a vm by virt-install</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">virt-install \</span><br><span class="line">    --name vm1 \</span><br><span class="line">    --ram=8192 \</span><br><span class="line">    --vcpus 4 \</span><br><span class="line">    --os-type linux \</span><br><span class="line">    --os-variant ubuntu20.04 \</span><br><span class="line">    --graphics none \</span><br><span class="line">    --disk /home/lab/cloud/client.qcow2,device=disk,bus=virtio \</span><br><span class="line">    --disk /home/lab/cloud/cloud-init.iso,device=cdrom \</span><br><span class="line">    --network bridge=br10,model=virtio \</span><br><span class="line">    --network bridge=virbr0,model=virtio \</span><br><span class="line">    --import</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;Cloud-init-guide&quot;&gt;&lt;a href=&quot;#Cloud-init-guide&quot; class=&quot;headerlink&quot; title=&quot;Cloud init guide&quot;&gt;&lt;/a&gt;Cloud init guide&lt;/h1&gt;&lt;p&gt;ref: &lt;a href=&quot;</summary>
      
    
    
    
    <category term="Linux" scheme="https://nfshanq.github.io/categories/Linux/"/>
    
    <category term="Virt" scheme="https://nfshanq.github.io/categories/Linux/Virt/"/>
    
    
    <category term="virt" scheme="https://nfshanq.github.io/tags/virt/"/>
    
  </entry>
  
  <entry>
    <title>guestfish</title>
    <link href="https://nfshanq.github.io/2021/12/19/Linux/virt/guestfish/"/>
    <id>https://nfshanq.github.io/2021/12/19/Linux/virt/guestfish/</id>
    <published>2021-12-19T03:27:40.000Z</published>
    <updated>2021-12-28T09:02:15.439Z</updated>
    
    <content type="html"><![CDATA[<h1 id="guestfish"><a href="#guestfish" class="headerlink" title="guestfish"></a>guestfish</h1><h2 id="open-a-image-file"><a href="#open-a-image-file" class="headerlink" title="open a image file"></a>open a image file</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">guestfish --rw -a ./ubuntu-20.04-server-cloudimg-amd64.img</span><br><span class="line">Welcome to guestfish, the guest filesystem shell <span class="keyword">for</span></span><br><span class="line">editing virtual machine filesystems and disk images.</span><br><span class="line"></span><br><span class="line">Type: ‘<span class="built_in">help</span>’ <span class="keyword">for</span> <span class="built_in">help</span> on commands</span><br><span class="line">      ‘man’ to <span class="built_in">read</span> the manual</span><br><span class="line">      ‘quit’ to quit the shell</span><br><span class="line"></span><br><span class="line">&gt;&lt;fs&gt;</span><br><span class="line">&gt;&lt;fs&gt; run</span><br><span class="line">&gt;&lt;fs&gt; list-filesystems</span><br><span class="line">&gt;&lt;fs&gt; mount /dev/sda1 /</span><br><span class="line">&gt;&lt;fs&gt; cat /etc/shadow</span><br><span class="line">&gt;&lt;fs&gt; quit</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;guestfish&quot;&gt;&lt;a href=&quot;#guestfish&quot; class=&quot;headerlink&quot; title=&quot;guestfish&quot;&gt;&lt;/a&gt;guestfish&lt;/h1&gt;&lt;h2 id=&quot;open-a-image-file&quot;&gt;&lt;a href=&quot;#open-a-i</summary>
      
    
    
    
    <category term="Linux" scheme="https://nfshanq.github.io/categories/Linux/"/>
    
    <category term="Virt" scheme="https://nfshanq.github.io/categories/Linux/Virt/"/>
    
    
    <category term="virt" scheme="https://nfshanq.github.io/tags/virt/"/>
    
  </entry>
  
  <entry>
    <title>virt customize</title>
    <link href="https://nfshanq.github.io/2021/12/19/Linux/virt/virt-customize/"/>
    <id>https://nfshanq.github.io/2021/12/19/Linux/virt/virt-customize/</id>
    <published>2021-12-19T03:27:40.000Z</published>
    <updated>2021-12-19T11:37:58.200Z</updated>
    
    <content type="html"><![CDATA[<h1 id="virt-customize-tools"><a href="#virt-customize-tools" class="headerlink" title="virt-customize tools"></a>virt-customize tools</h1><p>ref: <a href="https://libguestfs.org/virt-customize.1.html">https://libguestfs.org/virt-customize.1.html</a></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt install -y libguestfs-tools</span><br></pre></td></tr></table></figure><h2 id="command-help"><a href="#command-help" class="headerlink" title="command help"></a>command help</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br></pre></td><td class="code"><pre><span class="line">virt-customize --<span class="built_in">help</span></span><br><span class="line">virt-customize: customize a virtual machine</span><br><span class="line"></span><br><span class="line"> virt-customize [--options] -d domname</span><br><span class="line"></span><br><span class="line"> virt-customize [--options] -a disk.img [-a disk.img ...]</span><br><span class="line"></span><br><span class="line">A short summary of the options is given below.  For detailed <span class="built_in">help</span> please</span><br><span class="line"><span class="built_in">read</span> the man page virt-customize(1).</span><br><span class="line"></span><br><span class="line">Options:</span><br><span class="line">  -a, --add &lt;file&gt;                    Add disk image file</span><br><span class="line">  --append-line &lt;FILE:LINE&gt;           Append line(s) to the file</span><br><span class="line">  --attach &lt;iso&gt;                      Attach data disk/ISO during install</span><br><span class="line">  --attach-format &lt;format&gt;            Set attach disk format</span><br><span class="line">  -c, --connect &lt;uri&gt;                 Set libvirt URI</span><br><span class="line">  --chmod &lt;PERMISSIONS:FILE&gt;          Change the permissions of a file</span><br><span class="line">  --color, --colors, --colour, --colours</span><br><span class="line">                                      Use ANSI colour sequences even <span class="keyword">if</span> not tty</span><br><span class="line">  --commands-from-file &lt;FILENAME&gt;     Read customize commands from file</span><br><span class="line">  --copy &lt;SOURCE:DEST&gt;                Copy files <span class="keyword">in</span> disk image</span><br><span class="line">  --copy-in &lt;LOCALPATH:REMOTEDIR&gt;     Copy <span class="built_in">local</span> files or directories into image</span><br><span class="line">  -d, --domain &lt;domain&gt;               Set libvirt guest name</span><br><span class="line">  --delete &lt;PATH&gt;                     Delete a file or directory</span><br><span class="line">  -n, --dryrun, --dry-run             Perform a dry run</span><br><span class="line">  --echo-keys                         Don’t turn off <span class="built_in">echo</span> <span class="keyword">for</span> passphrases</span><br><span class="line">  --edit &lt;FILE:EXPR&gt;                  Edit file using Perl expression</span><br><span class="line">  --firstboot &lt;SCRIPT&gt;                Run script at first guest boot</span><br><span class="line">  --firstboot-command &lt;<span class="string">&#x27;CMD+ARGS&#x27;</span>&gt;    Run <span class="built_in">command</span> at first guest boot</span><br><span class="line">  --firstboot-install &lt;PKG,PKG..&gt;     Add package(s) to install at first boot</span><br><span class="line">  --format &lt;format&gt;                   Set format (default: auto)</span><br><span class="line">  --<span class="built_in">help</span>                              Display brief <span class="built_in">help</span></span><br><span class="line">  --hostname &lt;HOSTNAME&gt;               Set the hostname</span><br><span class="line">  --install &lt;PKG,PKG..&gt;               Add package(s) to install</span><br><span class="line">  --key &lt;SELECTOR&gt;                    Specify a LUKS key</span><br><span class="line">  --keys-from-stdin                   Read passphrases from stdin</span><br><span class="line">  --link &lt;TARGET:LINK[:LINK..]&gt;       Create symbolic links</span><br><span class="line">  -m, --memsize &lt;mb&gt;                  Set memory size</span><br><span class="line">  --mkdir &lt;DIR&gt;                       Create a directory</span><br><span class="line">  --move &lt;SOURCE:DEST&gt;                Move files <span class="keyword">in</span> disk image</span><br><span class="line">  --network                           Enable appliance network (default)</span><br><span class="line">  --no-logfile                        Scrub build <span class="built_in">log</span> file</span><br><span class="line">  --no-network                        Disable appliance network</span><br><span class="line">  --password &lt;USER:SELECTOR&gt;          Set user password</span><br><span class="line">  --password-crypto &lt;md5|sha256|sha512&gt;</span><br><span class="line">                                      Set password crypto</span><br><span class="line">  -q, --quiet                         Don’t <span class="built_in">print</span> progress messages</span><br><span class="line">  --root-password &lt;SELECTOR&gt;          Set root password</span><br><span class="line">  --run &lt;SCRIPT&gt;                      Run script <span class="keyword">in</span> disk image</span><br><span class="line">  --run-command &lt;<span class="string">&#x27;CMD+ARGS&#x27;</span>&gt;          Run <span class="built_in">command</span> <span class="keyword">in</span> disk image</span><br><span class="line">  --scrub &lt;FILE&gt;                      Scrub a file</span><br><span class="line">  --selinux-relabel                   Relabel files with correct SELinux labels</span><br><span class="line">  --sm-attach &lt;SELECTOR&gt;              Attach to a subscription-manager pool</span><br><span class="line">  --sm-credentials &lt;SELECTOR&gt;         Credentials <span class="keyword">for</span> subscription-manager</span><br><span class="line">  --sm-register                       Register using subscription-manager</span><br><span class="line">  --sm-remove                         Remove all the subscriptions</span><br><span class="line">  --sm-unregister                     Unregister using subscription-manager</span><br><span class="line">  --smp &lt;vcpus&gt;                       Set number of vCPUs</span><br><span class="line">  --ssh-inject &lt;USER[:SELECTOR]&gt;      Inject a public key into the guest</span><br><span class="line">  --timezone &lt;TIMEZONE&gt;               Set the default timezone</span><br><span class="line">  --touch &lt;FILE&gt;                      Run touch on a file</span><br><span class="line">  --truncate &lt;FILE&gt;                   Truncate a file to zero size</span><br><span class="line">  --truncate-recursive &lt;PATH&gt;         Recursively truncate all files <span class="keyword">in</span> directory</span><br><span class="line">  --uninstall &lt;PKG,PKG..&gt;             Uninstall package(s)</span><br><span class="line">  --update                            Update packages</span><br><span class="line">  --upload &lt;FILE:DEST&gt;                Upload <span class="built_in">local</span> file to destination</span><br><span class="line">  -V, --version                       Display version and <span class="built_in">exit</span></span><br><span class="line">  -v, --verbose                       Enable libguestfs debugging messages</span><br><span class="line">  --write &lt;FILE:CONTENT&gt;              Write file</span><br><span class="line">  -x                                  Enable tracing of libguestfs calls</span><br></pre></td></tr></table></figure><h2 id="1-设置root密码"><a href="#1-设置root密码" class="headerlink" title="1 设置root密码"></a>1 设置root密码</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \</span><br><span class="line">               --root-password password:lab123</span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[   0.0] Examining the guest ...</span><br><span class="line">[   4.3] Setting a random seed</span><br><span class="line">virt-customize: warning: random seed could not be <span class="built_in">set</span> <span class="keyword">for</span> this <span class="built_in">type</span> of</span><br><span class="line">guest</span><br><span class="line">[   4.4] Setting the machine ID <span class="keyword">in</span> /etc/machine-id</span><br><span class="line">[   4.4] Setting passwords</span><br><span class="line">[   5.8] Finishing off</span><br></pre></td></tr></table></figure><h2 id="2-在Image中安装软件包"><a href="#2-在Image中安装软件包" class="headerlink" title="2 在Image中安装软件包"></a>2 在Image中安装软件包</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \</span><br><span class="line">               --install net-tools,unzip</span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[   0.0] Examining the guest ...</span><br><span class="line">[   2.2] Setting a random seed</span><br><span class="line">virt-customize: warning: random seed could not be <span class="built_in">set</span> <span class="keyword">for</span> this <span class="built_in">type</span> of</span><br><span class="line">guest</span><br><span class="line">[   2.3] Installing packages: net-tools unzip</span><br><span class="line">[   9.2] Finishing off</span><br></pre></td></tr></table></figure><h2 id="3-copy-文件到镜像中"><a href="#3-copy-文件到镜像中" class="headerlink" title="3 copy 文件到镜像中"></a>3 copy 文件到镜像中</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \</span><br><span class="line">               --upload ../http_test:/http_test</span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[   0.0] Examining the guest ...</span><br><span class="line">[   2.2] Setting a random seed</span><br><span class="line">virt-customize: warning: random seed could not be <span class="built_in">set</span> <span class="keyword">for</span> this <span class="built_in">type</span> of</span><br><span class="line">guest</span><br><span class="line">[   2.2] Uploading: ../http_test to /http_test</span><br><span class="line">[   2.3] Finishing off</span><br></pre></td></tr></table></figure><h2 id="4-修改时区"><a href="#4-修改时区" class="headerlink" title="4 修改时区"></a>4 修改时区</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \</span><br><span class="line">               --timezone  <span class="string">&quot;Asia/Shanghai&quot;</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="5-上传SSH公钥"><a href="#5-上传SSH公钥" class="headerlink" title="5 上传SSH公钥"></a>5 上传SSH公钥</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \</span><br><span class="line">               --ssh-inject lab:file:./id_rsa.pub</span><br></pre></td></tr></table></figure><h2 id="6-在文件中添加内容"><a href="#6-在文件中添加内容" class="headerlink" title="6 在文件中添加内容"></a>6 在文件中添加内容</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \</span><br><span class="line">               --append-line <span class="string">&#x27;/etc/hosts:10.0.0.1 foo&#x27;</span></span><br></pre></td></tr></table></figure><h2 id="7-create-a-user-in-image"><a href="#7-create-a-user-in-image" class="headerlink" title="7 create a user in image"></a>7 create a user in image</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img  \</span><br><span class="line">               --run-command <span class="string">&#x27;useradd test  -s /bin/bash -m&#x27;</span></span><br><span class="line"></span><br><span class="line">virt-customize -a ubuntu-20.04-server-cloudimg-amd64.img \</span><br><span class="line">              --password <span class="built_in">test</span>:password:lab123</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[   0.0] Examining the guest ...</span><br><span class="line">[   2.2] Setting a random seed</span><br><span class="line">virt-customize: warning: random seed could not be <span class="built_in">set</span> <span class="keyword">for</span> this <span class="built_in">type</span> of</span><br><span class="line">guest</span><br><span class="line">[   2.3] Running: useradd <span class="built_in">test</span>  -s /bin/bash -m</span><br><span class="line">[   2.4] Finishing off</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">[   0.0] Examining the guest ...</span><br><span class="line">[   2.2] Setting a random seed</span><br><span class="line">virt-customize: warning: random seed could not be <span class="built_in">set</span> <span class="keyword">for</span> this <span class="built_in">type</span> of</span><br><span class="line">guest</span><br><span class="line">[   2.3] Setting passwords</span><br><span class="line">[   3.6] Finishing off</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;virt-customize-tools&quot;&gt;&lt;a href=&quot;#virt-customize-tools&quot; class=&quot;headerlink&quot; title=&quot;virt-customize tools&quot;&gt;&lt;/a&gt;virt-customize tools&lt;/h1&gt;&lt;</summary>
      
    
    
    
    <category term="Linux" scheme="https://nfshanq.github.io/categories/Linux/"/>
    
    <category term="Virt" scheme="https://nfshanq.github.io/categories/Linux/Virt/"/>
    
    
    <category term="virt" scheme="https://nfshanq.github.io/tags/virt/"/>
    
  </entry>
  
  <entry>
    <title>create vm example by virt-install</title>
    <link href="https://nfshanq.github.io/2021/12/19/Linux/virt/create-vm/"/>
    <id>https://nfshanq.github.io/2021/12/19/Linux/virt/create-vm/</id>
    <published>2021-12-19T03:27:40.000Z</published>
    <updated>2021-12-25T11:57:10.472Z</updated>
    
    <content type="html"><![CDATA[<h1 id="create-vm-example"><a href="#create-vm-example" class="headerlink" title="create vm example"></a>create vm example</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="built_in">set</span> -e</span><br><span class="line"><span class="built_in">set</span> -x</span><br><span class="line">ID=<span class="variable">$1</span></span><br><span class="line"></span><br><span class="line">BACKFILE_IMG=<span class="string">&quot;./ubuntu-20.04-server-cloudimg-amd64.img&quot;</span></span><br><span class="line">TARGET_IMG=<span class="string">&quot;client-<span class="variable">$&#123;ID&#125;</span>.qcow2&quot;</span></span><br><span class="line">TARGET_IMG_SIZE=<span class="string">&quot;20G&quot;</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">ADDFILES=<span class="string">&quot;http_test&quot;</span></span><br><span class="line"></span><br><span class="line">HOSTNAME=<span class="string">&quot;client-<span class="variable">$&#123;ID&#125;</span>&quot;</span></span><br><span class="line">IF1_NAME=<span class="string">&quot;enp1s0&quot;</span></span><br><span class="line">IF1_IPADDRESS=<span class="string">&quot;11.1.11.<span class="subst">$(( 20 + $&#123;ID&#125; )</span>)/24&quot;</span></span><br><span class="line">IF1_GATEWAY=<span class="string">&quot;11.1.11.1&quot;</span></span><br><span class="line"></span><br><span class="line">IF2_NAME=<span class="string">&quot;enp2s0&quot;</span></span><br><span class="line">IF2_IPADDRESS=<span class="string">&quot;192.168.122.<span class="subst">$(( 30 + $&#123;ID&#125; )</span>)/24&quot;</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">## create a image from backfile</span></span><br><span class="line">qemu-img create -f qcow2 -b <span class="variable">$BACKFILE_IMG</span> <span class="variable">$TARGET_IMG</span></span><br><span class="line"><span class="comment">## resize target image size</span></span><br><span class="line">qemu-img resize <span class="variable">$TARGET_IMG</span> <span class="variable">$TARGET_IMG_SIZE</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## add user to image</span></span><br><span class="line"><span class="comment">#virt-customize -a $&#123;TARGET_IMG&#125;  --run-command &#x27;useradd lab -s /bin/bash -m -p &quot;&quot; -G sudo ; chage -d 99999 lab&#x27;</span></span><br><span class="line"><span class="comment">## set user password</span></span><br><span class="line"><span class="comment">#virt-customize -a $&#123;TARGET_IMG&#125; --password test:password:lab123</span></span><br><span class="line"><span class="comment">## set root password</span></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --root-password password:lab123</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">## upload http_test and script</span></span><br><span class="line">http_test_script=$(mktemp /tmp/httpXXXXXXXX.sh)</span><br><span class="line">cat &gt; <span class="variable">$&#123;http_test_script&#125;</span> &lt;&lt; <span class="string">EOF</span></span><br><span class="line"><span class="string">#!/bin/bash</span></span><br><span class="line"><span class="string">set -x</span></span><br><span class="line"><span class="string">while true</span></span><br><span class="line"><span class="string">do</span></span><br><span class="line"><span class="string">  /http_test -c 11.1.10.10 -s 90000000</span></span><br><span class="line"><span class="string">  sleep  \$(( \$RANDOM % 1 ))</span></span><br><span class="line"><span class="string">done</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --upload http_test:/http_test</span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --upload <span class="variable">$&#123;http_test_script&#125;</span>:/http.sh</span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --run-command <span class="string">&#x27;chmod +x /http.sh&#x27;</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## run script on image on firstboot</span></span><br><span class="line">script=$(mktemp /tmp/scriptXXXXXXXXXX.sh)</span><br><span class="line">cat &gt; <span class="variable">$&#123;script&#125;</span> &lt;&lt; <span class="string">EOF</span></span><br><span class="line"><span class="string">#!/bin/bash</span></span><br><span class="line"><span class="string">ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -q -N &quot;&quot;</span></span><br><span class="line"><span class="string">ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -q -N &quot;&quot;</span></span><br><span class="line"><span class="string">chmod 0600 /etc/ssh/ssh_host*</span></span><br><span class="line"><span class="string">echo PermitRootLogin yes &gt;&gt; /etc/ssh/sshd_config</span></span><br><span class="line"><span class="string">sed -i &quot;s/PasswordAuthentication no/PasswordAuthentication yes/g&quot; /etc/ssh/sshd_config</span></span><br><span class="line"><span class="string">hostnamectl set-hostname $&#123;HOSTNAME&#125;</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --firstboot <span class="variable">$&#123;script&#125;</span></span><br><span class="line">rm -rf <span class="variable">$&#123;script&#125;</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## create network config file</span></span><br><span class="line">network_cfg_tmp=$(mktemp /tmp/networkXXXXXXXX.yaml)</span><br><span class="line">cat &gt; <span class="variable">$&#123;network_cfg_tmp&#125;</span> &lt;&lt; <span class="string">EOF</span></span><br><span class="line"><span class="string">network:</span></span><br><span class="line"><span class="string">  ethernets:</span></span><br><span class="line"><span class="string">    $&#123;IF1_NAME&#125;:</span></span><br><span class="line"><span class="string">      dhcp4: false</span></span><br><span class="line"><span class="string">      addresses:</span></span><br><span class="line"><span class="string">      - $&#123;IF1_IPADDRESS&#125;</span></span><br><span class="line"><span class="string">      gateway4: $&#123;IF1_GATEWAY&#125;</span></span><br><span class="line"><span class="string">    $&#123;IF2_NAME&#125;:</span></span><br><span class="line"><span class="string">      addresses: [ $&#123;IF2_IPADDRESS&#125; ]</span></span><br><span class="line"><span class="string">  version: 2</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## add network config</span></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --run-command <span class="string">&#x27;rm -rf /etc/netplan/*&#x27;</span></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --upload <span class="variable">$&#123;network_cfg_tmp&#125;</span>:/etc/netplan/00-installer-config.yaml</span><br><span class="line">rm -rf <span class="variable">$&#123;network_cfg_tmp&#125;</span></span><br><span class="line"></span><br><span class="line"><span class="comment">### add start script</span></span><br><span class="line">start_script=$(mktemp /tmp/startXXXXXXXX)</span><br><span class="line"></span><br><span class="line">cat &gt; <span class="variable">$&#123;start_script&#125;</span> &lt;&lt; <span class="string">EOF</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[Unit]</span></span><br><span class="line"><span class="string">Description=start http test</span></span><br><span class="line"><span class="string">After=network.target</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">[Service]</span></span><br><span class="line"><span class="string">ExecStart=bash -c /http.sh</span></span><br><span class="line"><span class="string">[Install]</span></span><br><span class="line"><span class="string">WantedBy=multi-user.target</span></span><br><span class="line"><span class="string">EOF</span></span><br><span class="line"></span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --upload <span class="variable">$&#123;start_script&#125;</span>:/lib/systemd/system/http_test.service</span><br><span class="line">virt-customize -a <span class="variable">$&#123;TARGET_IMG&#125;</span> --run-command <span class="string">&#x27;systemctl daemon-reload; systemctl enable http_test&#x27;</span></span><br><span class="line">rm -rf <span class="variable">$&#123;start_script&#125;</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">## boot vm</span></span><br><span class="line">virt-install \</span><br><span class="line">    --name <span class="variable">$&#123;HOSTNAME&#125;</span> \</span><br><span class="line">    --ram=8192 \</span><br><span class="line">    --vcpus 4 \</span><br><span class="line">    --os-type linux \</span><br><span class="line">    --os-variant ubuntu20.04 \</span><br><span class="line">    --graphics none \</span><br><span class="line">    --disk /home/lab/cloud/<span class="variable">$&#123;TARGET_IMG&#125;</span>,device=disk,bus=virtio \</span><br><span class="line">    --network bridge=br11,model=virtio \</span><br><span class="line">    --network bridge=virbr0,model=virtio \</span><br><span class="line">    --noautoconsole \</span><br><span class="line">    --import</span><br></pre></td></tr></table></figure><h2 id="resize-the-disk-size"><a href="#resize-the-disk-size" class="headerlink" title="resize the disk size"></a>resize the disk size</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">growpart /dev/vda 1</span><br><span class="line">resize2fs /dev/vda1</span><br><span class="line"></span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;create-vm-example&quot;&gt;&lt;a href=&quot;#create-vm-example&quot; class=&quot;headerlink&quot; title=&quot;create vm example&quot;&gt;&lt;/a&gt;create vm example&lt;/h1&gt;&lt;figure class</summary>
      
    
    
    
    <category term="Linux" scheme="https://nfshanq.github.io/categories/Linux/"/>
    
    <category term="Virt" scheme="https://nfshanq.github.io/categories/Linux/Virt/"/>
    
    
    <category term="virt" scheme="https://nfshanq.github.io/tags/virt/"/>
    
  </entry>
  
  <entry>
    <title>bluefield-bfb-install</title>
    <link href="https://nfshanq.github.io/2021/12/19/mellanox/bluefield/bluefield-bfb-install/"/>
    <id>https://nfshanq.github.io/2021/12/19/mellanox/bluefield/bluefield-bfb-install/</id>
    <published>2021-12-19T03:21:03.000Z</published>
    <updated>2021-12-19T11:47:24.574Z</updated>
    
    <content type="html"><![CDATA[<h1 id="bluefield-bfb-install"><a href="#bluefield-bfb-install" class="headerlink" title="bluefield bfb install"></a>bluefield bfb install</h1><p>ref: <a href="https://docs.nvidia.com/doca/sdk/installation-guide/index.html">https://docs.nvidia.com/doca/sdk/installation-guide/index.html</a></p><h2 id="1-install-rshim-and-pv"><a href="#1-install-rshim-and-pv" class="headerlink" title="1. install rshim and pv"></a>1. install rshim and pv</h2><p>install rshim:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">dpkg -i rshim_2.0.6-3.ge329c69_amd64.deb</span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Selecting previously unselected package rshim.</span><br><span class="line">(Reading database ... 71752 files and directories currently installed.)</span><br><span class="line">Preparing to unpack rshim_2.0.6-3.ge329c69_amd64.deb ...</span><br><span class="line">Unpacking rshim (2.0.6-3.ge329c69) ...</span><br><span class="line">Setting up rshim (2.0.6-3.ge329c69) ...</span><br><span class="line">Created symlink /etc/systemd/system/multi-user.target.wants/rshim.service → /lib/systemd/system/rshim.service.</span><br><span class="line">Processing triggers <span class="keyword">for</span> man-db (2.9.1-1) ...</span><br></pre></td></tr></table></figure><p>install pv:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">apt install -y pv</span><br></pre></td></tr></table></figure><h2 id="2-start-and-check-rshim-service"><a href="#2-start-and-check-rshim-service" class="headerlink" title="2 start and check rshim service"></a>2 start and check rshim service</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">systemctl is-enabled rshim</span><br><span class="line">enabled</span><br><span class="line"></span><br><span class="line">systemctl start rshim</span><br><span class="line"></span><br><span class="line">systemctl status rshim</span><br><span class="line">● rshim.service - rshim driver <span class="keyword">for</span> BlueField SoC</span><br><span class="line">     Loaded: loaded (/lib/systemd/system/rshim.service; enabled; vendor preset: enabled)</span><br><span class="line">     Active: active (running) since Fri 2021-12-17 07:01:48 UTC; 14min ago</span><br><span class="line">       Docs: man:rshim(8)</span><br><span class="line">    Process: 10663 ExecStart=/usr/sbin/rshim <span class="variable">$OPTIONS</span> (code=exited, status=0/SUCCESS)</span><br><span class="line">   Main PID: 10666 (rshim)</span><br><span class="line">      Tasks: 6 (<span class="built_in">limit</span>: 629145)</span><br><span class="line">     Memory: 2.2M</span><br><span class="line">     CGroup: /system.slice/rshim.service</span><br><span class="line">             └─10666 /usr/sbin/rshim</span><br><span class="line"></span><br><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Starting rshim driver <span class="keyword">for</span> BlueField SoC...</span><br><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 systemd[1]: Started rshim driver <span class="keyword">for</span> BlueField SoC.</span><br><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2</span><br><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2</span><br><span class="line">Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached</span><br><span class="line">Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open</span><br><span class="line">Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close</span><br><span class="line"></span><br></pre></td></tr></table></figure><h3 id="2-1-check-rshim"><a href="#2-1-check-rshim" class="headerlink" title="2.1 check rshim"></a>2.1 check rshim</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># ls -lh /dev/rshim0/</span></span><br><span class="line">total 0</span><br><span class="line">crw------- 1 root root 235, 0 Dec 17 07:01 boot</span><br><span class="line">crw------- 1 root root 234, 0 Dec 17 07:01 console</span><br><span class="line">crw------- 1 root root 511, 0 Dec 17 07:01 misc</span><br><span class="line">crw------- 1 root root 236, 0 Dec 17 07:01 rshim</span><br><span class="line"></span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: Probing pcie-0000:3b:00.2</span><br><span class="line">Dec 17 07:01:48 ubuntu2004-lab1 rshim[10666]: create rshim pcie-0000:3b:00.2</span><br><span class="line">Dec 17 07:01:49 ubuntu2004-lab1 rshim[10666]: rshim0 attached</span><br><span class="line">Dec 17 07:08:03 ubuntu2004-lab1 rshim[10666]: rshim0 boot open</span><br><span class="line">Dec 17 07:08:13 ubuntu2004-lab1 rshim[10666]: rshim0 boot close</span><br><span class="line"></span><br><span class="line">pcie-0000:3b:00.2  --&gt; rshim0</span><br></pre></td></tr></table></figure><h2 id="3-install-bfb-to-bluefield-NIC"><a href="#3-install-bfb-to-bluefield-NIC" class="headerlink" title="3 install bfb to bluefield NIC"></a>3 install bfb to bluefield NIC</h2><h3 id="3-1-create-login-password"><a href="#3-1-create-login-password" class="headerlink" title="3.1 create login password"></a>3.1 create login password</h3><p>The password is lab123 </p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">&quot;ubuntu_PASSWORD=&#x27;<span class="subst">$(openssl passwd -1 lab123)</span>&#x27;&quot;</span> | tee bf.cfg</span><br></pre></td></tr></table></figure><h3 id="3-2-install-bfb-to-NIC"><a href="#3-2-install-bfb-to-NIC" class="headerlink" title="3.2 install bfb to NIC"></a>3.2 install bfb to NIC</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bfb-install --rshim &lt;rshimN&gt; --bfb &lt;image_path.bfb&gt; --config bf.cfg</span><br></pre></td></tr></table></figure><p>example:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bfb-install --rshim rshim0 --bfb DOCA_v1.2.0_BlueField_OS_Ubuntu_20.04-5.4.0-1022-bluefield-5.5-1.0.3.2-3.8.0.11969-1.signed-aarch64.bfb --config bf.cfg</span><br></pre></td></tr></table></figure><p>output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">Pushing bfb + cfg</span><br><span class="line"> 623MiB 0:00:54 [11.4MiB/s] [                                       &lt;=&gt;                                               ]</span><br><span class="line">Collecting BlueField booting status. Press Ctrl+C to stop…</span><br><span class="line"> INFO[BL2]: start</span><br><span class="line"> INFO[BL2]: DDR POST passed</span><br><span class="line"> INFO[BL2]: UEFI loaded</span><br><span class="line"> INFO[BL31]: start</span><br><span class="line"> INFO[BL31]: runtime</span><br><span class="line"> INFO[UEFI]: UPVS valid</span><br><span class="line"> INFO[UEFI]: eMMC init</span><br><span class="line"> INFO[UEFI]: eMMC probed</span><br><span class="line"> INFO[UEFI]: PMI: updates started</span><br><span class="line"> INFO[UEFI]: PMI: boot image update</span><br><span class="line"> INFO[UEFI]: PMI: updates completed, status 0</span><br><span class="line"> INFO[UEFI]: PCIe enum start</span><br><span class="line"> INFO[UEFI]: PCIe enum end</span><br><span class="line"> INFO[MISC]: Found bf.cfg</span><br><span class="line"> INFO[MISC]: Ubuntu installation started</span><br><span class="line"> INFO[MISC]: Installing OS image</span><br><span class="line"> INFO[MISC]: Changing the default password <span class="keyword">for</span> user ubuntu</span><br><span class="line"> INFO[MISC]: Installation finished</span><br></pre></td></tr></table></figure><h2 id="4-connect-bluefield-nic-by-console-port-open-another-session"><a href="#4-connect-bluefield-nic-by-console-port-open-another-session" class="headerlink" title="4 connect bluefield nic by console port (open another session)"></a>4 connect bluefield nic by console port (open another session)</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">screen /dev/rshim0/console</span><br></pre></td></tr></table></figure><p>console output:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">write counter to semaphore: Operation not permitted</span><br><span class="line">INFO: Changing the default password <span class="keyword">for</span> user ubuntu</span><br><span class="line">Boot0000* focal</span><br><span class="line">INFO: Installation finished</span><br><span class="line">INFO: Rebooting...</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Mellanox BlueField-2 A1 BL1 V1.1</span><br><span class="line">NOTICE:  BL2R: v2.2(release):3.8.0-25-ge146e67</span><br><span class="line">NOTICE:  BL2R: Built : 15:12:50, Nov 30 2021</span><br><span class="line">NOTICE:  BL2R built <span class="keyword">for</span> hw (ver 1)</span><br><span class="line">NOTICE:  BL2R: Booting BL2</span><br><span class="line">NOTICE:  BL2: v2.2(release):3.8.0-25-ge146e67</span><br><span class="line">NOTICE:  BL2: Built : 15:12:50, Nov 30 2021</span><br><span class="line">NOTICE:  BL2 built <span class="keyword">for</span> hw (ver 1)</span><br><span class="line">NOTICE:  Running as MBF2H332A-AECO system</span><br><span class="line">NOTICE:  No SPD detected on MSS0 DIMM0</span><br><span class="line">NOTICE:  No SPD detected on MSS0 DIMM1</span><br><span class="line">NOTICE:  Finished initializing DDR</span><br><span class="line">NOTICE:  DDR POST passed.</span><br><span class="line">NOTICE:  BL31: v2.2(release):3.8.0-25-ge146e67</span><br><span class="line">NOTICE:  BL31: Built : 15:12:50, Nov 30 2021</span><br><span class="line">NOTICE:  BL31 built <span class="keyword">for</span> hw (ver 1)</span><br><span class="line">UEFI firmware (version BlueField:3.8.0-34-gb1e3ae0 built at 15:15:10 on Nov 30 2021)</span><br><span class="line">Press &lt;ESC&gt; twice to enter UEFI menu</span><br><span class="line">3 seconds remaining</span><br><span class="line">2 seconds remaining</span><br><span class="line">1 seconds remaining</span><br><span class="line">0</span><br><span class="line">[    0.608703] rtc-efi rtc-efi: hctosys: unable to <span class="built_in">read</span> the hardware clock</span><br><span class="line">[   10.715782] mlx5_core 0000:03:00.0: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): <span class="built_in">hash</span> based LAG is not supported by current device</span><br><span class="line">[   12.861453] mlx5_core 0000:03:00.1: mlx5_devlink_eswitch_lag_port_select_mode_set:4632:(pid 887): <span class="built_in">hash</span> based LAG is not supported by current device</span><br><span class="line"></span><br><span class="line">Ubuntu 20.04.3 LTS localhost hvc0</span><br><span class="line"></span><br><span class="line">localhost login: </span><br></pre></td></tr></table></figure><p>The login username is ubuntu. The password is lab123</p><h2 id="5-connect-bluefield-use-by-ssh"><a href="#5-connect-bluefield-use-by-ssh" class="headerlink" title="5 connect bluefield use by ssh"></a>5 connect bluefield use by ssh</h2><h3 id="5-1-config-tmfifo-net0"><a href="#5-1-config-tmfifo-net0" class="headerlink" title="5.1 config tmfifo_net0"></a>5.1 config tmfifo_net0</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ip add add 192.168.100.1/30 dev tmfifo_net0</span><br></pre></td></tr></table></figure><p>check interface</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">root@ubuntu2004-lab1:~# ip address show tmfifo_net0</span><br><span class="line">16: tmfifo_net0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000</span><br><span class="line">    link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff</span><br><span class="line">    inet 192.168.100.1/30 scope global tmfifo_net0</span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line">    inet6 fe80::21a:caff:feff:ff02/64 scope link</span><br><span class="line">       valid_lft forever preferred_lft forever</span><br><span class="line"></span><br><span class="line">root@ubuntu2004-lab1:~# ping 192.168.100.2</span><br><span class="line">PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.</span><br><span class="line">64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=2.46 ms</span><br></pre></td></tr></table></figure><p>connect bluefield by ssh</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">root@ubuntu2004-lab1:~# ssh -l ubuntu 192.168.100.2</span><br><span class="line">The authenticity of host &#x27;192.168.100.2 (192.168.100.2)&#x27; can&#x27;t be established.</span><br><span class="line">ECDSA key fingerprint is SHA256:QqO4i9YSGBI6pp1N8JJKqnTFokEEn1JVh+sdxcYPIb4.</span><br><span class="line">Are you sure you want to continue connecting (yes/no/[fingerprint])? yes</span><br><span class="line">Warning: Permanently added &#x27;192.168.100.2&#x27; (ECDSA) to the list of known hosts.</span><br><span class="line">ubuntu@192.168.100.2&#x27;s password:</span><br><span class="line">Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-1022-bluefield aarch64)</span><br><span class="line"></span><br><span class="line"> * Documentation:  https://help.ubuntu.com</span><br><span class="line"> * Management:     https://landscape.canonical.com</span><br><span class="line"> * Support:        https://ubuntu.com/advantage</span><br><span class="line"></span><br><span class="line">  System information as of Tue Sep  7 18:44:49 UTC 2021</span><br><span class="line"></span><br><span class="line">  System load:  0.11              Processes:                    231</span><br><span class="line">  Usage of /:   5.1% of 57.10GB   Users logged in:              1</span><br><span class="line">  Memory usage: 5%                IPv4 address for tmfifo_net0: 192.168.100.2</span><br><span class="line">  Swap usage:   0%</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">0 updates can be applied immediately.</span><br><span class="line"></span><br><span class="line">Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;bluefield-bfb-install&quot;&gt;&lt;a href=&quot;#bluefield-bfb-install&quot; class=&quot;headerlink&quot; title=&quot;bluefield bfb install&quot;&gt;&lt;/a&gt;bluefield bfb install&lt;/</summary>
      
    
    
    
    <category term="Smart-NIC" scheme="https://nfshanq.github.io/categories/Smart-NIC/"/>
    
    
    <category term="Smart NIC" scheme="https://nfshanq.github.io/tags/Smart-NIC/"/>
    
  </entry>
  
  <entry>
    <title>bluefield nic double arp packets</title>
    <link href="https://nfshanq.github.io/2021/12/19/mellanox/bluefield/bluefield-double-arp/"/>
    <id>https://nfshanq.github.io/2021/12/19/mellanox/bluefield/bluefield-double-arp/</id>
    <published>2021-12-19T03:21:03.000Z</published>
    <updated>2021-12-31T06:24:11.793Z</updated>
    
    <content type="html"><![CDATA[<h1 id="1-Prepere"><a href="#1-Prepere" class="headerlink" title="1. Prepere"></a>1. Prepere</h1><h2 id="1-1"><a href="#1-1" class="headerlink" title="1.1"></a>1.1</h2><p>enable hash mode</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mlxconfig -d /dev/mst/&lt;device-name&gt; s LAG_RESOURCE_ALLOCATION=1</span><br></pre></td></tr></table></figure><p>Add/edit the following field from /etc/mellanox/mlnx-bf.conf as follows:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">LAG_HASH_MODE=<span class="string">&quot;yes&quot;</span></span><br></pre></td></tr></table></figure><h2 id="1-2-Delete-default-ovs-and-sf-interface"><a href="#1-2-Delete-default-ovs-and-sf-interface" class="headerlink" title="1.2 Delete default ovs and sf interface"></a>1.2 Delete default ovs and sf interface</h2><p>delete default ovs</p><p>command:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">ovs-vsctl del-br ovsbr1</span><br><span class="line">ovs-vsctl del-br ovsbr2</span><br></pre></td></tr></table></figure><p>delete sf interface</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">sf_to_delete=$(mlnx-sf -a show | grep <span class="string">&quot;SF Index&quot;</span> | cut -d<span class="string">&quot; &quot;</span> -f3)</span><br><span class="line"><span class="keyword">for</span> sf <span class="keyword">in</span> <span class="variable">$sf_to_delete</span>; <span class="keyword">do</span></span><br><span class="line">mlnx-sf -a delete -i <span class="string">&quot;<span class="variable">$sf</span>&quot;</span>;</span><br><span class="line"><span class="keyword">done</span></span><br></pre></td></tr></table></figure><p>modify the config<br>file: /etc/mellanox/mlnx-sf.conf</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">root@localhost:/home/ubuntu<span class="comment"># cat /etc/mellanox/mlnx-sf.conf</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#/sbin/mlnx-sf --action create --device 0000:03:00.0 --sfnum 0 --hwaddr 02:2a:e6:4d:e1:ad</span></span><br><span class="line"><span class="comment">#/sbin/mlnx-sf --action create --device 0000:03:00.1 --sfnum 0 --hwaddr 02:06:f3:ce:ad:5a</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="1-3-Hide-the-second-PF-on-host"><a href="#1-3-Hide-the-second-PF-on-host" class="headerlink" title="1.3 Hide the second PF on host"></a>1.3 Hide the second PF on host</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Hide the second PF on host</span></span><br><span class="line">pf_hidden=$(mlxconfig -d <span class="string">&quot;<span class="variable">$mst_dev</span>&quot;</span>  q HIDE_PORT2_PF | grep True)</span><br><span class="line"><span class="keyword">if</span> [ -z <span class="string">&quot;<span class="variable">$pf_hidden</span>&quot;</span> ]; <span class="keyword">then</span></span><br><span class="line"><span class="built_in">echo</span> y | mlxconfig -d <span class="string">&quot;<span class="variable">$mst_dev</span>&quot;</span> s HIDE_PORT2_PF=True</span><br><span class="line"><span class="built_in">echo</span> Please power cycle host server to hide the second unused host representative</span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure><p>Reboot the bluefield NIC</p><h1 id="2-config-the-bond-interface"><a href="#2-config-the-bond-interface" class="headerlink" title="2. config the bond interface"></a>2. config the bond interface</h1><h2 id="2-1-command"><a href="#2-1-command" class="headerlink" title="2.1 command:"></a>2.1 command:</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">ip link add bond0 <span class="built_in">type</span> bond</span><br><span class="line">ip link <span class="built_in">set</span> bond0 down</span><br><span class="line">ip link <span class="built_in">set</span> bond0 <span class="built_in">type</span> bond miimon 100 mode 4</span><br><span class="line">ip link <span class="built_in">set</span> p0 down</span><br><span class="line">ip link <span class="built_in">set</span> p1 down</span><br><span class="line">ip link <span class="built_in">set</span> p0 master bond0</span><br><span class="line">ip link <span class="built_in">set</span> p1 master bond0</span><br><span class="line">ip link <span class="built_in">set</span> p0 up</span><br><span class="line">ip link <span class="built_in">set</span> p1 up</span><br><span class="line">ip link <span class="built_in">set</span> bond0 up</span><br><span class="line">ovs-vsctl add-br bf-lag</span><br><span class="line">ovs-vsctl add-port bf-lag bond0</span><br><span class="line">ovs-vsctl add-port bf-lag pf0hpf</span><br><span class="line">ip link <span class="built_in">set</span> bf-lag up</span><br></pre></td></tr></table></figure><h2 id="2-2-netplan-config"><a href="#2-2-netplan-config" class="headerlink" title="2.2 netplan config:"></a>2.2 netplan config:</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">network:</span><br><span class="line">    ethernets:</span><br><span class="line">        oob_net0:</span><br><span class="line">          addresses: [100.64.31.11/24]</span><br><span class="line">          gateway4: 100.64.31.1</span><br><span class="line">          nameservers:</span><br><span class="line">            addresses: [114.114.114.114]</span><br><span class="line">        tmfifo_net0:</span><br><span class="line">            addresses:</span><br><span class="line">            - 192.168.100.2/30</span><br><span class="line">            dhcp4: <span class="literal">false</span></span><br><span class="line">            routes:</span><br><span class="line">            -   metric: 1025</span><br><span class="line">                to: 0.0.0.0/0</span><br><span class="line">                via: 192.168.100.1</span><br><span class="line">        p0:</span><br><span class="line">          dhcp4: <span class="literal">false</span></span><br><span class="line">        p1:</span><br><span class="line">          dhcp4: <span class="literal">false</span></span><br><span class="line">        pf0hpf:</span><br><span class="line">          dhcp4: <span class="literal">false</span></span><br><span class="line">    bonds:</span><br><span class="line">      bond0:</span><br><span class="line">        dhcp4: <span class="literal">false</span></span><br><span class="line">        interfaces:</span><br><span class="line">          - p0</span><br><span class="line">          - p1</span><br><span class="line">        parameters:</span><br><span class="line">          mode: 802.3ad</span><br><span class="line">          lacp-rate: fast</span><br><span class="line">          transmit-hash-policy: layer3+4</span><br><span class="line">          mii-monitor-interval: 1</span><br><span class="line">    <span class="comment">#renderer: NetworkManager</span></span><br><span class="line">    version: 2</span><br></pre></td></tr></table></figure><h2 id="2-3-check-bond-interface-status"><a href="#2-3-check-bond-interface-status" class="headerlink" title="2.3 check bond interface status"></a>2.3 check bond interface status</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">root@localhost:/home/ubuntu<span class="comment"># cat /sys/class/net/bond0/bonding/slaves</span></span><br><span class="line">p0 p1 </span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre></td><td class="code"><pre><span class="line">root@localhost:/home/ubuntu<span class="comment"># cat /proc/net/bonding/bond0</span></span><br><span class="line">Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)</span><br><span class="line"></span><br><span class="line">Bonding Mode: IEEE 802.3ad Dynamic link aggregation</span><br><span class="line">Transmit Hash Policy: layer3+4 (1)</span><br><span class="line">MII Status: up</span><br><span class="line">MII Polling Interval (ms): 1</span><br><span class="line">Up Delay (ms): 0</span><br><span class="line">Down Delay (ms): 0</span><br><span class="line">Peer Notification Delay (ms): 0</span><br><span class="line"></span><br><span class="line">802.3ad info</span><br><span class="line">LACP rate: fast</span><br><span class="line">Min links: 0</span><br><span class="line">Aggregator selection policy (ad_select): stable</span><br><span class="line">System priority: 65535</span><br><span class="line">System MAC address: 7a:3f:79:7b:96:fb</span><br><span class="line">Active Aggregator Info:</span><br><span class="line">Aggregator ID: 1</span><br><span class="line">Number of ports: 2</span><br><span class="line">Actor Key: 15</span><br><span class="line">Partner Key: 32</span><br><span class="line">Partner Mac Address: c0:42:d0:08:9a:80</span><br><span class="line"></span><br><span class="line">Slave Interface: p1</span><br><span class="line">MII Status: up</span><br><span class="line">Speed: 10000 Mbps</span><br><span class="line">Duplex: full</span><br><span class="line">Link Failure Count: 1</span><br><span class="line">Permanent HW addr: 08:c0:eb:2b:18:11</span><br><span class="line">Slave queue ID: 0</span><br><span class="line">Aggregator ID: 1</span><br><span class="line">Actor Churn State: none</span><br><span class="line">Partner Churn State: none</span><br><span class="line">Actor Churned Count: 0</span><br><span class="line">Partner Churned Count: 0</span><br><span class="line">details actor lacp pdu:</span><br><span class="line">    system priority: 65535</span><br><span class="line">    system mac address: 7a:3f:79:7b:96:fb</span><br><span class="line">    port key: 15</span><br><span class="line">    port priority: 255</span><br><span class="line">    port number: 1</span><br><span class="line">    port state: 63</span><br><span class="line">details partner lacp pdu:</span><br><span class="line">    system priority: 127</span><br><span class="line">    system mac address: c0:42:d0:08:9a:80</span><br><span class="line">    oper key: 32</span><br><span class="line">    port priority: 127</span><br><span class="line">    port number: 16</span><br><span class="line">    port state: 63</span><br><span class="line"></span><br><span class="line">Slave Interface: p0</span><br><span class="line">MII Status: up</span><br><span class="line">Speed: 10000 Mbps</span><br><span class="line">Duplex: full</span><br><span class="line">Link Failure Count: 1</span><br><span class="line">Permanent HW addr: 08:c0:eb:2b:18:10</span><br><span class="line">Slave queue ID: 0</span><br><span class="line">Aggregator ID: 1</span><br><span class="line">Actor Churn State: none</span><br><span class="line">Partner Churn State: none</span><br><span class="line">Actor Churned Count: 0</span><br><span class="line">Partner Churned Count: 0</span><br><span class="line">details actor lacp pdu:</span><br><span class="line">    system priority: 65535</span><br><span class="line">    system mac address: 7a:3f:79:7b:96:fb</span><br><span class="line">    port key: 15</span><br><span class="line">    port priority: 255</span><br><span class="line">    port number: 2</span><br><span class="line">    port state: 63</span><br><span class="line">details partner lacp pdu:</span><br><span class="line">    system priority: 127</span><br><span class="line">    system mac address: c0:42:d0:08:9a:80</span><br><span class="line">    oper key: 32</span><br><span class="line">    port priority: 127</span><br><span class="line">    port number: 2</span><br><span class="line">    port state: 63</span><br></pre></td></tr></table></figure><h2 id="2-4-remove-or-add-slave-interface-to-bonding"><a href="#2-4-remove-or-add-slave-interface-to-bonding" class="headerlink" title="2.4 remove or add slave interface to bonding"></a>2.4 remove or add slave interface to bonding</h2><p>remove </p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">&quot;-p0&quot;</span> &gt; /sys/class/net/bond0/bonding/slaves</span><br></pre></td></tr></table></figure><p>add</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">&quot;+p0&quot;</span> &gt; /sys/class/net/bond0/bonding/slaves</span><br></pre></td></tr></table></figure><h1 id="3-use-tc-to-copy-arp-packets"><a href="#3-use-tc-to-copy-arp-packets" class="headerlink" title="3. use tc to copy arp packets"></a>3. use tc to copy arp packets</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">tc qdisc add dev pf0hpf handle ffff: ingress</span><br><span class="line"></span><br><span class="line">tc filter add dev pf0hpf parent ffff: protocol arp prio 1  flower skip_hw dst_mac ff:ff:ff:ff:ff:ff action mirred egress mirror dev p0 pipe action mirred egress mirror dev p1</span><br></pre></td></tr></table></figure><p>or<br>only match arp request packets</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tc filter add dev pf0hpf parent ffff: protocol arp prio 1 flower skip_hw arp_op request action mirred egress mirror dev p0 pipe action mirred egress mirror dev p1</span><br></pre></td></tr></table></figure><p>802.1Q vlan</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">tc filter add dev pf0hpf parent ffff: protocol 802.1Q prio 1 flower skip_hw vlan_ethtype arp action mirred egress mirror dev p0 pipe action mirred egress mirror dev p1</span><br><span class="line"></span><br></pre></td></tr></table></figure><p>ref: <a href="https://man7.org/linux/man-pages/man8/tc-flower.8.html">https://man7.org/linux/man-pages/man8/tc-flower.8.html</a></p><h2 id="3-1-check-tc-filter-rule"><a href="#3-1-check-tc-filter-rule" class="headerlink" title="3.1 check tc filter rule"></a>3.1 check tc filter rule</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br></pre></td><td class="code"><pre><span class="line">root@localhost:/home/ubuntu<span class="comment"># tc -s -p filter ls dev pf0hpf parent ffff:</span></span><br><span class="line">filter ingress protocol arp pref 1 flower chain 0</span><br><span class="line">filter ingress protocol arp pref 1 flower chain 0 handle 0x1</span><br><span class="line">  dst_mac ff:ff:ff:ff:ff:ff</span><br><span class="line">  eth_type arp</span><br><span class="line">  not_in_hw</span><br><span class="line">action order 1: mirred (Egress Mirror to device p0) pipe</span><br><span class="line"> index 3 ref 1 <span class="built_in">bind</span> 1 installed 1514 sec used 0 sec</span><br><span class="line"> Action statistics:</span><br><span class="line">Sent 69644 bytes 1514 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line"></span><br><span class="line">action order 2: mirred (Egress Mirror to device p1) pipe</span><br><span class="line"> index 5 ref 1 <span class="built_in">bind</span> 1 installed 1514 sec used 0 sec</span><br><span class="line"> Action statistics:</span><br><span class="line">Sent 69644 bytes 1514 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line"></span><br><span class="line">filter ingress protocol LLDP pref 2 flower chain 0</span><br><span class="line">filter ingress protocol LLDP pref 2 flower chain 0 handle 0x1</span><br><span class="line">  dst_mac 01:80:c2:00:00:0e</span><br><span class="line">  src_mac 00:50:56:50:86:1d</span><br><span class="line">  eth_type 88cc</span><br><span class="line">  in_hw in_hw_count 1</span><br><span class="line">action order 1: gact action drop</span><br><span class="line"> random <span class="built_in">type</span> none pass val 0</span><br><span class="line"> index 1 ref 1 <span class="built_in">bind</span> 1 installed 9 sec used 9 sec</span><br><span class="line">Action statistics:</span><br><span class="line">Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line">cookie 45b15fb785436a48db7baabfebadfc89</span><br><span class="line">no_percpu</span><br><span class="line"></span><br><span class="line">filter ingress protocol arp pref 5 flower chain 0</span><br><span class="line">filter ingress protocol arp pref 5 flower chain 0 handle 0x1</span><br><span class="line">  dst_mac ff:ff:ff:ff:ff:ff</span><br><span class="line">  src_mac 00:0c:29:72:e7:c4</span><br><span class="line">  eth_type arp</span><br><span class="line">  arp_sip 11.1.230.10</span><br><span class="line">  arp_tip 11.1.230.1</span><br><span class="line">  arp_op request</span><br><span class="line">  not_in_hw</span><br><span class="line">action order 1: skbedit  ptype host pipe</span><br><span class="line"> index 1 ref 1 <span class="built_in">bind</span> 1 installed 7 sec used 7 sec</span><br><span class="line">Action statistics:</span><br><span class="line">Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line"></span><br><span class="line">action order 2: mirred (Ingress Mirror to device bf-lag) pipe</span><br><span class="line"> index 1 ref 1 <span class="built_in">bind</span> 1 installed 7 sec used 7 sec</span><br><span class="line"> Action statistics:</span><br><span class="line">Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line">cookie 688501ce534a508863c24fbf6f995a36</span><br><span class="line">no_percpu</span><br><span class="line"></span><br><span class="line">action order 3: mirred (Egress Redirect to device bond0) stolen</span><br><span class="line"> index 2 ref 1 <span class="built_in">bind</span> 1 installed 7 sec used 7 sec</span><br><span class="line"> Action statistics:</span><br><span class="line">Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)</span><br><span class="line">backlog 0b 0p requeues 0</span><br><span class="line">cookie 688501ce534a508863c24fbf6f995a36</span><br><span class="line">no_percpu</span><br></pre></td></tr></table></figure><h2 id="3-2-delete-tc-filter"><a href="#3-2-delete-tc-filter" class="headerlink" title="3.2 delete tc filter"></a>3.2 delete tc filter</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tc filter del dev pf0hpf parent ffff: protocol arp prio 1</span><br></pre></td></tr></table></figure><h1 id="4-interface-up-down-script-in-networkd-dispatcher"><a href="#4-interface-up-down-script-in-networkd-dispatcher" class="headerlink" title="4 interface up down script in networkd-dispatcher"></a>4 interface up down script in networkd-dispatcher</h1><h2 id="4-1-networkd-dispatcher-service"><a href="#4-1-networkd-dispatcher-service" class="headerlink" title="4.1 networkd-dispatcher service"></a>4.1 networkd-dispatcher service</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">networkd-dispatcher.service - Dispatcher daemon <span class="keyword">for</span> systemd-networkd</span><br><span class="line">     Loaded: loaded (/lib/systemd/system/networkd-dispatcher.service; enabled; vendor preset: enabled)</span><br><span class="line">     Active: active (running) since Thu 2021-12-30 01:59:25 UTC; 1h 46min ago</span><br><span class="line">   Main PID: 1402437 (networkd-dispat)</span><br><span class="line">      Tasks: 1 (<span class="built_in">limit</span>: 19076)</span><br><span class="line">     Memory: 12.4M</span><br><span class="line">     CGroup: /system.slice/networkd-dispatcher.service</span><br><span class="line">             └─1402437 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers --script-dir /etc/networkd-dispatcher</span><br><span class="line"></span><br><span class="line">Dec 30 03:24:14 localhost.localdomain networkd-dispatcher[1449900]: p1 change to enslaved, add tc filter rule</span><br><span class="line">Dec 30 03:31:35 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/no-carrier.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:31:51 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/enslaved.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:31:51 localhost.localdomain networkd-dispatcher[1454738]: p0 change to enslaved, add tc filter rule</span><br><span class="line">Dec 30 03:32:23 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/no-carrier.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:32:32 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/enslaved.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:32:32 localhost.localdomain networkd-dispatcher[1455161]: p0 change to enslaved, add tc filter rule</span><br><span class="line">Dec 30 03:32:41 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/no-carrier.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:32:47 localhost.localdomain networkd-dispatcher[1402437]: INFO:Invoking <span class="string">&#x27;/etc/networkd-dispatcher/enslaved.d/50arp-mirror.sh&#x27;</span> <span class="keyword">for</span> interface p0</span><br><span class="line">Dec 30 03:32:47 localhost.localdomain networkd-dispatcher[1455339]: p0 change to enslaved, add tc filter rule</span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="4-2-modify-networkd-dispatcher-configure"><a href="#4-2-modify-networkd-dispatcher-configure" class="headerlink" title="4.2  modify networkd-dispatcher configure"></a>4.2  modify networkd-dispatcher configure</h2> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">cat /etc/default/networkd-dispatcher</span><br><span class="line"><span class="comment"># Specify command line options here. This config file is used</span></span><br><span class="line"><span class="comment"># by the included systemd service file.</span></span><br><span class="line">networkd_dispatcher_args=<span class="string">&quot;--run-startup-triggers --script-dir /etc/networkd-dispatcher&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># -v for debug information</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><h2 id="4-3-add-script-dir-directory"><a href="#4-3-add-script-dir-directory" class="headerlink" title="4.3 add script-dir directory"></a>4.3 add script-dir directory</h2> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mkdir -p /etc/networkd-dispatcher/&#123;enslaved.d,no-carrier.d&#125;</span><br></pre></td></tr></table></figure><h2 id="4-4-add-script"><a href="#4-4-add-script" class="headerlink" title="4.4 add script"></a>4.4 add script</h2><p>  enslaved.d/50arp-mirror.sh<br> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$IFACE</span> == <span class="string">&quot;p0&quot;</span> || <span class="variable">$IFACE</span> == <span class="string">&quot;p1&quot;</span> ]];  <span class="keyword">then</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">&quot;<span class="variable">$IFACE</span> change to enslaved, add tc filter rule&quot;</span> | tee -a /var/<span class="built_in">log</span>/syslog;</span><br><span class="line">tc filter add dev pf0hpf parent ffff: protocol 802.1Q prio 2 flower skip_hw vlan_ethtype arp action mirred egress mirror dev p0 pipe action mirred egress mirror dev p1;</span><br><span class="line">tc filter add dev pf0hpf parent ffff: protocol arp prio 3 flower skip_hw arp_op request action mirred egress mirror dev p0 pipe action mirred egress mirror dev p1;</span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure><br> no-carrier.d/50arp-mirror.sh</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"> <span class="comment">#!/bin/bash</span></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$IFACE</span> == <span class="string">&quot;p0&quot;</span> || <span class="variable">$IFACE</span> == <span class="string">&quot;p1&quot;</span> ]];  <span class="keyword">then</span></span><br><span class="line">    <span class="built_in">echo</span> <span class="string">&quot;<span class="variable">$IFACE</span> change to no-carrie, delete tc filter rule&quot;</span> | tee -a /var/<span class="built_in">log</span>/syslog;</span><br><span class="line">    tc filter del dev pf0hpf parent ffff: protocol 802.1Q prio 2;</span><br><span class="line">    tc filter del dev pf0hpf parent ffff: protocol arp prio 3;</span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure><p> add script execute </p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">chmod +x enslaved.d/50arp-mirror.sh  </span><br><span class="line">chmod +x no-carrier.d/50arp-mirror.sh</span><br></pre></td></tr></table></figure><p>ref: <a href="https://gitlab.com/craftyguy/networkd-dispatcher/-/tree/master">networkd-dispatcher</a></p><h1 id="5-log"><a href="#5-log" class="headerlink" title="5 log"></a>5 log</h1><p>/var/log/syslog</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">Sep  7 18:52:14 localhost kernel: [  898.282891] tc mirred to Houston: device bf-lag is down</span><br><span class="line">Sep  7 18:52:15 localhost kernel: [  899.283762] tc mirred to Houston: device bf-lag is down</span><br><span class="line">Sep  7 18:52:16 localhost kernel: [  900.284601] tc mirred to Houston: device bf-lag is down</span><br><span class="line">Sep  7 18:52:17 localhost systemd[1]: Starting Cleanup of Temporary Directories...</span><br><span class="line">Sep  7 18:52:17 localhost systemd[1]: systemd-tmpfiles-clean.service: Succeeded.</span><br><span class="line">Sep  7 18:52:17 localhost systemd[1]: Finished Cleanup of Temporary Directories.</span><br><span class="line">Sep  7 18:52:17 localhost kernel: [  901.285391] tc mirred to Houston: device bf-lag is down</span><br><span class="line">Sep  7 18:52:18 localhost kernel: [  902.286219] tc mirred to Houston: device bf-lag is down</span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;1-Prepere&quot;&gt;&lt;a href=&quot;#1-Prepere&quot; class=&quot;headerlink&quot; title=&quot;1. Prepere&quot;&gt;&lt;/a&gt;1. Prepere&lt;/h1&gt;&lt;h2 id=&quot;1-1&quot;&gt;&lt;a href=&quot;#1-1&quot; class=&quot;headerli</summary>
      
    
    
    
    <category term="Smart-NIC" scheme="https://nfshanq.github.io/categories/Smart-NIC/"/>
    
    
    <category term="Smart NIC" scheme="https://nfshanq.github.io/tags/Smart-NIC/"/>
    
  </entry>
  
  <entry>
    <title>rke2-install</title>
    <link href="https://nfshanq.github.io/2021/12/18/kubernetes/rancher/rke2-install/"/>
    <id>https://nfshanq.github.io/2021/12/18/kubernetes/rancher/rke2-install/</id>
    <published>2021-12-18T22:48:47.000Z</published>
    <updated>2021-12-19T09:22:12.216Z</updated>
    
    <content type="html"><![CDATA[<h1 id="1-系统准备"><a href="#1-系统准备" class="headerlink" title="1 系统准备"></a>1 系统准备</h1><h2 id="1-1-DNS-配置"><a href="#1-1-DNS-配置" class="headerlink" title="1.1 DNS 配置"></a>1.1 DNS 配置</h2><p>或者修改/etc/hosts 文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">100.64.20.101 k3s-001</span><br><span class="line">100.64.20.102 k3s-002</span><br><span class="line">100.64.20.103 k3s-003</span><br></pre></td></tr></table></figure><h2 id="1-2-disable-SELINUX"><a href="#1-2-disable-SELINUX" class="headerlink" title="1.2. disable SELINUX"></a>1.2. disable SELINUX</h2><h2 id="1-3-linux-module"><a href="#1-3-linux-module" class="headerlink" title="1.3. linux module"></a>1.3. linux module</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat /etc/sysctl.conf</span></span><br><span class="line">net.bridge.bridge-nf-call-ip6tables = 1</span><br><span class="line">net.bridge.bridge-nf-call-iptables = 1</span><br><span class="line">net.ipv4.ip_forward = 1</span><br><span class="line"><span class="comment"># modprobe br_netfilter</span></span><br><span class="line"><span class="comment"># sysctl -p</span></span><br></pre></td></tr></table></figure><h2 id="1-4-修改Firewall-规则"><a href="#1-4-修改Firewall-规则" class="headerlink" title="1.4. 修改Firewall 规则"></a>1.4. 修改Firewall 规则</h2><p>master 需要open TCP 9345 port</p><h2 id="1-5-networkmanager-的修改"><a href="#1-5-networkmanager-的修改" class="headerlink" title="1.5. networkmanager 的修改"></a>1.5. networkmanager 的修改</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat /etc/NetworkManager/conf.d/rke2-canal.conf</span></span><br><span class="line">[keyfile]</span><br><span class="line">unmanaged-devices=interface-name:cali*;interface-name:flannel*</span><br></pre></td></tr></table></figure><h1 id="2-安装rke2"><a href="#2-安装rke2" class="headerlink" title="2. 安装rke2"></a>2. 安装rke2</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Master node:</span><br><span class="line"><span class="comment"># curl -sfL http://rancher-mirror.rancher.cn/rke2/install.sh | INSTALL_RKE2_MIRROR=cn sh -</span></span><br><span class="line">worker node:</span><br><span class="line"><span class="comment"># curl -sfL http://rancher-mirror.rancher.cn/rke2/install.sh | INSTALL_RKE2_MIRROR=cn INSTALL_RKE2_TYPE=&quot;agent&quot;  sh -</span></span><br></pre></td></tr></table></figure><p>指定一个安装的版本</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl -sfL http://rancher-mirror.rancher.cn/rke2/install.sh | INSTALL_RKE2_VERIOIN= vX.Y.Z- INSTALL_RKE2_MIRROR=cn sh -</span><br></pre></td></tr></table></figure><h2 id="2-1-启动-第一个-rke2-master-节点"><a href="#2-1-启动-第一个-rke2-master-节点" class="headerlink" title="2.1 启动 第一个 rke2 master 节点"></a>2.1 启动 第一个 rke2 master 节点</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># systemctl enable rke2-server</span></span><br><span class="line"><span class="comment"># systemctl start rke2-server</span></span><br></pre></td></tr></table></figure><p>在第一次启动后，在/var/lib/rancher/rke2 目录下会创建 rke2 的环境</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">root@k3s-001:/var/lib/rancher/rke2<span class="comment"># ls -lh</span></span><br><span class="line">total 12K</span><br><span class="line">drwxr-xr-x 7 root root 4.0K Nov 19 08:08 agent</span><br><span class="line">lrwxrwxrwx 1 root root   58 Nov 19 08:03 bin -&gt; /var/lib/rancher/rke2/data/v1.21.6-rke2r1-fd8a733b61b5/bin</span><br><span class="line">drwxr-xr-x 3 root root 4.0K Nov 19 08:03 data</span><br><span class="line">drwx------ 7 root root 4.0K Nov 19 08:07 server</span><br><span class="line">root@k3s-001:/var/lib/rancher/rke2<span class="comment"># ls -lh ./bin/</span></span><br><span class="line">total 269M</span><br><span class="line">-rwxr-xr-x 1 root root  33M Nov 19 08:03 containerd</span><br><span class="line">-rwxr-xr-x 1 root root 6.2M Nov 19 08:03 containerd-shim</span><br><span class="line">-rwxr-xr-x 1 root root  11M Nov 19 08:03 containerd-shim-runc-v1</span><br><span class="line">-rwxr-xr-x 1 root root  11M Nov 19 08:03 containerd-shim-runc-v2</span><br><span class="line">-rwxr-xr-x 1 root root  23M Nov 19 08:03 crictl</span><br><span class="line">-rwxr-xr-x 1 root root  19M Nov 19 08:03 ctr</span><br><span class="line">-rwxr-xr-x 1 root root  47M Nov 19 08:03 kubectl</span><br><span class="line">-rwxr-xr-x 1 root root 112M Nov 19 08:03 kubelet</span><br><span class="line">-rwxr-xr-x 1 root root  11M Nov 19 08:03 runc</span><br><span class="line">-rwxr-xr-x 1 root root 307K Nov 19 08:03 socat</span><br><span class="line">root@k3s-001:/var/lib/rancher/rke2<span class="comment">#</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><p>在 /etc/rancher/rke2 目录下会生成 rke2.yaml 的文件。这是一个 kubectl 的配置文件，用于集群管理。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">root@k3s-001:~<span class="comment"># ls -l /etc/rancher/rke2/</span></span><br><span class="line">total 4</span><br><span class="line">-rw------- 1 root root 2961 Nov 19 08:03 rke2.yaml</span><br><span class="line">root@k3s-001:~<span class="comment">#</span></span><br></pre></td></tr></table></figure><p>通过kubectl 管理k3s 集群</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> KUBECONFIG=/etc/rancher/rke2/rke2.yaml</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:/var/lib/rancher/rke2/bin</span><br></pre></td></tr></table></figure><p>查看k3s cluster(rke2)</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">root@k3s-001:~<span class="comment"># kubectl get pod -A</span></span><br><span class="line">NAMESPACE     NAME                                                   READY   STATUS      RESTARTS   AGE</span><br><span class="line">kube-system   cloud-controller-manager-k3s-001                       1/1     Running     0          19h</span><br><span class="line">kube-system   etcd-k3s-001                                           1/1     Running     0          19h</span><br><span class="line">kube-system   helm-install-rke2-canal-nwqr2                          0/1     Completed   0          19h</span><br><span class="line">kube-system   helm-install-rke2-coredns-86s8w                        0/1     Completed   0          19h</span><br><span class="line">kube-system   helm-install-rke2-ingress-nginx-t2bvr                  0/1     Completed   0          19h</span><br><span class="line">kube-system   helm-install-rke2-metrics-server-2q2f7                 0/1     Completed   0          19h</span><br><span class="line">kube-system   kube-apiserver-k3s-001                                 1/1     Running     0          19h</span><br><span class="line">kube-system   kube-controller-manager-k3s-001                        1/1     Running     0          19h</span><br><span class="line">kube-system   kube-proxy-k3s-001                                     1/1     Running     0          19h</span><br><span class="line">kube-system   kube-scheduler-k3s-001                                 1/1     Running     0          19h</span><br><span class="line">kube-system   rke2-canal-46v6k                                       2/2     Running     0          19h</span><br><span class="line">kube-system   rke2-coredns-rke2-coredns-5c84cbd6bd-tdlcn             1/1     Running     0          19h</span><br><span class="line">kube-system   rke2-coredns-rke2-coredns-autoscaler-78bdc5d6c-gj4w6   1/1     Running     0          19h</span><br><span class="line">kube-system   rke2-ingress-nginx-controller-gxszn                    1/1     Running     0          19h</span><br><span class="line">kube-system   rke2-metrics-server-5df7d77b5b-p2j7t                   1/1     Running     0          19h</span><br><span class="line">root@k3s-001:~<span class="comment"># kubectl get node -A</span></span><br><span class="line">NAME      STATUS   ROLES                       AGE   VERSION</span><br><span class="line">k3s-001   Ready    control-plane,etcd,master   19h   v1.21.6+rke2r1</span><br><span class="line">root@k3s-001:~<span class="comment">#</span></span><br></pre></td></tr></table></figure><p>修改第一个节点的 rke2 配置文件：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat /etc/rancher/rke2/config.yaml</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># token path /var/lib/rancher/rke2/server/node-token</span></span><br><span class="line">token: K10b74ab83885c946d61159a734036a4f964f54acd80a5a74042cf9adff34a983b0::server:82ed82eb9668d57283c1efd84c5c1a05</span><br><span class="line">node-name: <span class="string">&quot;k3s-001&quot;</span></span><br><span class="line"><span class="comment"># node-taint:</span></span><br><span class="line">node-label:</span><br><span class="line">  - <span class="string">&quot;node=Master&quot;</span></span><br><span class="line">  - <span class="string">&quot;k3s-001=Master&quot;</span></span><br><span class="line"> </span><br></pre></td></tr></table></figure><p>重启rke2 server</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># systemctl daemon-reload</span></span><br><span class="line"><span class="comment"># systemctl restart rke2-server</span></span><br></pre></td></tr></table></figure><h2 id="2-2-添加第二个master-节点"><a href="#2-2-添加第二个master-节点" class="headerlink" title="2.2 添加第二个master 节点"></a>2.2 添加第二个master 节点</h2><p>在第二个节点上添加 配置文件(先不要启动第二节点的rke2-server 服务，如果之前启动过，那么会有一些配置文件不一致导致无法加入集群)</p><p>第二个节点的配置文件：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># mkdir -p /etc/rancher/rke2/</span></span><br><span class="line"><span class="comment"># cat /etc/rancher/rke2/config.yaml</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># token path /var/lib/rancher/rke2/server/node-token</span></span><br><span class="line">server: <span class="string">&quot;https://100.64.20.101:9345&quot;</span></span><br><span class="line"><span class="comment"># token path /var/lib/rancher/rke2/server/node-token (from first master node)</span></span><br><span class="line">token: K10b74ab83885c946d61159a734036a4f964f54acd80a5a74042cf9adff34a983b0::server:82ed82eb9668d57283c1efd84c5c1a05</span><br><span class="line"></span><br><span class="line">node-name: <span class="string">&quot;k3s-002&quot;</span></span><br><span class="line"><span class="comment"># node-taint:</span></span><br><span class="line">node-label:</span><br><span class="line">  - <span class="string">&quot;node=Master&quot;</span></span><br><span class="line">  - <span class="string">&quot;k3s-002=Master&quot;</span></span><br></pre></td></tr></table></figure><h2 id="2-3-添加第三个节点作为work-节点"><a href="#2-3-添加第三个节点作为work-节点" class="headerlink" title="2.3 添加第三个节点作为work 节点"></a>2.3 添加第三个节点作为work 节点</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># mkdir -p /etc/rancher/rke2/</span></span><br><span class="line"><span class="comment"># cat /etc/rancher/rke2/config.yaml</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># token path /var/lib/rancher/rke2/server/node-token</span></span><br><span class="line">server: <span class="string">&quot;https://100.64.20.101:9345&quot;</span></span><br><span class="line"><span class="comment"># token path /var/lib/rancher/rke2/server/node-token (from first master node)</span></span><br><span class="line">token: K10b74ab83885c946d61159a734036a4f964f54acd80a5a74042cf9adff34a983b0::server:82ed82eb9668d57283c1efd84c5c1a05</span><br><span class="line">node-name: <span class="string">&quot;k3s-003&quot;</span></span><br><span class="line"><span class="comment"># node-taint:</span></span><br><span class="line">node-label:</span><br><span class="line">  - <span class="string">&quot;node=Worker&quot;</span></span><br><span class="line">  - <span class="string">&quot;k3s-003=Worker&quot;</span></span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">systemctl daemon-reload</span><br><span class="line">systemctl restart rke2-agent</span><br></pre></td></tr></table></figure><h1 id="检查-containerd"><a href="#检查-containerd" class="headerlink" title="检查 containerd"></a>检查 containerd</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#/var/lib/rancher/rke2/bin/crictl --runtime-endpoint=unix:///run/k3s/containerd/containerd.sock ps</span></span><br></pre></td></tr></table></figure><h1 id="配置master-上的管理cli-环境变量"><a href="#配置master-上的管理cli-环境变量" class="headerlink" title="配置master 上的管理cli 环境变量"></a>配置master 上的管理cli 环境变量</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cd $HOME</span></span><br><span class="line">mkdir .kube</span><br><span class="line">ln -s /etc/rancher/rke2/rke2.yaml ~/.kube/config</span><br><span class="line">chmod 600 ~/.kube/config</span><br><span class="line">ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:/var/lib/rancher/rke2/bin</span><br></pre></td></tr></table></figure><h1 id="container-的image-rergistory-TBD"><a href="#container-的image-rergistory-TBD" class="headerlink" title="container 的image rergistory -TBD"></a>container 的image rergistory -TBD</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">cat /etc/rancher/rke2/registries.yaml</span><br><span class="line"></span><br><span class="line">mirrors:</span><br><span class="line">  aliyuns.com:</span><br><span class="line">    endpoint:</span><br><span class="line">      - <span class="string">&quot;&quot;</span></span><br><span class="line">configs:</span><br><span class="line">  </span><br></pre></td></tr></table></figure><p>配置etcd的 snapshots</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">token: K10b74ab83885c946d61159a734036a4f964f54acd80a5a74042cf9adff34a983b0::server:82ed82eb9668d57283c1efd84c5c1a05</span><br><span class="line">node-name: <span class="string">&quot;k3s-001&quot;</span></span><br><span class="line"><span class="comment"># node-taint:</span></span><br><span class="line">node-label:</span><br><span class="line">  - <span class="string">&quot;node=Master&quot;</span></span><br><span class="line">  - <span class="string">&quot;k3s-001=Master&quot;</span></span><br><span class="line"></span><br><span class="line">etcd-snapshot-retention: 2</span><br><span class="line"><span class="comment"># /var/lib/rancher/rke2/server/db/snapshots/</span></span><br><span class="line">etcd-snapshot-schedule-cron: <span class="string">&#x27;*/5 * * * *&#x27;</span></span><br></pre></td></tr></table></figure>]]></content>
    
    
      
      
    <summary type="html">&lt;h1 id=&quot;1-系统准备&quot;&gt;&lt;a href=&quot;#1-系统准备&quot; class=&quot;headerlink&quot; title=&quot;1 系统准备&quot;&gt;&lt;/a&gt;1 系统准备&lt;/h1&gt;&lt;h2 id=&quot;1-1-DNS-配置&quot;&gt;&lt;a href=&quot;#1-1-DNS-配置&quot; class=&quot;headerli</summary>
      
    
    
    
    <category term="Kubernetes" scheme="https://nfshanq.github.io/categories/Kubernetes/"/>
    
    
  </entry>
  
</feed>
