docs/installation-guide.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. SPDX-License-Identifier: CC-BY-4.0
   3 .. Copyright (C) 2019 Wind River Systems, Inc.
   4
   5
   6 INF Installation Guide
   7 ======================
   8
   9 .. contents::
  10    :depth: 3
  11    :local:
  12
  13 Abstract
  14 ********
  15
  16 This document describes how to install O-RAN INF image, example configuration (All-in-one Duplex)
  17 for better real time performance, and example deployment of Kubernetes cluster and plugins.
  18
  19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
  20 and container technology.
  21
  22
  23 Preface
  24 *******
  25
  26 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
  27
  28
  29 Hardware Requirements
  30 *********************
  31
  32 Following minimum hardware requirements must be met for installation of O-RAN INF image with AIO-DX:
  33
  34 +-------------------+----------------------------------------------------------------------------+
  35 | **HW Aspect**     | **Requirement**                                                            |
  36 |                   |                                                                            |
  37 +-------------------+----------------------------------------------------------------------------+
  38 | **# of servers**  | 2                                                                          |
  39 +-------------------+----------------------------------------------------------------------------+
  40 | **CPU**           | Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket          |
  41 |                   | or                                                                         |
  42 |                   | Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) |
  43 +-------------------+----------------------------------------------------------------------------+
  44 | **RAM**           | 32G                                                                        |
  45 |                   |                                                                            |
  46 +-------------------+----------------------------------------------------------------------------+
  47 | **Disk**          | * Disk 1: 500G(It's better to be SSD)                                      |
  48 |                   | * Disk 2: 1 or more 500 GB for Ceph OSD                                    |
  49 +-------------------+----------------------------------------------------------------------------+
  50 | **NICs**          | * OAM: 1x1GE                                                               |
  51 |                   | * Data: 1 or more x 10GE (optional)                                        |
  52 +-------------------+----------------------------------------------------------------------------+
  53 | **BIOS settings** | * Hyper-Threading technology enabled                                       |
  54 |                   | * Virtualization technology enabled                                        |
  55 |                   | * VT for directed I/O enabled                                              |
  56 |                   | * CPU power and performance policy set to performance                      |
  57 |                   | * CPU C state control disabled                                             |
  58 |                   | * Plug & play BMC detection disabled                                       |
  59 +-------------------+----------------------------------------------------------------------------+
  60
  61 Installation example (All-in-one Duplex) on HP ProLiant DL380p Gen8
  62 *******************************************************************
  63
  64 1. Installation for the first server from the O-RAN INF ISO image
  65 -----------------------------------------------------------------
  66
  67 -  Please see the README.md file for how to build the image.
  68 -  The Image is a live ISO image with CLI installer:
  69    inf-image-aio-installer-intel-corei7-64.iso
  70
  71 1.1 Burn the image to the USB device
  72 ++++++++++++++++++++++++++++++++++++
  73
  74 -  Assume the the usb device is /dev/sdX here
  75
  76 ::
  77
  78     $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
  79
  80 1.2 Install the first server (controller-0)
  81 +++++++++++++++++++++++++++++++++++++++++++
  82
  83 -  Reboot the target from the USB device.
  84
  85 -  Select “All-in-one Graphics console” or “All-in-one Serial console
  86    install” and press ENTER
  87
  88 -  Start the auto installation
  89
  90 -  It will reboot aotumatically after installation
  91
  92 2. Configuration and initialize the bootstrap
  93 ---------------------------------------------
  94
  95 2.1 First Login with sysadmin/sysadmin and change password
  96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  97
  98 2.2 Set OAM network before bootstrap
  99 ++++++++++++++++++++++++++++++++++++
 100
 101 ::
 102
 103     export OAM_DEV=eno3
 104     export CONTROLLER0_OAM_CIDR=128.224.210.110/24
 105     export DEFAULT_OAM_GATEWAY=128.224.210.1
 106     sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
 107     sudo ip link set up dev $OAM_DEV
 108     sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
 109
 110 2.3 Login the server through SSH with sysadmin
 111 ++++++++++++++++++++++++++++++++++++++++++++++
 112
 113 2.4 Prepare the localhost.yml for bootstrap
 114 +++++++++++++++++++++++++++++++++++++++++++
 115
 116 ::
 117
 118     cat << EOF > localhost.yml
 119     system_mode: duplex
 120     management_subnet: 192.168.18.0/24
 121     management_start_address: 192.168.18.2
 122     management_end_address: 192.168.18.50
 123     management_gateway_address: 192.168.18.1
 124     external_oam_subnet: 128.224.210.0/24
 125     external_oam_gateway_address: 128.224.210.1
 126     external_oam_floating_address: 128.224.210.110
 127     external_oam_node_0_address: 128.224.210.111
 128     external_oam_node_1_address: 128.224.210.112
 129     EOF
 130
 131 2.5 Run the ansible bootstrap
 132 +++++++++++++++++++++++++++++
 133
 134 ::
 135
 136     ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
 137
 138 After the bootstrap successfully finish, it will show as following:
 139
 140 ::
 141
 142     PLAY RECAP *************************************************************************************************************
 143     localhost                  : ok=257  changed=151  unreachable=0    failed=0    skipped=214  rescued=0    ignored=0
 144
 145 2.6 Congiure controller-0
 146 +++++++++++++++++++++++++
 147
 148 Acquire admin credentials:
 149
 150 ::
 151
 152     controller-0:~$ source /etc/platform/openrc
 153     [sysadmin@controller-0 ~(keystone_admin)]$
 154
 155 Configure the OAM and MGMT interfaces of controller-0 and specify the
 156 attached networks:
 157
 158 ::
 159
 160     OAM_IF=eno3
 161     MGMT_IF=eno1
 162     system host-if-modify controller-0 lo -c none
 163     IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
 164     for UUID in $IFNET_UUIDS; do
 165         system interface-network-remove ${UUID}
 166     done
 167
 168     system host-if-modify controller-0 $OAM_IF -n oam0
 169     system host-if-modify controller-0 $MGMT_IF -n pxeboot0
 170
 171     system host-if-modify controller-0 oam0 -c platform
 172     system interface-network-assign controller-0 oam0 oam
 173
 174     system host-if-modify controller-0 pxeboot0 -c platform
 175     system interface-network-assign controller-0  pxeboot0 pxeboot
 176
 177     system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
 178     system interface-network-assign controller-0 mgmt0 mgmt
 179
 180     system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
 181     system interface-network-assign controller-0 cluster0 cluster-host
 182
 183 Example output:
 184
 185 ::
 186
 187     [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
 188     [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
 189     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
 190     +-----------------+--------------------------------------+
 191     | Property        | Value                                |
 192     +-----------------+--------------------------------------+
 193     | ifname          | lo                                   |
 194     | iftype          | virtual                              |
 195     | ports           | []                                   |
 196     | imac            | 00:00:00:00:00:00                    |
 197     | imtu            | 1500                                 |
 198     | ifclass         | None                                 |
 199     | aemode          | None                                 |
 200     | schedpolicy     | None                                 |
 201     | txhashpolicy    | None                                 |
 202     | uuid            | 08c95952-892b-40b5-b17a-7d2ad46e725c |
 203     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 204     | vlan_id         | None                                 |
 205     | uses            | []                                   |
 206     | used_by         | []                                   |
 207     | created_at      | 2021-11-17T00:30:45.265032+00:00     |
 208     | updated_at      | 2021-11-17T01:03:39.031612+00:00     |
 209     | sriov_numvfs    | 0                                    |
 210     | sriov_vf_driver | None                                 |
 211     +-----------------+--------------------------------------+
 212     [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
 213     [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
 214     >     system interface-network-remove ${UUID}
 215     > done
 216     Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
 217     Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
 218
 219     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
 220     +-----------------+--------------------------------------+
 221     | Property        | Value                                |
 222     +-----------------+--------------------------------------+
 223     | ifname          | oam0                                 |
 224     | iftype          | ethernet                             |
 225     | ports           | [u'eno3']                            |
 226     | imac            | 24:6e:96:5d:0c:b2                    |
 227     | imtu            | 1500                                 |
 228     | ifclass         | None                                 |
 229     | aemode          | None                                 |
 230     | schedpolicy     | None                                 |
 231     | txhashpolicy    | None                                 |
 232     | uuid            | d8a048fa-67ef-43ac-8166-671be93caa30 |
 233     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 234     | vlan_id         | None                                 |
 235     | uses            | []                                   |
 236     | used_by         | []                                   |
 237     | created_at      | 2021-11-17T00:28:32.365863+00:00     |
 238     | updated_at      | 2021-11-17T01:03:45.090904+00:00     |
 239     | sriov_numvfs    | 0                                    |
 240     | sriov_vf_driver | None                                 |
 241     | accelerated     | [True]                               |
 242     +-----------------+--------------------------------------+
 243     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
 244     +-----------------+--------------------------------------+
 245     | Property        | Value                                |
 246     +-----------------+--------------------------------------+
 247     | ifname          | pxeboot0                             |
 248     | iftype          | ethernet                             |
 249     | ports           | [u'eno1']                            |
 250     | imac            | 24:6e:96:5d:0c:92                    |
 251     | imtu            | 1500                                 |
 252     | ifclass         | None                                 |
 253     | aemode          | None                                 |
 254     | schedpolicy     | None                                 |
 255     | txhashpolicy    | None                                 |
 256     | uuid            | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
 257     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 258     | vlan_id         | None                                 |
 259     | uses            | []                                   |
 260     | used_by         | []                                   |
 261     | created_at      | 2021-11-17T00:28:32.612230+00:00     |
 262     | updated_at      | 2021-11-17T01:03:47.341003+00:00     |
 263     | sriov_numvfs    | 0                                    |
 264     | sriov_vf_driver | None                                 |
 265     | accelerated     | [True]                               |
 266     +-----------------+--------------------------------------+
 267     [sysadmin@controller-0 ~(keystone_admin)]$
 268     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
 269     +-----------------+--------------------------------------+
 270     | Property        | Value                                |
 271     +-----------------+--------------------------------------+
 272     | ifname          | oam0                                 |
 273     | iftype          | ethernet                             |
 274     | ports           | [u'eno3']                            |
 275     | imac            | 24:6e:96:5d:0c:b2                    |
 276     | imtu            | 1500                                 |
 277     | ifclass         | platform                             |
 278     | aemode          | None                                 |
 279     | schedpolicy     | None                                 |
 280     | txhashpolicy    | None                                 |
 281     | uuid            | d8a048fa-67ef-43ac-8166-671be93caa30 |
 282     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 283     | vlan_id         | None                                 |
 284     | uses            | []                                   |
 285     | used_by         | []                                   |
 286     | created_at      | 2021-11-17T00:28:32.365863+00:00     |
 287     | updated_at      | 2021-11-17T01:03:49.368879+00:00     |
 288     | sriov_numvfs    | 0                                    |
 289     | sriov_vf_driver | None                                 |
 290     | accelerated     | [True]                               |
 291     +-----------------+--------------------------------------+
 292     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
 293     +--------------+--------------------------------------+
 294     | Property     | Value                                |
 295     +--------------+--------------------------------------+
 296     | hostname     | controller-0                         |
 297     | uuid         | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
 298     | ifname       | oam0                                 |
 299     | network_name | oam                                  |
 300     +--------------+--------------------------------------+
 301     [sysadmin@controller-0 ~(keystone_admin)]$
 302     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
 303     +-----------------+--------------------------------------+
 304     | Property        | Value                                |
 305     +-----------------+--------------------------------------+
 306     | ifname          | pxeboot0                             |
 307     | iftype          | ethernet                             |
 308     | ports           | [u'eno1']                            |
 309     | imac            | 24:6e:96:5d:0c:92                    |
 310     | imtu            | 1500                                 |
 311     | ifclass         | platform                             |
 312     | aemode          | None                                 |
 313     | schedpolicy     | None                                 |
 314     | txhashpolicy    | None                                 |
 315     | uuid            | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
 316     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 317     | vlan_id         | None                                 |
 318     | uses            | []                                   |
 319     | used_by         | []                                   |
 320     | created_at      | 2021-11-17T00:28:32.612230+00:00     |
 321     | updated_at      | 2021-11-17T01:03:53.143795+00:00     |
 322     | sriov_numvfs    | 0                                    |
 323     | sriov_vf_driver | None                                 |
 324     | accelerated     | [True]                               |
 325     +-----------------+--------------------------------------+
 326     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0  pxeboot0 pxeboot
 327     +--------------+--------------------------------------+
 328     | Property     | Value                                |
 329     +--------------+--------------------------------------+
 330     | hostname     | controller-0                         |
 331     | uuid         | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
 332     | ifname       | pxeboot0                             |
 333     | network_name | pxeboot                              |
 334     +--------------+--------------------------------------+
 335     [sysadmin@controller-0 ~(keystone_admin)]$
 336     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
 337     +-----------------+--------------------------------------+
 338     | Property        | Value                                |
 339     +-----------------+--------------------------------------+
 340     | ifname          | mgmt0                                |
 341     | iftype          | vlan                                 |
 342     | ports           | []                                   |
 343     | imac            | 24:6e:96:5d:0c:92                    |
 344     | imtu            | 1500                                 |
 345     | ifclass         | None                                 |
 346     | aemode          | None                                 |
 347     | schedpolicy     | None                                 |
 348     | txhashpolicy    | None                                 |
 349     | uuid            | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
 350     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 351     | vlan_id         | 18                                   |
 352     | uses            | [u'pxeboot0']                        |
 353     | used_by         | []                                   |
 354     | created_at      | 2021-11-17T01:03:57.303000+00:00     |
 355     | updated_at      | None                                 |
 356     | sriov_numvfs    | 0                                    |
 357     | sriov_vf_driver | None                                 |
 358     | accelerated     | [True]                               |
 359     +-----------------+--------------------------------------+
 360     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
 361     +--------------+--------------------------------------+
 362     | Property     | Value                                |
 363     +--------------+--------------------------------------+
 364     | hostname     | controller-0                         |
 365     | uuid         | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
 366     | ifname       | mgmt0                                |
 367     | network_name | mgmt                                 |
 368     +--------------+--------------------------------------+
 369     [sysadmin@controller-0 ~(keystone_admin)]$
 370     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
 371     +-----------------+--------------------------------------+
 372     | Property        | Value                                |
 373     +-----------------+--------------------------------------+
 374     | ifname          | cluster0                             |
 375     | iftype          | vlan                                 |
 376     | ports           | []                                   |
 377     | imac            | 24:6e:96:5d:0c:92                    |
 378     | imtu            | 1500                                 |
 379     | ifclass         | None                                 |
 380     | aemode          | None                                 |
 381     | schedpolicy     | None                                 |
 382     | txhashpolicy    | None                                 |
 383     | uuid            | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
 384     | ihost_uuid      | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
 385     | vlan_id         | 19                                   |
 386     | uses            | [u'pxeboot0']                        |
 387     | used_by         | []                                   |
 388     | created_at      | 2021-11-17T01:04:02.613518+00:00     |
 389     | updated_at      | None                                 |
 390     | sriov_numvfs    | 0                                    |
 391     | sriov_vf_driver | None                                 |
 392     | accelerated     | [True]                               |
 393     +-----------------+--------------------------------------+
 394     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
 395     +--------------+--------------------------------------+
 396     | Property     | Value                                |
 397     +--------------+--------------------------------------+
 398     | hostname     | controller-0                         |
 399     | uuid         | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
 400     | ifname       | cluster0                             |
 401     | network_name | cluster-host                         |
 402     +--------------+--------------------------------------+
 403     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
 404     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 405     | uuid                                 | name     | class    | type     | vlan id | ports     | uses i/f      | used by i/f             | attributes |
 406     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 407     | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0    | platform | vlan     | 18      | []        | [u'pxeboot0'] | []                      | MTU=1500   |
 408     | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None    | [u'eno1'] | []            | [u'mgmt0', u'cluster0'] | MTU=1500   |
 409     | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan     | 19      | []        | [u'pxeboot0'] | []                      | MTU=1500   |
 410     | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0     | platform | ethernet | None    | [u'eno3'] | []            | []                      | MTU=1500   |
 411     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 412
 413 Configure NTP Servers for network time synchronization:
 414
 415 ::
 416
 417     system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
 418
 419 Output
 420
 421 ::
 422
 423     [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
 424     +--------------+--------------------------------------+
 425     | Property     | Value                                |
 426     +--------------+--------------------------------------+
 427     | uuid         | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
 428     | ntpservers   | 0.pool.ntp.org,1.pool.ntp.org        |
 429     | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
 430     | created_at   | 2021-11-17T00:27:23.529571+00:00     |
 431     | updated_at   | None                                 |
 432     +--------------+--------------------------------------+
 433
 434 Add an OSD on controller-0 for Ceph:
 435
 436 ::
 437
 438     system host-disk-list controller-0
 439     system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
 440     system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
 441     system host-stor-list controller-0
 442
 443 Output
 444
 445 ::
 446
 447     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
 448     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 449     | uuid                                 | device_no | device_ | device_ | size_ | available_ | rpm          | serial_id                        | device_path                                     |
 450     |                                      | de        | num     | type    | gib   | gib        |              |                                  |                                                 |
 451     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 452     | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda  | 2048    | HDD     | 893.  | 644.726    | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
 453     |                                      |           |         |         | 75    |            |              |                                  |                                                 |
 454     |                                      |           |         |         |       |            |              |                                  |                                                 |
 455     | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb  | 2064    | HDD     | 1117. | 1117.247   | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
 456     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 457     |                                      |           |         |         |       |            |              |                                  |                                                 |
 458     | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc  | 2080    | HDD     | 1117. | 1117.247   | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
 459     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 460     |                                      |           |         |         |       |            |              |                                  |                                                 |
 461     | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd  | 2096    | HDD     | 1117. | 0.0        | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
 462     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 463     |                                      |           |         |         |       |            |              |                                  |                                                 |
 464     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 465     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
 466     +------------------+-------------------------------------------------------+
 467     | Property         | Value                                                 |
 468     +------------------+-------------------------------------------------------+
 469     | osdid            | 0                                                     |
 470     | function         | osd                                                   |
 471     | state            | configuring-on-unlock                                 |
 472     | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c                  |
 473     | journal_size_gib | 1024                                                  |
 474     | journal_path     | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
 475     | journal_node     | /dev/sdb2                                             |
 476     | uuid             | 0816c72f-a4f0-49ea-9a95-0f02c880717c                  |
 477     | ihost_uuid       | 16afe3a2-ba50-46b8-9fd7-09010059e8b9                  |
 478     | idisk_uuid       | 61b6f262-a51f-4310-aeac-373b1c1bbbc2                  |
 479     | tier_uuid        | 3af8c893-9dd4-40af-afc6-30bb79048448                  |
 480     | tier_name        | storage                                               |
 481     | created_at       | 2021-11-17T01:05:04.063823+00:00                      |
 482     | updated_at       | None                                                  |
 483     +------------------+-------------------------------------------------------+
 484     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
 485     +------------------+-------------------------------------------------------+
 486     | Property         | Value                                                 |
 487     +------------------+-------------------------------------------------------+
 488     | osdid            | 1                                                     |
 489     | function         | osd                                                   |
 490     | state            | configuring-on-unlock                                 |
 491     | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5                  |
 492     | journal_size_gib | 1024                                                  |
 493     | journal_path     | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
 494     | journal_node     | /dev/sdc2                                             |
 495     | uuid             | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5                  |
 496     | ihost_uuid       | 16afe3a2-ba50-46b8-9fd7-09010059e8b9                  |
 497     | idisk_uuid       | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79                  |
 498     | tier_uuid        | 3af8c893-9dd4-40af-afc6-30bb79048448                  |
 499     | tier_name        | storage                                               |
 500     | created_at       | 2021-11-17T01:05:06.939798+00:00                      |
 501     | updated_at       | None                                                  |
 502     +------------------+-------------------------------------------------------+
 503     [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
 504     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 505     | uuid                                 | function | osdid | state                 | idisk_uuid                           | journal_path                                          | journal_node | journal_size_gib | tier_name |
 506     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 507     | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd      | 0     | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2    | 1                | storage   |
 508     | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd      | 1     | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2    | 1                | storage   |
 509     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 510
 511 2.7 Unlock controller-0
 512 +++++++++++++++++++++++
 513
 514 ::
 515
 516     system host-unlock controller-0
 517
 518 Output:
 519
 520 ::
 521
 522     [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
 523     +-----------------------+-------------------------------------------------+
 524     | Property              | Value                                           |
 525     +-----------------------+-------------------------------------------------+
 526     | action                | none                                            |
 527     | administrative        | locked                                          |
 528     | availability          | online                                          |
 529     | bm_ip                 | None                                            |
 530     | bm_type               | none                                            |
 531     | bm_username           | None                                            |
 532     | boot_device           | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
 533     | capabilities          | {u'stor_function': u'monitor'}                  |
 534     | clock_synchronization | ntp                                             |
 535     | config_applied        | 6aa15fb4-8cb3-494e-b94e-95f85b560f22            |
 536     | config_status         | None                                            |
 537     | config_target         | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82            |
 538     | console               | ttyS0,115200                                    |
 539     | created_at            | 2021-11-17T00:28:01.983673+00:00                |
 540     | hostname              | controller-0                                    |
 541     | id                    | 1                                               |
 542     | install_output        | graphical                                       |
 543     | install_state         | None                                            |
 544     | install_state_info    | None                                            |
 545     | inv_state             | inventoried                                     |
 546     | invprovision          | provisioning                                    |
 547     | location              | {}                                              |
 548     | mgmt_ip               | 192.168.18.3                                    |
 549     | mgmt_mac              | 24:6e:96:5d:0c:92                               |
 550     | operational           | disabled                                        |
 551     | personality           | controller                                      |
 552     | reserved              | False                                           |
 553     | rootfs_device         | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
 554     | serialid              | None                                            |
 555     | software_load         | 21.05                                           |
 556     | subfunction_avail     | online                                          |
 557     | subfunction_oper      | disabled                                        |
 558     | subfunctions          | controller,worker,lowlatency                    |
 559     | task                  | Unlocking                                       |
 560     | tboot                 | false                                           |
 561     | ttys_dcd              | None                                            |
 562     | updated_at            | 2021-11-17T01:05:07.015414+00:00                |
 563     | uptime                | 3496                                            |
 564     | uuid                  | 16afe3a2-ba50-46b8-9fd7-09010059e8b9            |
 565     | vim_progress_status   | None                                            |
 566     +-----------------------+-------------------------------------------------+
 567
 568 Controller-0 will reboot to apply configuration changes and come into
 569 service. This can take 5-10 minutes, depending on the performance of the
 570 host machine.
 571
 572 Once the controller comes back up, check the status of controller-0
 573
 574 ::
 575
 576     controller-0:~$ source /etc/platform/openrc
 577     [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
 578     +----+--------------+-------------+----------------+-------------+--------------+
 579     | id | hostname     | personality | administrative | operational | availability |
 580     +----+--------------+-------------+----------------+-------------+--------------+
 581     | 1  | controller-0 | controller  | unlocked       | enabled     | available    |
 582     +----+--------------+-------------+----------------+-------------+--------------+
 583
 584 3. Installation for the second server (controller-1)
 585 ----------------------------------------------------
 586
 587 3.1 Power on the controller-1 server and force it to network boot
 588 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 589
 590 3.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node
 591 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 592
 593 3.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)
 594 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 595
 596 ::
 597
 598     [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
 599     +----+--------------+-------------+----------------+-------------+--------------+
 600     | id | hostname     | personality | administrative | operational | availability |
 601     +----+--------------+-------------+----------------+-------------+--------------+
 602     | 1  | controller-0 | controller  | unlocked       | enabled     | degraded     |
 603     | 2  | None         | None        | locked         | disabled    | offline      |
 604     +----+--------------+-------------+----------------+-------------+--------------+
 605
 606 3.4 Using the host id, set the personality of this host to 'controller':
 607 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 608
 609 ::
 610
 611     [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
 612     +-----------------------+--------------------------------------+
 613     | Property              | Value                                |
 614     +-----------------------+--------------------------------------+
 615     | action                | none                                 |
 616     | administrative        | locked                               |
 617     | availability          | offline                              |
 618     | bm_ip                 | None                                 |
 619     | bm_type               | None                                 |
 620     | bm_username           | None                                 |
 621     | boot_device           | /dev/sda                             |
 622     | capabilities          | {}                                   |
 623     | clock_synchronization | ntp                                  |
 624     | config_applied        | None                                 |
 625     | config_status         | None                                 |
 626     | config_target         | None                                 |
 627     | console               | ttyS0,115200                         |
 628     | created_at            | 2021-11-17T10:17:44.387813+00:00     |
 629     | hostname              | controller-1                         |
 630     | id                    | 2                                    |
 631     | install_output        | text                                 |
 632     | install_state         | None                                 |
 633     | install_state_info    | None                                 |
 634     | inv_state             | None                                 |
 635     | invprovision          | None                                 |
 636     | location              | {}                                   |
 637     | mgmt_ip               | 192.168.18.4                         |
 638     | mgmt_mac              | 24:6e:96:5d:38:ee                    |
 639     | operational           | disabled                             |
 640     | personality           | controller                           |
 641     | reserved              | False                                |
 642     | rootfs_device         | /dev/sda                             |
 643     | serialid              | None                                 |
 644     | software_load         | 21.05                                |
 645     | subfunction_avail     | not-installed                        |
 646     | subfunction_oper      | disabled                             |
 647     | subfunctions          | controller,worker,lowlatency         |
 648     | task                  | None                                 |
 649     | tboot                 | false                                |
 650     | ttys_dcd              | None                                 |
 651     | updated_at            | None                                 |
 652     | uptime                | 0                                    |
 653     | uuid                  | f069381d-9743-49cc-bf8b-eb4bd3972203 |
 654     | vim_progress_status   | None                                 |
 655     +-----------------------+--------------------------------------+
 656
 657 3.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
 658 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 659
 660 This can take 5-10 minutes, depending on the performance of the host
 661 machine.
 662
 663 ::
 664
 665     [root@controller-0 hieradata(keystone_admin)]$ system host-list
 666     +----+--------------+-------------+----------------+-------------+--------------+
 667     | id | hostname     | personality | administrative | operational | availability |
 668     +----+--------------+-------------+----------------+-------------+--------------+
 669     | 1  | controller-0 | controller  | unlocked       | enabled     | available    |
 670     | 2  | controller-1 | controller  | locked         | disabled    | online       |
 671     +----+--------------+-------------+----------------+-------------+--------------+
 672
 673 3.6 Configure controller-1
 674 ++++++++++++++++++++++++++
 675
 676 ::
 677
 678     OAM_IF=eno3
 679     MGMT_IF=eno1
 680     system host-if-modify controller-1 $OAM_IF -n oam0
 681     system host-if-modify controller-1 oam0 -c platform
 682     system interface-network-assign controller-1 oam0 oam
 683
 684     system host-if-add         -V 19 controller-1 cluster0 vlan pxeboot0
 685     system interface-network-assign controller-1 cluster0 cluster-host
 686
 687     system host-if-list controller-1
 688
 689     system host-disk-list controller-1
 690     system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
 691     system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
 692     system host-stor-list controller-1
 693
 694 Output:
 695
 696 ::
 697
 698     [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
 699     [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
 700     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
 701     +-----------------+--------------------------------------+
 702     | Property        | Value                                |
 703     +-----------------+--------------------------------------+
 704     | ifname          | oam0                                 |
 705     | iftype          | ethernet                             |
 706     | ports           | [u'eno3']                            |
 707     | imac            | 24:6e:96:5d:39:0e                    |
 708     | imtu            | 1500                                 |
 709     | ifclass         | None                                 |
 710     | aemode          | None                                 |
 711     | schedpolicy     | None                                 |
 712     | txhashpolicy    | None                                 |
 713     | uuid            | c2473511-d0d6-445d-9739-4d43dc029de9 |
 714     | ihost_uuid      | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
 715     | vlan_id         | None                                 |
 716     | uses            | []                                   |
 717     | used_by         | []                                   |
 718     | created_at      | 2021-11-22T14:01:32.365863+00:00     |
 719     | updated_at      | 2021-11-22T15:04:45.090904+00:00     |
 720     | sriov_numvfs    | 0                                    |
 721     | sriov_vf_driver | None                                 |
 722     | accelerated     | [True]                               |
 723     +-----------------+--------------------------------------+
 724     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
 725     +-----------------+--------------------------------------+
 726     | Property        | Value                                |
 727     +-----------------+--------------------------------------+
 728     | ifname          | oam0                                 |
 729     | iftype          | ethernet                             |
 730     | ports           | [u'eno3']                            |
 731     | imac            | 24:6e:96:5d:39:0e                    |
 732     | imtu            | 1500                                 |
 733     | ifclass         | platform                             |
 734     | aemode          | None                                 |
 735     | schedpolicy     | None                                 |
 736     | txhashpolicy    | None                                 |
 737     | uuid            | c2473511-d0d6-445d-9739-4d43dc029de9 |
 738     | ihost_uuid      | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
 739     | vlan_id         | None                                 |
 740     | uses            | []                                   |
 741     | used_by         | []                                   |
 742     | created_at      | 2021-11-22T14:05:16.052229+00:00     |
 743     | updated_at      | 2021-11-22T15:08:35.324634+00:00     |
 744     | sriov_numvfs    | 0                                    |
 745     | sriov_vf_driver | None                                 |
 746     | accelerated     | [True]                               |
 747     +-----------------+--------------------------------------+
 748     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
 749     +--------------+--------------------------------------+
 750     | Property     | Value                                |
 751     +--------------+--------------------------------------+
 752     | hostname     | controller-1                         |
 753     | uuid         | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
 754     | ifname       | oam0                                 |
 755     | network_name | oam                                  |
 756     +--------------+--------------------------------------+
 757     [sysadmin@controller-0 ~(keystone_admin)]$
 758
 759     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
 760     +-----------------+--------------------------------------+
 761     | Property        | Value                                |
 762     +-----------------+--------------------------------------+
 763     | ifname          | cluster0                             |
 764     | iftype          | vlan                                 |
 765     | ports           | []                                   |
 766     | imac            | 24:6e:96:5d:38:ee                    |
 767     | imtu            | 1500                                 |
 768     | ifclass         | None                                 |
 769     | aemode          | None                                 |
 770     | schedpolicy     | None                                 |
 771     | txhashpolicy    | None                                 |
 772     | uuid            | b6783682-b2aa-4135-90d2-676e1db41ae8 |
 773     | ihost_uuid      | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
 774     | vlan_id         | 19                                   |
 775     | uses            | [u'pxeboot0']                        |
 776     | used_by         | []                                   |
 777     | created_at      | 2021-11-22T15:08:43.932209+00:00     |
 778     | updated_at      | None                                 |
 779     | sriov_numvfs    | 0                                    |
 780     | sriov_vf_driver | None                                 |
 781     | accelerated     | [True]                               |
 782     +-----------------+--------------------------------------+
 783
 784     [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
 785     +--------------+--------------------------------------+
 786     | Property     | Value                                |
 787     +--------------+--------------------------------------+
 788     | hostname     | controller-1                         |
 789     | uuid         | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
 790     | ifname       | cluster0                             |
 791     | network_name | cluster-host                         |
 792     +--------------+--------------------------------------+
 793     [sysadmin@controller-0 ~(keystone_admin)]$
 794     [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
 795     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 796     | uuid                                 | name     | class    | type     | vlan id | ports     | uses i/f      | used by i/f             | attributes |
 797     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 798     | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan     | 19      | []        | [u'pxeboot0'] | []                      | MTU=1500   |
 799     | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None    | [u'eno1'] | []            | [u'mgmt0', u'cluster0'] | MTU=1500   |
 800     | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0    | platform | vlan     | 18      | []        | [u'pxeboot0'] | []                      | MTU=1500   |
 801     | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0     | platform | ethernet | None    | [u'eno3'] | []            | []                      | MTU=1500   |
 802     +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
 803     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
 804     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 805     | uuid                                 | device_no | device_ | device_ | size_ | available_ | rpm          | serial_id                        | device_path                                     |
 806     |                                      | de        | num     | type    | gib   | gib        |              |                                  |                                                 |
 807     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 808     | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda  | 2048    | HDD     | 893.  | 644.726    | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
 809     |                                      |           |         |         | 75    |            |              |                                  |                                                 |
 810     |                                      |           |         |         |       |            |              |                                  |                                                 |
 811     | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb  | 2064    | HDD     | 1117. | 1117.247   | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
 812     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 813     |                                      |           |         |         |       |            |              |                                  |                                                 |
 814     | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc  | 2080    | HDD     | 1117. | 1117.247   | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
 815     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 816     |                                      |           |         |         |       |            |              |                                  |                                                 |
 817     | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd  | 2096    | HDD     | 1117. | 1117.247   | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
 818     |                                      |           |         |         | 25    |            |              |                                  |                                                 |
 819     |                                      |           |         |         |       |            |              |                                  |                                                 |
 820     | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde  | 2112    | SSD     | 223.  | 223.568    | N/A          | PHDW730104QM240E                 | /dev/disk/by-path/pci-0000:d8:00.0-ata-1        |
 821     |                                      |           |         |         | 57    |            |              |                                  |                                                 |
 822     |                                      |           |         |         |       |            |              |                                  |                                                 |
 823     | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf  | 2128    | SSD     | 223.  | 223.568    | N/A          | PHDW730104LL240E                 | /dev/disk/by-path/pci-0000:d8:00.0-ata-2        |
 824     |                                      |           |         |         | 57    |            |              |                                  |                                                 |
 825     |                                      |           |         |         |       |            |              |                                  |                                                 |
 826     +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
 827     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
 828     +------------------+-------------------------------------------------------+
 829     | Property         | Value                                                 |
 830     +------------------+-------------------------------------------------------+
 831     | osdid            | 2                                                     |
 832     | function         | osd                                                   |
 833     | state            | configuring-on-unlock                                 |
 834     | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7                  |
 835     | journal_size_gib | 1024                                                  |
 836     | journal_path     | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
 837     | journal_node     | /dev/sdb2                                             |
 838     | uuid             | 54a218d8-0466-4366-9ef0-3ec5a952fde7                  |
 839     | ihost_uuid       | 63c930c7-2195-4d5a-870c-be610fd6b4fc                  |
 840     | idisk_uuid       | 1a3f0a36-5961-42e5-a271-e71db1c25d42                  |
 841     | tier_uuid        | 06b4740e-29db-4896-9600-03ee40fe0d6c                  |
 842     | tier_name        | storage                                               |
 843     | created_at       | 2021-11-22T15:11:55.641193+00:00                      |
 844     | updated_at       | None                                                  |
 845     +------------------+-------------------------------------------------------+
 846
 847     [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
 848     +------------------+-------------------------------------------------------+
 849     | Property         | Value                                                 |
 850     +------------------+-------------------------------------------------------+
 851     | osdid            | 3                                                     |
 852     | function         | osd                                                   |
 853     | state            | configuring-on-unlock                                 |
 854     | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c                  |
 855     | journal_size_gib | 1024                                                  |
 856     | journal_path     | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
 857     | journal_node     | /dev/sdc2                                             |
 858     | uuid             | 5be88c7a-3a94-4b97-9da5-b247bb89406c                  |
 859     | ihost_uuid       | 63c930c7-2195-4d5a-870c-be610fd6b4fc                  |
 860     | idisk_uuid       | eddd732f-2cea-49b3-86db-b722c0b1a1ae                  |
 861     | tier_uuid        | 06b4740e-29db-4896-9600-03ee40fe0d6c                  |
 862     | tier_name        | storage                                               |
 863     | created_at       | 2021-11-22T15:12:04.274839+00:00                      |
 864     | updated_at       | None                                                  |
 865     +------------------+-------------------------------------------------------+
 866
 867     [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
 868     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 869     | uuid                                 | function | osdid | state                 | idisk_uuid                           | journal_path                                          | journal_node | journal_size_gib | tier_name |
 870     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 871     | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd      | 2     | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2    | 1                | storage   |
 872     | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd      | 3     | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2    | 1                | storage   |
 873     +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
 874
 875 3.7 Unlock controller-1
 876 +++++++++++++++++++++++
 877
 878 Unlock controller-1 in order to bring it into service:
 879
 880 ::
 881
 882     [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
 883     +-----------------------+--------------------------------------+
 884     | Property              | Value                                |
 885     +-----------------------+--------------------------------------+
 886     | action                | none                                 |
 887     | administrative        | locked                               |
 888     | availability          | online                               |
 889     | bm_ip                 | None                                 |
 890     | bm_type               | None                                 |
 891     | bm_username           | None                                 |
 892     | boot_device           | /dev/sda                             |
 893     | capabilities          | {u'stor_function': u'monitor'}       |
 894     | clock_synchronization | ntp                                  |
 895     | config_applied        | None                                 |
 896     | config_status         | Config out-of-date                   |
 897     | config_target         | 9747e0ce-2319-409d-b75c-2475bc5065ac |
 898     | console               | ttyS0,115200                         |
 899     | created_at            | 2021-11-22T12:58:11.630526+00:00     |
 900     | hostname              | controller-1                         |
 901     | id                    | 3                                    |
 902     | install_output        | text                                 |
 903     | install_state         | None                                 |
 904     | install_state_info    | None                                 |
 905     | inv_state             | inventoried                          |
 906     | invprovision          | unprovisioned                        |
 907     | location              | {}                                   |
 908     | mgmt_ip               | 192.168.18.4                         |
 909     | mgmt_mac              | 24:6e:96:5d:38:ee                    |
 910     | operational           | disabled                             |
 911     | personality           | controller                           |
 912     | reserved              | False                                |
 913     | rootfs_device         | /dev/sda                             |
 914     | serialid              | None                                 |
 915     | software_load         | 21.05                                |
 916     | subfunction_avail     | online                               |
 917     | subfunction_oper      | disabled                             |
 918     | subfunctions          | controller,worker,lowlatency         |
 919     | task                  | Unlocking                            |
 920     | tboot                 | false                                |
 921     | ttys_dcd              | None                                 |
 922     | updated_at            | 2021-11-22T15:13:09.716324+00:00     |
 923     | uptime                | 752                                  |
 924     | uuid                  | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
 925     | vim_progress_status   | None                                 |
 926     +-----------------------+--------------------------------------+
 927
 928 Controller-1 will reboot in order to apply configuration changes and
 929 come into service. This can take 5-10 minutes, depending on the
 930 performance of the host machine.
 931
 932 ::
 933
 934     [root@controller-0 hieradata(keystone_admin)]$ system host-list
 935     +----+--------------+-------------+----------------+-------------+--------------+
 936     | id | hostname     | personality | administrative | operational | availability |
 937     +----+--------------+-------------+----------------+-------------+--------------+
 938     | 1  | controller-0 | controller  | unlocked       | enabled     | available    |
 939     | 2  | controller-1 | controller  | unlocked       | enabled     | available    |
 940     +----+--------------+-------------+----------------+-------------+--------------+
 941
 942     [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
 943     +-----------------------+-----------------------------------------------------------------------+
 944     | Property              | Value                                                                 |
 945     +-----------------------+-----------------------------------------------------------------------+
 946     | action                | none                                                                  |
 947     | administrative        | unlocked                                                              |
 948     | availability          | available                                                             |
 949     | bm_ip                 | None                                                                  |
 950     | bm_type               | None                                                                  |
 951     | bm_username           | None                                                                  |
 952     | boot_device           | /dev/sda                                                              |
 953     | capabilities          | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
 954     | clock_synchronization | ntp                                                                   |
 955     | config_applied        | 9747e0ce-2319-409d-b75c-2475bc5065ac                                  |
 956     | config_status         | None                                                                  |
 957     | config_target         | 9747e0ce-2319-409d-b75c-2475bc5065ac                                  |
 958     | console               | ttyS0,115200                                                          |
 959     | created_at            | 2021-11-22T12:58:11.630526+00:00                                      |
 960     | hostname              | controller-1                                                          |
 961     | id                    | 2                                                                     |
 962     | install_output        | text                                                                  |
 963     | install_state         | None                                                                  |
 964     | install_state_info    | None                                                                  |
 965     | inv_state             | inventoried                                                           |
 966     | invprovision          | provisioned                                                           |
 967     | location              | {}                                                                    |
 968     | mgmt_ip               | 192.168.18.4                                                          |
 969     | mgmt_mac              | 24:6e:96:5d:38:ee                                                     |
 970     | operational           | enabled                                                               |
 971     | personality           | controller                                                            |
 972     | reserved              | False                                                                 |
 973     | rootfs_device         | /dev/sda                                                              |
 974     | serialid              | None                                                                  |
 975     | software_load         | 21.05                                                                 |
 976     | subfunction_avail     | available                                                             |
 977     | subfunction_oper      | enabled                                                               |
 978     | subfunctions          | controller,worker,lowlatency                                          |
 979     | task                  |                                                                       |
 980     | tboot                 | false                                                                 |
 981     | ttys_dcd              | None                                                                  |
 982     | updated_at            | 2021-11-22T23:59:07.787759+00:00                                      |
 983     | uptime                | 31008                                                                 |
 984     | uuid                  | 63c930c7-2195-4d5a-870c-be610fd6b4fc                                  |
 985     | vim_progress_status   | services-enabled                                                      |
 986     +-----------------------+-----------------------------------------------------------------------+
 987
 988
 989 4. Simple use case for sriov
 990 ----------------------------
 991
 992 4.1 After controller-0 is rebooted and up running, download the DPDK
 993 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 994
 995 ::
 996
 997   [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
 998   [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
 999   Password:
1000   --2021-06-04 02:35:30--  https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
1001   Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
1002   Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
1003
1004   HTTP request sent, awaiting response... 200 OK
1005   Length: 10251680 (9.8M) [application/octet-stream]
1006   Saving to: ‘dpdk-17.11.10.tar.xz’
1007
1008   dpdk-17.11.10.tar.xz                        100%
1009   [========================================================================================>]   9.78M
1010   1.48MB/s    in 6.8s
1011
1012   2021-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
1013
1014   sudo tar xvf dpdk-17.11.10.tar.xz
1015
1016   sudo ln -s dpdk-stable-17.11.10 dpdk-stable
1017
1018 4.2 Prepare the yaml file for the network assignment container
1019 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1020
1021 The following the exmaple of the yaml file:
1022
1023 ::
1024
1025   [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
1026   > apiVersion: "k8s.cni.cncf.io/v1"
1027   > kind: NetworkAttachmentDefinition
1028   > metadata:
1029   >   name: sriov-data-dpdk-0
1030   >   annotations:
1031   >     k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
1032   > spec:
1033   >   config: '{
1034   >   "type": "sriov",
1035   >   "name": "sriov-data-dpdk-0"
1036   > }'
1037   >
1038   > ---
1039   > apiVersion: "k8s.cni.cncf.io/v1"
1040   > kind: NetworkAttachmentDefinition
1041   > metadata:
1042   >   name: sriov-data-dpdk-1
1043   >   annotations:
1044   >     k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
1045   > spec:
1046   >   config: '{
1047   >   "type": "sriov",
1048   >   "name": "sriov-data-dpdk-1"
1049   > }'
1050   > EOF
1051
1052 4.3 Run the network assignent container for the 2 VFs
1053 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1054
1055 ::
1056
1057   [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
1058   networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
1059   networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
1060
1061 4.4 Prepare the VF container yaml file
1062 ++++++++++++++++++++++++++++++++++++++
1063
1064 ::
1065
1066   [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
1067   > apiVersion: v1
1068   > kind: Pod
1069   metadata:
1070   > metadata:
1071   >   name: pod-with-dpdk-vfs-0
1072   >   annotations:
1073   >     k8s.v1.cni.cncf.io/networks: '[
1074   >             { "name": "sriov-data-dpdk-0" },
1075               { "name": "sriov-data-dpdk-1" }
1076   >             { "name": "sriov-data-dpdk-1" }
1077   >     ]'
1078   > spec:
1079   >   restartPolicy: Never
1080   >   containers:
1081   >   - name: pod-with-dpdk-vfs-0
1082   >     image: wrsnfv/ubuntu-dpdk-build:v0.3
1083   >     env:
1084   >     - name: RTE_SDK
1085   >       value: "/usr/src/dpdk"
1086   >     command:
1087   >     - sleep
1088   >     - infinity
1089   >     stdin: true
1090   >     tty: true
1091   >     securityContext:
1092   >       privileged: true
1093   >       capabilities:
1094   >         add:
1095   >         - ALL
1096   >     resources:
1097   >       requests:
1098   >         cpu: 4
1099   >         memory: 4Gi
1100   >         intel.com/pci_sriov_net_physnet0: '1'
1101   >         intel.com/pci_sriov_net_physnet1: '1'
1102   >       limits:
1103   >         cpu: 4
1104   >         hugepages-1Gi: 2Gi
1105   >         memory: 4Gi
1106   >         intel.com/pci_sriov_net_physnet0: '1'
1107   >         intel.com/pci_sriov_net_physnet1: '1'
1108   >     volumeMounts:
1109   >     - mountPath: /mnt/huge-1048576kB
1110   >       name: hugepage
1111   >     - name: dpdk-volume
1112   >       mountPath: /usr/src/dpdk
1113   >     - name: lib-volume
1114   >       mountPath: /lib/modules
1115   >     - name: src-volume
1116   >       mountPath: /usr/src/
1117   >   volumes:
1118   >   - name: hugepage
1119   >     emptyDir:
1120   >       medium: HugePages
1121   >   - name: dpdk-volume
1122   >     hostPath:
1123   >       path: /opt/dpdk-stable/
1124   >   - name: lib-volume
1125   >     hostPath:
1126   >       path: /lib/modules
1127   >   - name: src-volume
1128   >     hostPath:
1129   >       path: /usr/src/
1130   > EOF
1131
1132 4.5 Run the VF container
1133 ++++++++++++++++++++++++
1134
1135 Start the VF container.
1136
1137 ::
1138
1139   [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
1140   pod/pod-with-dpdk-vfs-0 created
1141
1142   [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
1143   NAME                  READY   STATUS    RESTARTS   AGE
1144   pod-with-dpdk-vfs-0   1/1     Running   0          6m40s
1145
1146 Login the VF container
1147
1148 ::
1149
1150   kubectl exec -it pod-with-dpdk-vfs-0 -- bash
1151
1152 Build the DPDK
1153
1154 ::
1155
1156   cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
1157
1158   root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
1159     HOSTCC  scripts/basic/fixdep
1160     HOSTCC  scripts/kconfig/conf.o
1161     HOSTCC  scripts/kconfig/confdata.o
1162     HOSTCC  scripts/kconfig/expr.o
1163     HOSTCC  scripts/kconfig/symbol.o
1164     HOSTCC  scripts/kconfig/preprocess.o
1165     HOSTCC  scripts/kconfig/zconf.lex.o
1166     HOSTCC  scripts/kconfig/zconf.tab.o
1167     HOSTLD  scripts/kconfig/conf
1168   scripts/kconfig/conf  --syncconfig Kconfig
1169     HOSTCC  arch/x86/tools/relocs_32.o
1170     HOSTCC  arch/x86/tools/relocs_64.o
1171     HOSTCC  arch/x86/tools/relocs_common.o
1172     HOSTLD  arch/x86/tools/relocs
1173     HOSTCC  scripts/genksyms/genksyms.o
1174     YACC    scripts/genksyms/parse.tab.c
1175     HOSTCC  scripts/genksyms/parse.tab.o
1176     LEX     scripts/genksyms/lex.lex.c
1177     YACC    scripts/genksyms/parse.tab.h
1178     HOSTCC  scripts/genksyms/lex.lex.o
1179     HOSTLD  scripts/genksyms/genksyms
1180     HOSTCC  scripts/bin2c
1181     HOSTCC  scripts/kallsyms
1182     HOSTCC  scripts/conmakehash
1183     HOSTCC  scripts/recordmcount
1184     HOSTCC  scripts/sortextable
1185     HOSTCC  scripts/asn1_compiler
1186     HOSTCC  scripts/sign-file
1187     HOSTCC  scripts/extract-cert
1188     CC      scripts/mod/empty.o
1189     HOSTCC  scripts/mod/mk_elfconfig
1190     MKELF   scripts/mod/elfconfig.h
1191     HOSTCC  scripts/mod/modpost.o
1192     CC      scripts/mod/devicetable-offsets.s
1193     UPD     scripts/mod/devicetable-offsets.h
1194     HOSTCC  scripts/mod/file2alias.o
1195     HOSTCC  scripts/mod/sumversion.o
1196     HOSTLD  scripts/mod/modpost
1197     CC      kernel/bounds.s
1198     CC      arch/x86/kernel/asm-offsets.s
1199     CALL    scripts/checksyscalls.sh
1200
1201 Build the test_pmd application
1202
1203 ::
1204
1205   cd $RTE_SDK
1206   ./usertools/dpdk-setup.sh
1207   Option: 14
1208     CC config.o
1209     CC iofwd.o
1210     CC macfwd.o
1211     CC macswap.o
1212     CC flowgen.o
1213     CC rxonly.o
1214     CC txonly.o
1215     CC csumonly.o
1216     CC icmpecho.o
1217     CC tm.o
1218     LD testpmd
1219     INSTALL-APP testpmd
1220     INSTALL-MAP testpmd.map
1221   == Build app/proc_info
1222     CC main.o
1223     LD dpdk-procinfo
1224     INSTALL-APP dpdk-procinfo
1225     INSTALL-MAP dpdk-procinfo.map
1226   == Build app/pdump
1227     CC main.o
1228     LD dpdk-pdump
1229     INSTALL-APP dpdk-pdump
1230     INSTALL-MAP dpdk-pdump.map
1231   == Build app/test-crypto-perf
1232     CC main.o
1233     CC cperf_ops.o
1234     CC cperf_options_parsing.o
1235     CC cperf_test_vectors.o
1236     CC cperf_test_throughput.o
1237     CC cperf_test_latency.o
1238     CC cperf_test_pmd_cyclecount.o
1239     CC cperf_test_verify.o
1240     CC cperf_test_vector_parsing.o
1241     CC cperf_test_common.o
1242     LD dpdk-test-crypto-perf
1243     INSTALL-APP dpdk-test-crypto-perf
1244     INSTALL-MAP dpdk-test-crypto-perf.map
1245   == Build app/test-eventdev
1246     CC evt_main.o
1247     CC evt_options.o
1248     CC evt_test.o
1249     CC parser.o
1250     CC test_order_common.o
1251     CC test_order_queue.o
1252     CC test_order_atq.o
1253     CC test_perf_common.o
1254     CC test_perf_queue.o
1255     CC test_perf_atq.o
1256     LD dpdk-test-eventdev
1257     INSTALL-APP dpdk-test-eventdev
1258     INSTALL-MAP dpdk-test-eventdev.map
1259   Build complete [x86_64-native-linuxapp-gcc]
1260   Installation cannot run with T defined and DESTDIR undefined
1261   ------------------------------------------------------------------------------
1262   RTE_TARGET exported as x86_64-native-linuxapp-gcc
1263   ------------------------------------------------------------------------------
1264
1265   Press enter to continue ...
1266
1267 Check the VF PCI information:
1268
1269 ::
1270
1271   root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1272   PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1273   PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1274
1275 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
1276
1277 ::
1278
1279   [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1280   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1281   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1282   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1283   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1284   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1285   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1286
1287   [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1288   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1289   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1290   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1291   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1292   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1293   lrwxrwxrwx 1 root root 0 Jun  4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1294
1295   [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1296   [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1297
1298
1299 Get back to pod and run the test_pmd
1300
1301 ::
1302
1303   cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1304
1305   taskset -p --cpu-list 1
1306   pid 1's current affinity list: 2,3,18,19
1307
1308   ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1309   start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1310   peer=1,"9e:fd:e6:dd:c1:01"
1311   EAL: Detected 32 lcore(s)
1312   EAL: No free hugepages reported in hugepages-2048kB
1313   EAL: Probing VFIO support...
1314   EAL: VFIO support initialized
1315   EAL: PCI device 0000:05:11.0 on NUMA socket 0
1316   EAL:   probe driver: 8086:10ed net_ixgbe_vf
1317   EAL:   using IOMMU type 1 (Type 1)
1318   EAL: PCI device 0000:05:11.1 on NUMA socket 0
1319   EAL:   probe driver: 8086:10ed net_ixgbe_vf
1320   Auto-start selected
1321   Ports to start sending a burst of packets first
1322   Warning: lsc_interrupt needs to be off when  using tx_first. Disabling.
1323   USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1324   Configuring Port 0 (socket 0)
1325   Port 0: 9E:FD:E6:DD:C1:01
1326   Configuring Port 1 (socket 0)
1327   Port 1: 9E:FD:E6:DD:C1:02
1328   Checking link statuses...
1329   Port0 Link Up. speed 10000 Mbps- full-duplex
1330   Port1 Link Up. speed 10000 Mbps- full-duplex
1331   Done
1332   No commandline core given, start packet forwarding
1333   io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1334   disabled
1335   Logical Core 3 (socket 0) forwards packets on 2 streams:
1336     RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1337     RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1338
1339     io packet forwarding packets/burst=32
1340     nb forwarding cores=1 - nb forwarding ports=2
1341     port 0:
1342     CRC stripping enabled
1343     RX queues=1 - RX desc=128 - RX free threshold=32
1344     RX threshold registers: pthresh=8 hthresh=8  wthresh=0
1345     TX queues=1 - TX desc=512 - TX free threshold=32
1346     TX threshold registers: pthresh=32 hthresh=0  wthresh=0
1347     TX RS bit threshold=32 - TXQ flags=0xf01
1348     port 1:
1349     CRC stripping enabled
1350     RX queues=1 - RX desc=128 - RX free threshold=32
1351     RX threshold registers: pthresh=8 hthresh=8  wthresh=0
1352     TX queues=1 - TX desc=512 - TX free threshold=32
1353     TX threshold registers: pthresh=32 hthresh=0  wthresh=0
1354     TX RS bit threshold=32 - TXQ flags=0xf01
1355
1356   Port statistics ====================================
1357     ######################## NIC statistics for port 0  ########################
1358     RX-packets: 56         RX-missed: 0          RX-bytes:  4096
1359     RX-errors: 0
1360     RX-nombuf:  0
1361     TX-packets: 64         TX-errors: 0          TX-bytes:  4096
1362
1363     Throughput (since last show)
1364     Rx-pps:            0
1365     Tx-pps:            0
1366     ############################################################################
1367
1368     ######################## NIC statistics for port 1  ########################
1369     RX-packets: 432        RX-missed: 0          RX-bytes:  27712
1370     RX-errors: 0
1371     RX-nombuf:  0
1372     TX-packets: 461        TX-errors: 0          TX-bytes:  30080
1373
1374     Throughput (since last show)
1375     Rx-pps:            0
1376     Tx-pps:            0
1377     ############################################################################
1378
1379   Port statistics ====================================
1380     ######################## NIC statistics for port 0  ########################
1381     RX-packets: 14124641   RX-missed: 0          RX-bytes:  903977344
1382     RX-errors: 0
1383     RX-nombuf:  0
1384     TX-packets: 14170205   TX-errors: 0          TX-bytes:  906893376
1385
1386     Throughput (since last show)
1387     Rx-pps:      7068409
1388     Tx-pps:      7091206
1389     ############################################################################
1390
1391
1392
1393
1394
1395
1396 References
1397 **********
1398
1399 - `StarlingX`_
1400
1401 .. _`StarlingX`: https://docs.starlingx.io/
1402