From a7cf2c70216ab0cd6a5070d48f26883e3527d5ae Mon Sep 17 00:00:00 2001 From: Jackie Huang Date: Thu, 1 Dec 2022 12:04:06 +0800 Subject: [PATCH] docs/installation-guide: update for g-release Issue-ID: INF-295 Signed-off-by: Jackie Huang Change-Id: Iaedf3ab151daafd591b007b7c322688828390ec1 --- docs/installation-guide.rst | 1393 ++----------------------------------------- 1 file changed, 34 insertions(+), 1359 deletions(-) diff --git a/docs/installation-guide.rst b/docs/installation-guide.rst index cff7571..7215c81 100644 --- a/docs/installation-guide.rst +++ b/docs/installation-guide.rst @@ -10,1389 +10,64 @@ INF Installation Guide :depth: 3 :local: -Abstract +Overview ******** -This document describes how to install O-RAN INF image, example configuration (All-in-one Duplex) -for better real time performance, and example deployment of Kubernetes cluster and plugins. +O-RAN INF is a downstream project of `StarlingX`_, and use the same installation and deployment methods. -The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux -and container technology. +Please see the detail of all the supported `Deployment Configurations`_. +Notes: For Yocto based image, only "All-in-one Simplex" and "All-in-one Duplex (up to 50 worker nodes)" are supported. + +.. _`Deployment Configurations`: https://docs.starlingx.io/r/stx.7.0/deploy/index-deploy-da06a98b83b1.html Preface ******* -Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide. +Before starting the installation and deployment of O-RAN INF, you need to download the released ISO image or build from source as described in developer-guide. + +The INF project supports Multi OS and the latest released images for each based OS can be dwonloaded in: + - CentOS 7: `inf-image-centos-all-x86-64.iso`_ + - Debian 11 (bullseye): `inf-image-debian-all-x86-64.iso`_ + - Yocto 2.7 (warrior): `inf-image-yocto-aio-x86-64.iso`_ +.. _`inf-image-debian-all-x86-64.iso`: https://nexus.o-ran-sc.org/content/sites/images/org/o-ran-sc/pti/rtp/g-release/inf-image-debian-all-x86-64.iso +.. _`inf-image-centos-all-x86-64.iso`: https://nexus.o-ran-sc.org/content/sites/images/org/o-ran-sc/pti/rtp/g-release/inf-image-centos-all-x86-64.iso +.. _`inf-image-yocto-aio-x86-64.iso`: https://nexus.o-ran-sc.org/content/sites/images/org/o-ran-sc/pti/rtp/g-release/inf-image-yocto-aio-x86-64.iso Hardware Requirements ********************* -Following minimum hardware requirements must be met for installation of O-RAN INF image with AIO-DX: - -+-------------------+----------------------------------------------------------------------------+ -| **HW Aspect** | **Requirement** | -| | | -+-------------------+----------------------------------------------------------------------------+ -| **# of servers** | 2 | -+-------------------+----------------------------------------------------------------------------+ -| **CPU** | Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket | -| | or | -| | Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) | -+-------------------+----------------------------------------------------------------------------+ -| **RAM** | 32G | -| | | -+-------------------+----------------------------------------------------------------------------+ -| **Disk** | * Disk 1: 500G(It's better to be SSD) | -| | * Disk 2: 1 or more 500 GB for Ceph OSD | -+-------------------+----------------------------------------------------------------------------+ -| **NICs** | * OAM: 1x1GE | -| | * Data: 1 or more x 10GE (optional) | -+-------------------+----------------------------------------------------------------------------+ -| **BIOS settings** | * Hyper-Threading technology enabled | -| | * Virtualization technology enabled | -| | * VT for directed I/O enabled | -| | * CPU power and performance policy set to performance | -| | * CPU C state control disabled | -| | * Plug & play BMC detection disabled | -+-------------------+----------------------------------------------------------------------------+ - -Installation example (All-in-one Duplex) on HP ProLiant DL380p Gen8 -******************************************************************* - -1. Installation for the first server from the O-RAN INF ISO image ------------------------------------------------------------------ - -- Please see the README.md file for how to build the image. -- The Image is a live ISO image with CLI installer: - inf-image-aio-installer-intel-corei7-64.iso - -1.1 Burn the image to the USB device -++++++++++++++++++++++++++++++++++++ - -- Assume the the usb device is /dev/sdX here - -:: - - $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M - -1.2 Install the first server (controller-0) -+++++++++++++++++++++++++++++++++++++++++++ - -- Reboot the target from the USB device. - -- Select “All-in-one Graphics console” or “All-in-one Serial console - install” and press ENTER - -- Start the auto installation - -- It will reboot aotumatically after installation - -2. Configuration and initialize the bootstrap ---------------------------------------------- - -2.1 First Login with sysadmin/sysadmin and change password -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -2.2 Set OAM network before bootstrap -++++++++++++++++++++++++++++++++++++ - -:: - - export OAM_DEV=eno3 - export CONTROLLER0_OAM_CIDR=128.224.210.110/24 - export DEFAULT_OAM_GATEWAY=128.224.210.1 - sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV - sudo ip link set up dev $OAM_DEV - sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV - -2.3 Login the server through SSH with sysadmin -++++++++++++++++++++++++++++++++++++++++++++++ - -2.4 Prepare the localhost.yml for bootstrap -+++++++++++++++++++++++++++++++++++++++++++ - -:: - - cat << EOF > localhost.yml - system_mode: duplex - management_subnet: 192.168.18.0/24 - management_start_address: 192.168.18.2 - management_end_address: 192.168.18.50 - management_gateway_address: 192.168.18.1 - external_oam_subnet: 128.224.210.0/24 - external_oam_gateway_address: 128.224.210.1 - external_oam_floating_address: 128.224.210.110 - external_oam_node_0_address: 128.224.210.111 - external_oam_node_1_address: 128.224.210.112 - EOF - -2.5 Run the ansible bootstrap -+++++++++++++++++++++++++++++ - -:: - - ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv - -After the bootstrap successfully finish, it will show as following: - -:: - - PLAY RECAP ************************************************************************************************************* - localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0 - -2.6 Congiure controller-0 -+++++++++++++++++++++++++ - -Acquire admin credentials: - -:: - - controller-0:~$ source /etc/platform/openrc - [sysadmin@controller-0 ~(keystone_admin)]$ - -Configure the OAM and MGMT interfaces of controller-0 and specify the -attached networks: - -:: - - OAM_IF=eno3 - MGMT_IF=eno1 - system host-if-modify controller-0 lo -c none - IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') - for UUID in $IFNET_UUIDS; do - system interface-network-remove ${UUID} - done - - system host-if-modify controller-0 $OAM_IF -n oam0 - system host-if-modify controller-0 $MGMT_IF -n pxeboot0 - - system host-if-modify controller-0 oam0 -c platform - system interface-network-assign controller-0 oam0 oam - - system host-if-modify controller-0 pxeboot0 -c platform - system interface-network-assign controller-0 pxeboot0 pxeboot - - system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0 - system interface-network-assign controller-0 mgmt0 mgmt - - system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0 - system interface-network-assign controller-0 cluster0 cluster-host - -Example output: - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3 - [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1 - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | lo | - | iftype | virtual | - | ports | [] | - | imac | 00:00:00:00:00:00 | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-17T00:30:45.265032+00:00 | - | updated_at | 2021-11-17T01:03:39.031612+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}') - [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do - > system interface-network-remove ${UUID} - > done - Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e - Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8 - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | oam0 | - | iftype | ethernet | - | ports | [u'eno3'] | - | imac | 24:6e:96:5d:0c:b2 | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-17T00:28:32.365863+00:00 | - | updated_at | 2021-11-17T01:03:45.090904+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | pxeboot0 | - | iftype | ethernet | - | ports | [u'eno1'] | - | imac | 24:6e:96:5d:0c:92 | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-17T00:28:32.612230+00:00 | - | updated_at | 2021-11-17T01:03:47.341003+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | oam0 | - | iftype | ethernet | - | ports | [u'eno3'] | - | imac | 24:6e:96:5d:0c:b2 | - | imtu | 1500 | - | ifclass | platform | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-17T00:28:32.365863+00:00 | - | updated_at | 2021-11-17T01:03:49.368879+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-0 | - | uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 | - | ifname | oam0 | - | network_name | oam | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | pxeboot0 | - | iftype | ethernet | - | ports | [u'eno1'] | - | imac | 24:6e:96:5d:0c:92 | - | imtu | 1500 | - | ifclass | platform | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-17T00:28:32.612230+00:00 | - | updated_at | 2021-11-17T01:03:53.143795+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-0 | - | uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c | - | ifname | pxeboot0 | - | network_name | pxeboot | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | mgmt0 | - | iftype | vlan | - | ports | [] | - | imac | 24:6e:96:5d:0c:92 | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | 18 | - | uses | [u'pxeboot0'] | - | used_by | [] | - | created_at | 2021-11-17T01:03:57.303000+00:00 | - | updated_at | None | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-0 | - | uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 | - | ifname | mgmt0 | - | network_name | mgmt | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | cluster0 | - | iftype | vlan | - | ports | [] | - | imac | 24:6e:96:5d:0c:92 | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vlan_id | 19 | - | uses | [u'pxeboot0'] | - | used_by | [] | - | created_at | 2021-11-17T01:04:02.613518+00:00 | - | updated_at | None | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-0 | - | uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 | - | ifname | cluster0 | - | network_name | cluster-host | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 | - | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 | - | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 | - | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 | - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - -Configure NTP Servers for network time synchronization: - -:: - - system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org - -Output - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 | - | ntpservers | 0.pool.ntp.org,1.pool.ntp.org | - | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a | - | created_at | 2021-11-17T00:27:23.529571+00:00 | - | updated_at | None | - +--------------+--------------------------------------+ - -Add an OSD on controller-0 for Ceph: - -:: - - system host-disk-list controller-0 - system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} - system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {} - system host-stor-list controller-0 - -Output - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | - | | de | num | type | gib | gib | | | | - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | - | | | | | 75 | | | | | - | | | | | | | | | | - | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} - +------------------+-------------------------------------------------------+ - | Property | Value | - +------------------+-------------------------------------------------------+ - | osdid | 0 | - | function | osd | - | state | configuring-on-unlock | - | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c | - | journal_size_gib | 1024 | - | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | - | journal_node | /dev/sdb2 | - | uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | - | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 | - | tier_name | storage | - | created_at | 2021-11-17T01:05:04.063823+00:00 | - | updated_at | None | - +------------------+-------------------------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {} - +------------------+-------------------------------------------------------+ - | Property | Value | - +------------------+-------------------------------------------------------+ - | osdid | 1 | - | function | osd | - | state | configuring-on-unlock | - | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | - | journal_size_gib | 1024 | - | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | - | journal_node | /dev/sdc2 | - | uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | - | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | - | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 | - | tier_name | storage | - | created_at | 2021-11-17T01:05:06.939798+00:00 | - | updated_at | None | - +------------------+-------------------------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0 - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage | - | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage | - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - -2.7 Unlock controller-0 -+++++++++++++++++++++++ - -:: - - system host-unlock controller-0 - -Output: - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0 - +-----------------------+-------------------------------------------------+ - | Property | Value | - +-----------------------+-------------------------------------------------+ - | action | none | - | administrative | locked | - | availability | online | - | bm_ip | None | - | bm_type | none | - | bm_username | None | - | boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | - | capabilities | {u'stor_function': u'monitor'} | - | clock_synchronization | ntp | - | config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 | - | config_status | None | - | config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 | - | console | ttyS0,115200 | - | created_at | 2021-11-17T00:28:01.983673+00:00 | - | hostname | controller-0 | - | id | 1 | - | install_output | graphical | - | install_state | None | - | install_state_info | None | - | inv_state | inventoried | - | invprovision | provisioning | - | location | {} | - | mgmt_ip | 192.168.18.3 | - | mgmt_mac | 24:6e:96:5d:0c:92 | - | operational | disabled | - | personality | controller | - | reserved | False | - | rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | - | serialid | None | - | software_load | 21.05 | - | subfunction_avail | online | - | subfunction_oper | disabled | - | subfunctions | controller,worker,lowlatency | - | task | Unlocking | - | tboot | false | - | ttys_dcd | None | - | updated_at | 2021-11-17T01:05:07.015414+00:00 | - | uptime | 3496 | - | uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 | - | vim_progress_status | None | - +-----------------------+-------------------------------------------------+ - -Controller-0 will reboot to apply configuration changes and come into -service. This can take 5-10 minutes, depending on the performance of the -host machine. - -Once the controller comes back up, check the status of controller-0 - -:: - - controller-0:~$ source /etc/platform/openrc - [sysadmin@controller-0 ~(keystone_admin)]$ system host-list - +----+--------------+-------------+----------------+-------------+--------------+ - | id | hostname | personality | administrative | operational | availability | - +----+--------------+-------------+----------------+-------------+--------------+ - | 1 | controller-0 | controller | unlocked | enabled | available | - +----+--------------+-------------+----------------+-------------+--------------+ - -3. Installation for the second server (controller-1) ----------------------------------------------------- - -3.1 Power on the controller-1 server and force it to network boot -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -3.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -3.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None) -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-list - +----+--------------+-------------+----------------+-------------+--------------+ - | id | hostname | personality | administrative | operational | availability | - +----+--------------+-------------+----------------+-------------+--------------+ - | 1 | controller-0 | controller | unlocked | enabled | degraded | - | 2 | None | None | locked | disabled | offline | - +----+--------------+-------------+----------------+-------------+--------------+ - -3.4 Using the host id, set the personality of this host to 'controller': -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller - +-----------------------+--------------------------------------+ - | Property | Value | - +-----------------------+--------------------------------------+ - | action | none | - | administrative | locked | - | availability | offline | - | bm_ip | None | - | bm_type | None | - | bm_username | None | - | boot_device | /dev/sda | - | capabilities | {} | - | clock_synchronization | ntp | - | config_applied | None | - | config_status | None | - | config_target | None | - | console | ttyS0,115200 | - | created_at | 2021-11-17T10:17:44.387813+00:00 | - | hostname | controller-1 | - | id | 2 | - | install_output | text | - | install_state | None | - | install_state_info | None | - | inv_state | None | - | invprovision | None | - | location | {} | - | mgmt_ip | 192.168.18.4 | - | mgmt_mac | 24:6e:96:5d:38:ee | - | operational | disabled | - | personality | controller | - | reserved | False | - | rootfs_device | /dev/sda | - | serialid | None | - | software_load | 21.05 | - | subfunction_avail | not-installed | - | subfunction_oper | disabled | - | subfunctions | controller,worker,lowlatency | - | task | None | - | tboot | false | - | ttys_dcd | None | - | updated_at | None | - | uptime | 0 | - | uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 | - | vim_progress_status | None | - +-----------------------+--------------------------------------+ +* For INF platform Hardware Requirements, refer to `System Hardware Requirements`_. +* For INF Openstack Hardware Requirements, refer to `Openstack Hardware Requirements`_. -3.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'. -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +* And you can also refer to the `Verified Commercial Hardware`_. -This can take 5-10 minutes, depending on the performance of the host -machine. +.. _`System Hardware Requirements`: https://docs.starlingx.io/planning/kubernetes/starlingx-hardware-requirements.html +.. _`Verified Commercial Hardware`: https://docs.starlingx.io/planning/kubernetes/verified-commercial-hardware.html +.. _`Openstack Hardware Requirements`: https://docs.starlingx.io/planning/openstack/hardware-requirements.html -:: +Installation +************ - [root@controller-0 hieradata(keystone_admin)]$ system host-list - +----+--------------+-------------+----------------+-------------+--------------+ - | id | hostname | personality | administrative | operational | availability | - +----+--------------+-------------+----------------+-------------+--------------+ - | 1 | controller-0 | controller | unlocked | enabled | available | - | 2 | controller-1 | controller | locked | disabled | online | - +----+--------------+-------------+----------------+-------------+--------------+ +Platform Installation +--------------------- -3.6 Configure controller-1 -++++++++++++++++++++++++++ +INF uses the same installation and deployment methods of StarlingX, please refer to `StarlingX Installation`_ for the detail installation steps. -:: +.. _`StarlingX Installation`: https://docs.starlingx.io/r/stx.7.0/deploy_install_guides/r7_release/index-install-r7-8966076f0e81.html - OAM_IF=eno3 - MGMT_IF=eno1 - system host-if-modify controller-1 $OAM_IF -n oam0 - system host-if-modify controller-1 oam0 -c platform - system interface-network-assign controller-1 oam0 oam - system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0 - system interface-network-assign controller-1 cluster0 cluster-host +Applications Installation +------------------------- - system host-if-list controller-1 +Here are the example applications installations: +* `Install O-RAN O2`_ +* `Install FlexRAN`_ - system host-disk-list controller-1 - system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} - system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {} - system host-stor-list controller-1 +.. _`Install O-RAN O2`: https://docs.starlingx.io/admintasks/kubernetes/oran-o2-application-b50a0c899e66.html +.. _`Install FlexRAN`: https://docs.starlingx.io/sample_apps/flexran/deploy-flexran-2203-on-starlingx-1d1b15ecb16f.html -Output: - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3 - [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1 - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | oam0 | - | iftype | ethernet | - | ports | [u'eno3'] | - | imac | 24:6e:96:5d:39:0e | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 | - | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-22T14:01:32.365863+00:00 | - | updated_at | 2021-11-22T15:04:45.090904+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | oam0 | - | iftype | ethernet | - | ports | [u'eno3'] | - | imac | 24:6e:96:5d:39:0e | - | imtu | 1500 | - | ifclass | platform | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 | - | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | vlan_id | None | - | uses | [] | - | used_by | [] | - | created_at | 2021-11-22T14:05:16.052229+00:00 | - | updated_at | 2021-11-22T15:08:35.324634+00:00 | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-1 | - | uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 | - | ifname | oam0 | - | network_name | oam | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0 - +-----------------+--------------------------------------+ - | Property | Value | - +-----------------+--------------------------------------+ - | ifname | cluster0 | - | iftype | vlan | - | ports | [] | - | imac | 24:6e:96:5d:38:ee | - | imtu | 1500 | - | ifclass | None | - | aemode | None | - | schedpolicy | None | - | txhashpolicy | None | - | uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 | - | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | vlan_id | 19 | - | uses | [u'pxeboot0'] | - | used_by | [] | - | created_at | 2021-11-22T15:08:43.932209+00:00 | - | updated_at | None | - | sriov_numvfs | 0 | - | sriov_vf_driver | None | - | accelerated | [True] | - +-----------------+--------------------------------------+ - - [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host - +--------------+--------------------------------------+ - | Property | Value | - +--------------+--------------------------------------+ - | hostname | controller-1 | - | uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 | - | ifname | cluster0 | - | network_name | cluster-host | - +--------------+--------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 | - | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 | - | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 | - | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 | - +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | - | | de | num | type | gib | gib | | | | - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 | - | | | | | 75 | | | | | - | | | | | | | | | | - | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 | - | | | | | 25 | | | | | - | | | | | | | | | | - | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 | - | | | | | 57 | | | | | - | | | | | | | | | | - | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 | - | | | | | 57 | | | | | - | | | | | | | | | | - +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+ - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {} - +------------------+-------------------------------------------------------+ - | Property | Value | - +------------------+-------------------------------------------------------+ - | osdid | 2 | - | function | osd | - | state | configuring-on-unlock | - | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | - | journal_size_gib | 1024 | - | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | - | journal_node | /dev/sdb2 | - | uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | - | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | - | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c | - | tier_name | storage | - | created_at | 2021-11-22T15:11:55.641193+00:00 | - | updated_at | None | - +------------------+-------------------------------------------------------+ - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {} - +------------------+-------------------------------------------------------+ - | Property | Value | - +------------------+-------------------------------------------------------+ - | osdid | 3 | - | function | osd | - | state | configuring-on-unlock | - | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c | - | journal_size_gib | 1024 | - | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | - | journal_node | /dev/sdc2 | - | uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c | - | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae | - | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c | - | tier_name | storage | - | created_at | 2021-11-22T15:12:04.274839+00:00 | - | updated_at | None | - +------------------+-------------------------------------------------------+ - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1 - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage | - | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage | - +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ - -3.7 Unlock controller-1 -+++++++++++++++++++++++ - -Unlock controller-1 in order to bring it into service: - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1 - +-----------------------+--------------------------------------+ - | Property | Value | - +-----------------------+--------------------------------------+ - | action | none | - | administrative | locked | - | availability | online | - | bm_ip | None | - | bm_type | None | - | bm_username | None | - | boot_device | /dev/sda | - | capabilities | {u'stor_function': u'monitor'} | - | clock_synchronization | ntp | - | config_applied | None | - | config_status | Config out-of-date | - | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac | - | console | ttyS0,115200 | - | created_at | 2021-11-22T12:58:11.630526+00:00 | - | hostname | controller-1 | - | id | 3 | - | install_output | text | - | install_state | None | - | install_state_info | None | - | inv_state | inventoried | - | invprovision | unprovisioned | - | location | {} | - | mgmt_ip | 192.168.18.4 | - | mgmt_mac | 24:6e:96:5d:38:ee | - | operational | disabled | - | personality | controller | - | reserved | False | - | rootfs_device | /dev/sda | - | serialid | None | - | software_load | 21.05 | - | subfunction_avail | online | - | subfunction_oper | disabled | - | subfunctions | controller,worker,lowlatency | - | task | Unlocking | - | tboot | false | - | ttys_dcd | None | - | updated_at | 2021-11-22T15:13:09.716324+00:00 | - | uptime | 752 | - | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | vim_progress_status | None | - +-----------------------+--------------------------------------+ - -Controller-1 will reboot in order to apply configuration changes and -come into service. This can take 5-10 minutes, depending on the -performance of the host machine. - -:: - - [root@controller-0 hieradata(keystone_admin)]$ system host-list - +----+--------------+-------------+----------------+-------------+--------------+ - | id | hostname | personality | administrative | operational | availability | - +----+--------------+-------------+----------------+-------------+--------------+ - | 1 | controller-0 | controller | unlocked | enabled | available | - | 2 | controller-1 | controller | unlocked | enabled | available | - +----+--------------+-------------+----------------+-------------+--------------+ - - [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1 - +-----------------------+-----------------------------------------------------------------------+ - | Property | Value | - +-----------------------+-----------------------------------------------------------------------+ - | action | none | - | administrative | unlocked | - | availability | available | - | bm_ip | None | - | bm_type | None | - | bm_username | None | - | boot_device | /dev/sda | - | capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} | - | clock_synchronization | ntp | - | config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac | - | config_status | None | - | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac | - | console | ttyS0,115200 | - | created_at | 2021-11-22T12:58:11.630526+00:00 | - | hostname | controller-1 | - | id | 2 | - | install_output | text | - | install_state | None | - | install_state_info | None | - | inv_state | inventoried | - | invprovision | provisioned | - | location | {} | - | mgmt_ip | 192.168.18.4 | - | mgmt_mac | 24:6e:96:5d:38:ee | - | operational | enabled | - | personality | controller | - | reserved | False | - | rootfs_device | /dev/sda | - | serialid | None | - | software_load | 21.05 | - | subfunction_avail | available | - | subfunction_oper | enabled | - | subfunctions | controller,worker,lowlatency | - | task | | - | tboot | false | - | ttys_dcd | None | - | updated_at | 2021-11-22T23:59:07.787759+00:00 | - | uptime | 31008 | - | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc | - | vim_progress_status | services-enabled | - +-----------------------+-----------------------------------------------------------------------+ - - -4. Simple use case for sriov ----------------------------- - -4.1 After controller-0 is rebooted and up running, download the DPDK -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -:: - - [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt - [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz - Password: - --2021-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz - Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ... - Connecting to fast.dpdk.org|151.101.2.49|:443... connected. - - HTTP request sent, awaiting response... 200 OK - Length: 10251680 (9.8M) [application/octet-stream] - Saving to: ‘dpdk-17.11.10.tar.xz’ - - dpdk-17.11.10.tar.xz 100% - [========================================================================================>] 9.78M - 1.48MB/s in 6.8s - - 2021-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680] - - sudo tar xvf dpdk-17.11.10.tar.xz - - sudo ln -s dpdk-stable-17.11.10 dpdk-stable - -4.2 Prepare the yaml file for the network assignment container -++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -The following the exmaple of the yaml file: - -:: - - [sysadmin@controller-0 sriov(keystone_admin)]$ cat < netdef-data-dpdk.yaml - > apiVersion: "k8s.cni.cncf.io/v1" - > kind: NetworkAttachmentDefinition - > metadata: - > name: sriov-data-dpdk-0 - > annotations: - > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0 - > spec: - > config: '{ - > "type": "sriov", - > "name": "sriov-data-dpdk-0" - > }' - > - > --- - > apiVersion: "k8s.cni.cncf.io/v1" - > kind: NetworkAttachmentDefinition - > metadata: - > name: sriov-data-dpdk-1 - > annotations: - > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1 - > spec: - > config: '{ - > "type": "sriov", - > "name": "sriov-data-dpdk-1" - > }' - > EOF - -4.3 Run the network assignent container for the 2 VFs -+++++++++++++++++++++++++++++++++++++++++++++++++++++ - -:: - - [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml - networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created - networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created - -4.4 Prepare the VF container yaml file -++++++++++++++++++++++++++++++++++++++ - -:: - - [sysadmin@controller-0 sriov(keystone_admin)]$ cat < pod-with-dpdk-vfs-0.yaml - > apiVersion: v1 - > kind: Pod - metadata: - > metadata: - > name: pod-with-dpdk-vfs-0 - > annotations: - > k8s.v1.cni.cncf.io/networks: '[ - > { "name": "sriov-data-dpdk-0" }, - { "name": "sriov-data-dpdk-1" } - > { "name": "sriov-data-dpdk-1" } - > ]' - > spec: - > restartPolicy: Never - > containers: - > - name: pod-with-dpdk-vfs-0 - > image: wrsnfv/ubuntu-dpdk-build:v0.3 - > env: - > - name: RTE_SDK - > value: "/usr/src/dpdk" - > command: - > - sleep - > - infinity - > stdin: true - > tty: true - > securityContext: - > privileged: true - > capabilities: - > add: - > - ALL - > resources: - > requests: - > cpu: 4 - > memory: 4Gi - > intel.com/pci_sriov_net_physnet0: '1' - > intel.com/pci_sriov_net_physnet1: '1' - > limits: - > cpu: 4 - > hugepages-1Gi: 2Gi - > memory: 4Gi - > intel.com/pci_sriov_net_physnet0: '1' - > intel.com/pci_sriov_net_physnet1: '1' - > volumeMounts: - > - mountPath: /mnt/huge-1048576kB - > name: hugepage - > - name: dpdk-volume - > mountPath: /usr/src/dpdk - > - name: lib-volume - > mountPath: /lib/modules - > - name: src-volume - > mountPath: /usr/src/ - > volumes: - > - name: hugepage - > emptyDir: - > medium: HugePages - > - name: dpdk-volume - > hostPath: - > path: /opt/dpdk-stable/ - > - name: lib-volume - > hostPath: - > path: /lib/modules - > - name: src-volume - > hostPath: - > path: /usr/src/ - > EOF - -4.5 Run the VF container -++++++++++++++++++++++++ - -Start the VF container. - -:: - - [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml - pod/pod-with-dpdk-vfs-0 created - - [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod - NAME READY STATUS RESTARTS AGE - pod-with-dpdk-vfs-0 1/1 Running 0 6m40s - -Login the VF container - -:: - - kubectl exec -it pod-with-dpdk-vfs-0 -- bash - -Build the DPDK - -:: - - cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build - - root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare - HOSTCC scripts/basic/fixdep - HOSTCC scripts/kconfig/conf.o - HOSTCC scripts/kconfig/confdata.o - HOSTCC scripts/kconfig/expr.o - HOSTCC scripts/kconfig/symbol.o - HOSTCC scripts/kconfig/preprocess.o - HOSTCC scripts/kconfig/zconf.lex.o - HOSTCC scripts/kconfig/zconf.tab.o - HOSTLD scripts/kconfig/conf - scripts/kconfig/conf --syncconfig Kconfig - HOSTCC arch/x86/tools/relocs_32.o - HOSTCC arch/x86/tools/relocs_64.o - HOSTCC arch/x86/tools/relocs_common.o - HOSTLD arch/x86/tools/relocs - HOSTCC scripts/genksyms/genksyms.o - YACC scripts/genksyms/parse.tab.c - HOSTCC scripts/genksyms/parse.tab.o - LEX scripts/genksyms/lex.lex.c - YACC scripts/genksyms/parse.tab.h - HOSTCC scripts/genksyms/lex.lex.o - HOSTLD scripts/genksyms/genksyms - HOSTCC scripts/bin2c - HOSTCC scripts/kallsyms - HOSTCC scripts/conmakehash - HOSTCC scripts/recordmcount - HOSTCC scripts/sortextable - HOSTCC scripts/asn1_compiler - HOSTCC scripts/sign-file - HOSTCC scripts/extract-cert - CC scripts/mod/empty.o - HOSTCC scripts/mod/mk_elfconfig - MKELF scripts/mod/elfconfig.h - HOSTCC scripts/mod/modpost.o - CC scripts/mod/devicetable-offsets.s - UPD scripts/mod/devicetable-offsets.h - HOSTCC scripts/mod/file2alias.o - HOSTCC scripts/mod/sumversion.o - HOSTLD scripts/mod/modpost - CC kernel/bounds.s - CC arch/x86/kernel/asm-offsets.s - CALL scripts/checksyscalls.sh - -Build the test_pmd application - -:: - - cd $RTE_SDK - ./usertools/dpdk-setup.sh - Option: 14 - CC config.o - CC iofwd.o - CC macfwd.o - CC macswap.o - CC flowgen.o - CC rxonly.o - CC txonly.o - CC csumonly.o - CC icmpecho.o - CC tm.o - LD testpmd - INSTALL-APP testpmd - INSTALL-MAP testpmd.map - == Build app/proc_info - CC main.o - LD dpdk-procinfo - INSTALL-APP dpdk-procinfo - INSTALL-MAP dpdk-procinfo.map - == Build app/pdump - CC main.o - LD dpdk-pdump - INSTALL-APP dpdk-pdump - INSTALL-MAP dpdk-pdump.map - == Build app/test-crypto-perf - CC main.o - CC cperf_ops.o - CC cperf_options_parsing.o - CC cperf_test_vectors.o - CC cperf_test_throughput.o - CC cperf_test_latency.o - CC cperf_test_pmd_cyclecount.o - CC cperf_test_verify.o - CC cperf_test_vector_parsing.o - CC cperf_test_common.o - LD dpdk-test-crypto-perf - INSTALL-APP dpdk-test-crypto-perf - INSTALL-MAP dpdk-test-crypto-perf.map - == Build app/test-eventdev - CC evt_main.o - CC evt_options.o - CC evt_test.o - CC parser.o - CC test_order_common.o - CC test_order_queue.o - CC test_order_atq.o - CC test_perf_common.o - CC test_perf_queue.o - CC test_perf_atq.o - LD dpdk-test-eventdev - INSTALL-APP dpdk-test-eventdev - INSTALL-MAP dpdk-test-eventdev.map - Build complete [x86_64-native-linuxapp-gcc] - Installation cannot run with T defined and DESTDIR undefined - ------------------------------------------------------------------------------ - RTE_TARGET exported as x86_64-native-linuxapp-gcc - ------------------------------------------------------------------------------ - - Press enter to continue ... - -Check the VF PCI information: - -:: - - root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM - PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1 - PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0 - -Exit from pod back to host to find which VFs are assigned to this pod by check the pci address: - -:: - - [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn* - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2 - - [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn* - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1 - lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3 - - [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01 - [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02 - - -Get back to pod and run the test_pmd - -:: - - cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/ - - taskset -p --cpu-list 1 - pid 1's current affinity list: 2,3,18,19 - - ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto- - start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth- - peer=1,"9e:fd:e6:dd:c1:01" - EAL: Detected 32 lcore(s) - EAL: No free hugepages reported in hugepages-2048kB - EAL: Probing VFIO support... - EAL: VFIO support initialized - EAL: PCI device 0000:05:11.0 on NUMA socket 0 - EAL: probe driver: 8086:10ed net_ixgbe_vf - EAL: using IOMMU type 1 (Type 1) - EAL: PCI device 0000:05:11.1 on NUMA socket 0 - EAL: probe driver: 8086:10ed net_ixgbe_vf - Auto-start selected - Ports to start sending a burst of packets first - Warning: lsc_interrupt needs to be off when using tx_first. Disabling. - USER1: create a new mbuf pool : n=155456, size=2176, socket=0 - Configuring Port 0 (socket 0) - Port 0: 9E:FD:E6:DD:C1:01 - Configuring Port 1 (socket 0) - Port 1: 9E:FD:E6:DD:C1:02 - Checking link statuses... - Port0 Link Up. speed 10000 Mbps- full-duplex - Port1 Link Up. speed 10000 Mbps- full-duplex - Done - No commandline core given, start packet forwarding - io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages - disabled - Logical Core 3 (socket 0) forwards packets on 2 streams: - RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01 - RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02 - - io packet forwarding packets/burst=32 - nb forwarding cores=1 - nb forwarding ports=2 - port 0: - CRC stripping enabled - RX queues=1 - RX desc=128 - RX free threshold=32 - RX threshold registers: pthresh=8 hthresh=8 wthresh=0 - TX queues=1 - TX desc=512 - TX free threshold=32 - TX threshold registers: pthresh=32 hthresh=0 wthresh=0 - TX RS bit threshold=32 - TXQ flags=0xf01 - port 1: - CRC stripping enabled - RX queues=1 - RX desc=128 - RX free threshold=32 - RX threshold registers: pthresh=8 hthresh=8 wthresh=0 - TX queues=1 - TX desc=512 - TX free threshold=32 - TX threshold registers: pthresh=32 hthresh=0 wthresh=0 - TX RS bit threshold=32 - TXQ flags=0xf01 - - Port statistics ==================================== - ######################## NIC statistics for port 0 ######################## - RX-packets: 56 RX-missed: 0 RX-bytes: 4096 - RX-errors: 0 - RX-nombuf: 0 - TX-packets: 64 TX-errors: 0 TX-bytes: 4096 - - Throughput (since last show) - Rx-pps: 0 - Tx-pps: 0 - ############################################################################ - - ######################## NIC statistics for port 1 ######################## - RX-packets: 432 RX-missed: 0 RX-bytes: 27712 - RX-errors: 0 - RX-nombuf: 0 - TX-packets: 461 TX-errors: 0 TX-bytes: 30080 - - Throughput (since last show) - Rx-pps: 0 - Tx-pps: 0 - ############################################################################ - - Port statistics ==================================== - ######################## NIC statistics for port 0 ######################## - RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344 - RX-errors: 0 - RX-nombuf: 0 - TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376 - - Throughput (since last show) - Rx-pps: 7068409 - Tx-pps: 7091206 - ############################################################################ - - - - - - References ********** -- 2.16.6