X-Git-Url: https://gerrit.o-ran-sc.org/r/gitweb?a=blobdiff_plain;f=docs%2Finstallation-guide.rst;h=ae524a944cbbb844d549d9a63b400622f7d6d433;hb=5e0fe87d85c4cb20196ec940f14c7ff9fed1276f;hp=6bde8ba767593a8511b1e7271cac4cb80ab5588a;hpb=7ed3560f2741c1e5ea814592eca2d8e1a81dcc1e;p=pti%2Frtp.git diff --git a/docs/installation-guide.rst b/docs/installation-guide.rst index 6bde8ba..ae524a9 100644 --- a/docs/installation-guide.rst +++ b/docs/installation-guide.rst @@ -28,7 +28,7 @@ Version history | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version | | | | | | +--------------------+--------------------+--------------------+--------------------+ -| | | | | +| 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | | | | | @@ -53,16 +53,16 @@ Following minimum hardware requirements must be met for installation of O-RAN IN +--------------------+----------------------------------------------------+ | **# of servers** | 1 | +--------------------+----------------------------------------------------+ -| **CPU** | 2 | +| **CPU** | 4 | | | | +--------------------+----------------------------------------------------+ -| **RAM** | 4G | +| **RAM** | 16G | | | | +--------------------+----------------------------------------------------+ -| **Disk** | 20G | +| **Disk** | 500G * 2 | | | | +--------------------+----------------------------------------------------+ -| **NICs** | 1 | +| **NICs** | 2 | | | | +--------------------+----------------------------------------------------+ @@ -75,7 +75,7 @@ Software Installation and Deployment ```````````````````````````````````````````` - Please see the README.md file for how to build the image. -- The Image is a live ISO image with CLI installer: oran-image-inf-host-intel-x86-64.iso +- The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso 1.1 Burn the image to the USB device '''''''''''''''''''''''''''''''''''' @@ -84,279 +84,1075 @@ Software Installation and Deployment :: - $ sudo dd if=/path/to/oran-image-inf-host-intel-x86-64.iso of=/dev/sdX bs=1M - -1.2 Insert the USB device in the target to be booted. -''''''''''''''''''''''''''''''''''''''''''''''''''''' - -1.3 Reboot the target from the USB device. -'''''''''''''''''''''''''''''''''''''''''' - -1.4 Select "Graphics console install" or "Serial console install" and press ENTER -''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M -1.5 Select the hard disk and press ENTER -'''''''''''''''''''''''''''''''''''''''' +1.2 Install the server +'''''''''''''''''''''' -Notes: In this installer, you can only select which hard disk to install, the whole disk will be used and partitioned automatically. +- Reboot the target from the USB device. -- e.g. insert "sda" and press ENTER +- Select "Graphics console install" or "Serial console install" and press ENTER -1.6 Remove the USB device and press ENTER to reboot -''''''''''''''''''''''''''''''''''''''''''''''''''' +- Remove the USB device and press ENTER to reboot -2. Configuration for better real time performance -````````````````````````````````````````````````` -Notes: Some of the tuning options are machine specific or depend on use cases, -like the hugepages, isolcpus, rcu_nocbs, kthread_cpus, irqaffinity, nohz_full and -so on, please do not just copy and past. +2. Configuration and initialize the bootstrap +````````````````````````````````````````````` -- Edit the grub.cfg with the following example tuning options +2.1 First Login +''''''''''''''' +Example: :: - # Notes: the grub.cfg file path is different for legacy and UEFI mode - # For legacy mode: /boot/grub/grub.cfg - # For UEFI mode: /boot/EFI/BOOT/grub.cfg - - grub_cfg="/boot/grub/grub.cfg" - #grub_cfg="/boot/EFI/BOOT/grub.cfg" + ifconfig eno1 128.224.180.14/24 up - # In this example, 1-16 cores are isolated for real time processes - root@intel-x86-64:~# rt_tuning="crashkernel=auto biosdevname=0 iommu=pt usbcore.autosuspend=-1 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on cgroup_enable=memory skew_tick=1 hugepagesz=1G hugepages=4 default_hugepagesz=1G isolcpus=1-16 rcu_nocbs=1-16 kthread_cpus=0 irqaffinity=0 nohz=on nohz_full=1-16 intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable nosoftlockup idle=poll mce=ignore_ce" + ip route add default via 128.224.180.1 - # optional to add the console setting - root@intel-x86-64:~# console="console=ttyS0,115200" + ping 8.8.8.8 - root@intel-x86-64:~# sed -i "/linux / s/$/ $console $rt_tuning/" $grub_cfg +2.3 Login the server through SSH +'''''''''''''''''''''''''''''''' +2.4 Prepare the localhost.yml for bootstrap +''''''''''''''''''''''''''''''''''''''''''' -- Reboot the target +Example: :: - root@intel-x86-64:~# reboot - -3. Kubernetes cluster and plugins deployment instructions (All-in-one) -`````````````````````````````````````````````````````````````````````` -This instruction will show you how to deploy kubernetes cluster and plugins in an all-in-one example scenario after the above installation. - -3.1 Change the hostname (Optional) -'''''''''''''''''''''''''''''''''' - -:: + system_mode: simplex - # Assuming the hostname is oran-aio, ip address is - # please DO NOT copy and paste, use your actaul hostname and ip address - root@intel-x86-64:~# echo oran-aio > /etc/hostname - root@intel-x86-64:~# export AIO_HOST_IP="" - root@intel-x86-64:~# echo "$AIO_HOST_IP oran-aio" >> /etc/hosts + external_oam_subnet: 128.224.180.0/24 -3.2 Disable swap for Kubernetes -''''''''''''''''''''''''''''''' + external_oam_gateway_address: 128.224.180.1 -:: + external_oam_floating_address: 128.224.180.14 - root@intel-x86-64:~# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab - root@intel-x86-64:~# systemctl mask dev-sda4.swap + admin_username: admin -3.3 Set the proxy for docker (Optional) -''''''''''''''''''''''''''''''''''''''' + admin_password: Wind123! -- If you are under a firewall, you may need to set the proxy for docker to pull images + ansible_become_pass: Wind123! -:: + docker_http_proxy: http://xxx.xxx.xxx.xxx:8080 - root@intel-x86-64:~# HTTP_PROXY="http://:" - root@intel-x86-64:~# mkdir /etc/systemd/system/docker.service.d/ - root@intel-x86-64:~# cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf - [Service] - Environment="HTTP_PROXY=$HTTP_PROXY" "NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,$AIO_HOST_IP,10.244.0.0/16" - EOF + docker_https_proxy: https://xxx.xxx.xxx.xxx:8080 -3.4 Reboot the target +2.5 Run the bootstrap ''''''''''''''''''''' -:: - - root@intel-x86-64:~# reboot - -3.5 Initialize kubernetes cluster master -'''''''''''''''''''''''''''''''''''''''' +Use the following command: :: - root@oran-aio:~# kubeadm init --kubernetes-version v1.15.2 --pod-network-cidr=10.244.0.0/16 - root@oran-aio:~# mkdir -p $HOME/.kube - root@oran-aio:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config - root@oran-aio:~# chown $(id -u):$(id -g) $HOME/.kube/config + ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv -3.6 Make the master also works as a worker node -''''''''''''''''''''''''''''''''''''''''''''''' +After the bootstrap successfully finish, it will show as following: :: - root@oran-aio:~# kubectl taint nodes oran-aio node-role.kubernetes.io/master- + PLAY RECAP ****************************************************************************************************************************************** + localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0 -3.7 Deploy flannel -'''''''''''''''''' +2.6 Configure the OAM network +''''''''''''''''''''''''''''' -:: +Example: - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/flannel/kube-flannel.yml +:: -Check that the aio node is ready after flannel is successfully deployed and running + controller-0:~$ source /etc/platform/openrc + + [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1 + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform + + +-----------------+--------------------------------------+ + | Property | Value | + +-----------------+--------------------------------------+ + | ifname | eno1 | + +-----------------+--------------------------------------+ + | iftype | ethernet | + +-----------------+--------------------------------------+ + | ports | [u'eno1'] | + +-----------------+--------------------------------------+ + | imac | 3c:a8:2a:1d:31:d0 | + +-----------------+--------------------------------------+ + | imtu | 1500 | + +-----------------+--------------------------------------+ + | ifclass | platform | + +-----------------+--------------------------------------+ + | aemode | None | + +-----------------+--------------------------------------+ + | schedpolicy | None | + +-----------------+--------------------------------------+ + | txhashpolicy | None | + +-----------------+--------------------------------------+ + | uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 | + +-----------------+--------------------------------------+ + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-----------------+--------------------------------------+ + | vlan_id | None | + +-----------------+--------------------------------------+ + | uses | [] | + +-----------------+--------------------------------------+ + | used_by | [] | + +-----------------+--------------------------------------+ + | created_at | 2020-06-03T08:21:17.526303+00:00 | + +-----------------+--------------------------------------+ + | updated_at | 2020-06-03T10:02:54.682501+00:00 | + +-----------------+--------------------------------------+ + | sriov_numvfs | 0 | + +-----------------+--------------------------------------+ + | sriov_vf_driver | None | + +-----------------+--------------------------------------+ + | accelerated | [False] | + +-----------------+--------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam + + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | hostname | controller-0 | + +--------------+--------------------------------------+ + | uuid | 32592cc3-9d66-4961-a51e-23686da00290 | + +--------------+--------------------------------------+ + | ifname | eno1 | + +--------------+--------------------------------------+ + | network_name | oam | + +--------------+--------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ + [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0 + + +--------------+--------------------------------------+--------+--------------+ + | hostname | uuid | ifname | network_name | + +--------------+--------------------------------------+--------+--------------+ + | controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt | + +--------------+--------------------------------------+--------+--------------+ + | controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam | + +--------------+--------------------------------------+--------+--------------+ + | controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host | + +--------------+--------------------------------------+--------+--------------+ + +2.7 Configure the NTP server +'''''''''''''''''''''''''''' + +Example: :: + + [sysadmin@controller-0 ~(keystone_admin)]$ + [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org + + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 | + +--------------+--------------------------------------+ + | ntpservers | 0.pool.ntp.org,1.pool.ntp.org | + +--------------+--------------------------------------+ + | isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 | + +--------------+--------------------------------------+ + | created_at | 2020-06-03T08:19:47.812249+00:00 | + +--------------+--------------------------------------+ + | updated_at | None | + +--------------+--------------------------------------+ + +2.8 Configure the Storage by using the second hard drive as ceph backend +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Example: - root@oran-aio:~# kubectl get pods --all-namespaces |grep flannel - kube-system kube-flannel-ds-amd64-bwt52 1/1 Running 0 3m24s - - root@oran-aio:~# kubectl get nodes - NAME STATUS ROLES AGE VERSION - oran-aio Ready master 3m17s v1.15.2-dirty +:: -3.8 Deploy kubernetes dashboard -''''''''''''''''''''''''''''''' + [sysadmin@controller-0 ~(keystone_admin)]$ + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 + + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path | + | | de | num | type | gib | gib | | id | | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | | | | | | | | | | + | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + | | | | | | | | | | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {} + + +------------------+-------------------------------------------------------+ + | Property | Value | + +------------------+-------------------------------------------------------+ + | osdid | 0 | + +------------------+-------------------------------------------------------+ + | function | osd | + +------------------+-------------------------------------------------------+ + | state | configuring-on-unlock | + +------------------+-------------------------------------------------------+ + | journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | + +------------------+-------------------------------------------------------+ + | journal_size_gib | 1024 | + +------------------+-------------------------------------------------------+ + | journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | + +------------------+-------------------------------------------------------+ + | journal_node | /dev/sdb2 | + +------------------+-------------------------------------------------------+ + | uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | + +------------------+-------------------------------------------------------+ + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +------------------+-------------------------------------------------------+ + | idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | + +------------------+-------------------------------------------------------+ + | tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d | + +------------------+-------------------------------------------------------+ + | tier_name | storage | + +------------------+-------------------------------------------------------+ + | created_at | 2020-06-03T10:03:55.645744+00:00 | + +------------------+-------------------------------------------------------+ + | updated_at | None | + +------------------+-------------------------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0 + + +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name | + +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ + | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage | + +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+ + +2.9 Configure the SR-IOV enabled network +'''''''''''''''''''''''''''''''''''''''' -Deploy kubernetes dashboard +It needs : + - assign the label of SR-IOV + - change the CPU policy to static + - change the hugepage setting + - create the data network + - assign the SR-IOV interface to data network + +Example: :: - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard-admin.rbac.yaml - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard.yaml - -Verify that the dashboard is up and running + [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0 + + [sysadmin@controller-0 ~(keystone_admin)]$ + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled + + +-------------+--------------------------------------+ + | Property | Value | + +-------------+--------------------------------------+ + | uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b | + +-------------+--------------------------------------+ + | host_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-------------+--------------------------------------+ + | label_key | sriovdp | + +-------------+--------------------------------------+ + | label_value | enabled | + +-------------+--------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static + + +-------------+--------------------------------------+ + | Property | Value | + +-------------+--------------------------------------+ + | uuid | 4936f32b-6939-4115-b643-846bd50e7738 | + +-------------+--------------------------------------+ + | host_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-------------+--------------------------------------+ + | label_key | kube-cpu-mgr-policy | + +-------------+--------------------------------------+ + | label_value | static | + +-------------+--------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0 + + +-------------------------------------+--------------------------------------+ + | Property | Value | + +-------------------------------------+--------------------------------------+ + | Memory: Usable Total (MiB) | 0 | + +-------------------------------------+--------------------------------------+ + | Platform (MiB) | 7000 | + +-------------------------------------+--------------------------------------+ + | Available (MiB) | 0 | + +-------------------------------------+--------------------------------------+ + | Huge Pages Configured | True | + +-------------------------------------+--------------------------------------+ + | vSwitch Huge Pages: Size (MiB) | 1024 | + +-------------------------------------+--------------------------------------+ + | Total | 0 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | Required | None | + +-------------------------------------+--------------------------------------+ + | Application Pages (4K): Total | None | + +-------------------------------------+--------------------------------------+ + | Application Huge Pages (2M): Total | 0 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | Application Huge Pages (1G): Total | 0 | + +-------------------------------------+--------------------------------------+ + | Total Pending | 7 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 | + +-------------------------------------+--------------------------------------+ + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-------------------------------------+--------------------------------------+ + | inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 | + +-------------------------------------+--------------------------------------+ + | created_at | 2020-06-03T08:21:27.468357+00:00 | + +-------------------------------------+--------------------------------------+ + | updated_at | 2020-06-03T10:10:49.159080+00:00 | + +-------------------------------------+--------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1 + + +-------------------------------------+--------------------------------------+ + | Property | Value | + +-------------------------------------+--------------------------------------+ + | Memory: Usable Total (MiB) | 0 | + +-------------------------------------+--------------------------------------+ + | Platform (MiB) | 1000 | + +-------------------------------------+--------------------------------------+ + | Available (MiB) | 0 | + +-------------------------------------+--------------------------------------+ + | Huge Pages Configured | True | + +-------------------------------------+--------------------------------------+ + | vSwitch Huge Pages: Size (MiB) | 1024 | + +-------------------------------------+--------------------------------------+ + | Total | 0 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | Required | None | + +-------------------------------------+--------------------------------------+ + | Application Pages (4K): Total | None | + +-------------------------------------+--------------------------------------+ + | Application Huge Pages (2M): Total | 0 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | Application Huge Pages (1G): Total | 0 | + +-------------------------------------+--------------------------------------+ + | Total Pending | 7 | + +-------------------------------------+--------------------------------------+ + | Available | 0 | + +-------------------------------------+--------------------------------------+ + | uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 | + +-------------------------------------+--------------------------------------+ + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-------------------------------------+--------------------------------------+ + | inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f | + +-------------------------------------+--------------------------------------+ + | created_at | 2020-06-03T08:21:27.566279+00:00 | + +-------------------------------------+--------------------------------------+ + | updated_at | 2020-06-03T10:10:49.220907+00:00 | + +-------------------------------------+--------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0' + + [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1' + + [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0 + + [sysadmin@controller-0 ~(keystone_admin)]$ + + [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat + + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | id | 1 | + +--------------+--------------------------------------+ + | uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b | + +--------------+--------------------------------------+ + | name | physnet0 | + +--------------+--------------------------------------+ + | network_type | flat | + +--------------+--------------------------------------+ + | mtu | 1500 | + +--------------+--------------------------------------+ + | description | None | + +--------------+--------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat + + +--------------+--------------------------------------+ + | Property | Value | + +--------------+--------------------------------------+ + | id | 2 | + +--------------+--------------------------------------+ + | uuid | 899ca8ba-6800-4d62-aed4-3770878da738 | + +--------------+--------------------------------------+ + | name | physnet1 | + +--------------+--------------------------------------+ + | network_type | flat | + +--------------+--------------------------------------+ + | mtu | 1500 | + +--------------+--------------------------------------+ + | description | None | + +--------------+--------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0 + + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | uuid | name | type | pci address | device | processor | accelerated | device type | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + | 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] | + +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+ + + + [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0 + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF + + +-----------------+--------------------------------------+ + | Property | Value | + +-----------------+--------------------------------------+ + | ifname | sriov0 | + +-----------------+--------------------------------------+ + | iftype | ethernet | + +-----------------+--------------------------------------+ + | ports | [u'ens2f0'] | + +-----------------+--------------------------------------+ + | imac | 00:1b:21:b3:46:2c | + +-----------------+--------------------------------------+ + | imtu | 1500 | + +-----------------+--------------------------------------+ + | ifclass | pci-sriov | + +-----------------+--------------------------------------+ + | aemode | None | + +-----------------+--------------------------------------+ + | schedpolicy | None | + +-----------------+--------------------------------------+ + | txhashpolicy | None | + +-----------------+--------------------------------------+ + | uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 | + +-----------------+--------------------------------------+ + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-----------------+--------------------------------------+ + | vlan_id | None | + +-----------------+--------------------------------------+ + | uses | [] | + +-----------------+--------------------------------------+ + | used_by | [] | + +-----------------+--------------------------------------+ + | created_at | 2020-06-03T12:34:11.633254+00:00 | + +-----------------+--------------------------------------+ + | updated_at | 2020-06-04T02:01:28.168127+00:00 | + +-----------------+--------------------------------------+ + | sriov_numvfs | 6 | + +-----------------+--------------------------------------+ + | sriov_vf_driver | vfio | + +-----------------+--------------------------------------+ + | accelerated | [True] | + +-----------------+--------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0} + + +------------------+--------------------------------------+ + | Property | Value | + +------------------+--------------------------------------+ + | hostname | controller-0 | + +------------------+--------------------------------------+ + | uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 | + +------------------+--------------------------------------+ + | ifname | sriov0 | + +------------------+--------------------------------------+ + | datanetwork_name | physnet0 | + +------------------+--------------------------------------+ + + + + [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1 + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF + + +-----------------+--------------------------------------+ + | Property | Value | + +-----------------+--------------------------------------+ + | ifname | sriov1 | + +-----------------+--------------------------------------+ + | iftype | ethernet | + +-----------------+--------------------------------------+ + | ports | [u'ens2f1'] | + +-----------------+--------------------------------------+ + | imac | 00:1b:21:b3:46:2d | + +-----------------+--------------------------------------+ + | imtu | 1500 | + +-----------------+--------------------------------------+ + | ifclass | pci-sriov | + +-----------------+--------------------------------------+ + | aemode | None | + +-----------------+--------------------------------------+ + | schedpolicy | None | + +-----------------+--------------------------------------+ + | txhashpolicy | None | + +-----------------+--------------------------------------+ + | uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 | + | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e | + | vlan_id | None | + | uses | [] | + | used_by | [] | + | created_at | 2020-06-03T12:34:12.072624+00:00 | + | updated_at | 2020-06-04T02:03:51.405322+00:00 | + | sriov_numvfs | 6 | + | sriov_vf_driver | vfio | + | accelerated | [True] | + +-----------------+--------------------------------------+ + + [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1} + + +------------------+--------------------------------------+ + | Property | Value | + +------------------+--------------------------------------+ + | hostname | controller-0 | + +------------------+--------------------------------------+ + | uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f | + +------------------+--------------------------------------+ + | ifname | sriov1 | + +------------------+--------------------------------------+ + | datanetwork_name | physnet1 | + +------------------+--------------------------------------+ + +2.10 Unlock the server +'''''''''''''''''''''' + +Example: :: + + [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0 + + +-----------------------+-------------------------------------------------+ + | Property | Value | + +-----------------------+-------------------------------------------------+ + | action | none | + +-----------------------+-------------------------------------------------+ + | administrative | locked | + +-----------------------+-------------------------------------------------+ + | availability | online | + +-----------------------+-------------------------------------------------+ + | bm_ip | None | + +-----------------------+-------------------------------------------------+ + | bm_type | none | + +-----------------------+-------------------------------------------------+ + | bm_username | None | + +-----------------------+-------------------------------------------------+ + | boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 | + +-----------------------+-------------------------------------------------+ + | capabilities | {u'stor_function': u'monitor'} | + +-----------------------+-------------------------------------------------+ + | clock_synchronization | ntp | + +-----------------------+-------------------------------------------------+ + | config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea | + +-----------------------+-------------------------------------------------+ + | config_status | Config out-of-date | + +-----------------------+-------------------------------------------------+ + | config_target | bdaf20c7-c032-4aa2-839d-93da92051bea | + +-----------------------+-------------------------------------------------+ + | console | tty0 | + +-----------------------+-------------------------------------------------+ + | created_at | 2020-06-03T08:20:59.372602+00:00 | + +-----------------------+-------------------------------------------------+ + | hostname | controller-0 | + +-----------------------+-------------------------------------------------+ + | id | 1 | + +-----------------------+-------------------------------------------------+ + | install_output | graphical | + +-----------------------+-------------------------------------------------+ + | install_state | None | + +-----------------------+-------------------------------------------------+ + | install_state_info | None | + +-----------------------+-------------------------------------------------+ + | inv_state | inventoried | + +-----------------------+-------------------------------------------------+ + | invprovision | provisioning | + +-----------------------+-------------------------------------------------+ + | location | {} | + +-----------------------+-------------------------------------------------+ + | mgmt_ip | 192.168.204.2 | + +-----------------------+-------------------------------------------------+ + | mgmt_mac | 00:00:00:00:00:00 | + +-----------------------+-------------------------------------------------+ + | operational | disabled | + +-----------------------+-------------------------------------------------+ + | personality | controller | + +-----------------------+-------------------------------------------------+ + | reserved | False | + +-----------------------+-------------------------------------------------+ + | rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 | + +-----------------------+-------------------------------------------------+ + | serialid | None | + +-----------------------+-------------------------------------------------+ + | software_load | 19.12 | + +-----------------------+-------------------------------------------------+ + | subfunction_avail | online | + +-----------------------+-------------------------------------------------+ + | subfunction_oper | disabled | + +-----------------------+-------------------------------------------------+ + | subfunctions | controller,worker | + +-----------------------+-------------------------------------------------+ + | task | Unlocking | + +-----------------------+-------------------------------------------------+ + | tboot | false | + +-----------------------+-------------------------------------------------+ + | ttys_dcd | None | + +-----------------------+-------------------------------------------------+ + | updated_at | 2020-06-04T02:04:59.514854+00:00 | + +-----------------------+-------------------------------------------------+ + | uptime | 49475 | + +-----------------------+-------------------------------------------------+ + | uuid | 08c23843-afa4-4b80-a285-be126365276e | + +-----------------------+-------------------------------------------------+ + | vim_progress_status | None | + +-----------------------+-------------------------------------------------+ + +3. Simple use case for sriov +```````````````````````````` + +3.1 After controller-0 is rebooted and up running, download the DPDK +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' - # Check the pod for dashboard - root@oran-aio:~# kubectl get pods --all-namespaces |grep dashboard - kube-system kubernetes-dashboard-5b67bf4d5f-ghg4f 1/1 Running 0 64s - -Access the dashboard UI in a web browser with the url: https://:30443 +:: -- For detail usage, please refer to `Doc for dashboard`_ + [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt + [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz + Password: + --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz + Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ... + Connecting to fast.dpdk.org|151.101.2.49|:443... connected. + + HTTP request sent, awaiting response... 200 OK + Length: 10251680 (9.8M) [application/octet-stream] + Saving to: ‘dpdk-17.11.10.tar.xz’ + + dpdk-17.11.10.tar.xz 100% + [========================================================================================>] 9.78M + 1.48MB/s in 6.8s -.. _`Doc for dashboard`: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/ + 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680] -3.9 Deploy Multus-CNI -''''''''''''''''''''' + sudo tar xvf dpdk-17.11.10.tar.xz -:: + sudo ln -s dpdk-stable-17.11.10 dpdk-stable - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/multus-cni/multus-daemonset.yml +3.2 Prepare the yaml file for the network assignment container +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -Verify that the multus-cni is up and running +The following the exmaple of the yaml file: :: - root@oran-aio:~# kubectl get pods --all-namespaces | grep -i multus - kube-system kube-multus-ds-amd64-hjpk4 1/1 Running 0 7m34s - -- For further validating, please refer to the `Multus-CNI quick start`_ - -.. _`Multus-CNI quick start`: https://github.com/intel/multus-cni/blob/master/doc/quickstart.md - -3.10 Deploy NFD (node-feature-discovery) -'''''''''''''''''''''''''''''''''''''''' + [sysadmin@controller-0 sriov(keystone_admin)]$ cat < netdef-data-dpdk.yaml + > apiVersion: "k8s.cni.cncf.io/v1" + > kind: NetworkAttachmentDefinition + > metadata: + > name: sriov-data-dpdk-0 + > annotations: + > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0 + > spec: + > config: '{ + > "type": "sriov", + > "name": "sriov-data-dpdk-0" + > }' + > + > --- + > apiVersion: "k8s.cni.cncf.io/v1" + > kind: NetworkAttachmentDefinition + > metadata: + > name: sriov-data-dpdk-1 + > annotations: + > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1 + > spec: + > config: '{ + > "type": "sriov", + > "name": "sriov-data-dpdk-1" + > }' + > EOF + +3.3 Run the network assignent container for the 2 VFs +''''''''''''''''''''''''''''''''''''''''''''''''''''' :: - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-master.yaml - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-worker-daemonset.yaml + [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml + networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created + networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created -Verify that nfd-master and nfd-worker are up and running +3.4 Prepare the VF container yaml file +'''''''''''''''''''''''''''''''''''''' :: - root@oran-aio:~# kubectl get pods --all-namespaces |grep nfd - default nfd-master-7v75k 1/1 Running 0 91s - default nfd-worker-xn797 1/1 Running 0 24s - -Verify that the node is labeled by nfd: + [sysadmin@controller-0 sriov(keystone_admin)]$ cat < pod-with-dpdk-vfs-0.yaml + > apiVersion: v1 + > kind: Pod + metadata: + > metadata: + > name: pod-with-dpdk-vfs-0 + > annotations: + > k8s.v1.cni.cncf.io/networks: '[ + > { "name": "sriov-data-dpdk-0" }, + { "name": "sriov-data-dpdk-1" } + > { "name": "sriov-data-dpdk-1" } + > ]' + > spec: + > restartPolicy: Never + > containers: + > - name: pod-with-dpdk-vfs-0 + > image: wrsnfv/ubuntu-dpdk-build:v0.3 + > env: + > - name: RTE_SDK + > value: "/usr/src/dpdk" + > command: + > - sleep + > - infinity + > stdin: true + > tty: true + > securityContext: + > privileged: true + > capabilities: + > add: + > - ALL + > resources: + > requests: + > cpu: 4 + > memory: 4Gi + > intel.com/pci_sriov_net_physnet0: '1' + > intel.com/pci_sriov_net_physnet1: '1' + > limits: + > cpu: 4 + > hugepages-1Gi: 2Gi + > memory: 4Gi + > intel.com/pci_sriov_net_physnet0: '1' + > intel.com/pci_sriov_net_physnet1: '1' + > volumeMounts: + > - mountPath: /mnt/huge-1048576kB + > name: hugepage + > - name: dpdk-volume + > mountPath: /usr/src/dpdk + > - name: lib-volume + > mountPath: /lib/modules + > - name: src-volume + > mountPath: /usr/src/ + > volumes: + > - name: hugepage + > emptyDir: + > medium: HugePages + > - name: dpdk-volume + > hostPath: + > path: /opt/dpdk-stable/ + > - name: lib-volume + > hostPath: + > path: /lib/modules + > - name: src-volume + > hostPath: + > path: /usr/src/ + > EOF + +3.5 Run the VF container +'''''''''''''''''''''''' + +Start the VF container. :: - root@oran-aio:~# kubectl describe nodes|grep feature.node.kubernetes - feature.node.kubernetes.io/cpu-cpuid.AESNI=true - feature.node.kubernetes.io/cpu-cpuid.AVX=true - feature.node.kubernetes.io/cpu-cpuid.AVX2=true - (...snip...) + [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml + pod/pod-with-dpdk-vfs-0 created -3.11 Deploy CMK (CPU-Manager-for-Kubernetes) -'''''''''''''''''''''''''''''''''''''''''''' + [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod + NAME READY STATUS RESTARTS AGE + pod-with-dpdk-vfs-0 1/1 Running 0 6m40s -Build the CMK docker image +Login the VF container :: - root@oran-aio:~# cd /opt/kubernetes_plugins/cpu-manager-for-kubernetes/ - root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# make + kubectl exec -it pod-with-dpdk-vfs-0 -- bash -Verify that the cmk docker images is built successfully +Build the DPDK :: - root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# docker images|grep cmk - cmk v1.3.1 3fec5f753b05 44 minutes ago 765MB - -Edit the template yaml file for your deployment: - - The template file is: /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml - - The options you may need to change: + cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build + + root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare + HOSTCC scripts/basic/fixdep + HOSTCC scripts/kconfig/conf.o + HOSTCC scripts/kconfig/confdata.o + HOSTCC scripts/kconfig/expr.o + HOSTCC scripts/kconfig/symbol.o + HOSTCC scripts/kconfig/preprocess.o + HOSTCC scripts/kconfig/zconf.lex.o + HOSTCC scripts/kconfig/zconf.tab.o + HOSTLD scripts/kconfig/conf + scripts/kconfig/conf --syncconfig Kconfig + HOSTCC arch/x86/tools/relocs_32.o + HOSTCC arch/x86/tools/relocs_64.o + HOSTCC arch/x86/tools/relocs_common.o + HOSTLD arch/x86/tools/relocs + HOSTCC scripts/genksyms/genksyms.o + YACC scripts/genksyms/parse.tab.c + HOSTCC scripts/genksyms/parse.tab.o + LEX scripts/genksyms/lex.lex.c + YACC scripts/genksyms/parse.tab.h + HOSTCC scripts/genksyms/lex.lex.o + HOSTLD scripts/genksyms/genksyms + HOSTCC scripts/bin2c + HOSTCC scripts/kallsyms + HOSTCC scripts/conmakehash + HOSTCC scripts/recordmcount + HOSTCC scripts/sortextable + HOSTCC scripts/asn1_compiler + HOSTCC scripts/sign-file + HOSTCC scripts/extract-cert + CC scripts/mod/empty.o + HOSTCC scripts/mod/mk_elfconfig + MKELF scripts/mod/elfconfig.h + HOSTCC scripts/mod/modpost.o + CC scripts/mod/devicetable-offsets.s + UPD scripts/mod/devicetable-offsets.h + HOSTCC scripts/mod/file2alias.o + HOSTCC scripts/mod/sumversion.o + HOSTLD scripts/mod/modpost + CC kernel/bounds.s + CC arch/x86/kernel/asm-offsets.s + CALL scripts/checksyscalls.sh + +Build the test_pmd application :: - # You can change the value for the following env: - env: - - name: HOST_LIST - # Change this to modify the the host list to be initialized - value: "oran-aio" - - name: NUM_EXCLUSIVE_CORES - # Change this to modify the value passed to `--num-exclusive-cores` flag - value: "4" - - name: NUM_SHARED_CORES - # Change this to modify the value passed to `--num-shared-cores` flag - value: "1" - - name: CMK_IMG - # Change his ONLY if you built the docker images with a different tag name - value: "cmk:v1.3.1" - -Or you can also refer to `CMK operator manual`_ + cd $RTE_SDK + ./usertools/dpdk-setup.sh + Option: 14 + CC config.o + CC iofwd.o + CC macfwd.o + CC macswap.o + CC flowgen.o + CC rxonly.o + CC txonly.o + CC csumonly.o + CC icmpecho.o + CC tm.o + LD testpmd + INSTALL-APP testpmd + INSTALL-MAP testpmd.map + == Build app/proc_info + CC main.o + LD dpdk-procinfo + INSTALL-APP dpdk-procinfo + INSTALL-MAP dpdk-procinfo.map + == Build app/pdump + CC main.o + LD dpdk-pdump + INSTALL-APP dpdk-pdump + INSTALL-MAP dpdk-pdump.map + == Build app/test-crypto-perf + CC main.o + CC cperf_ops.o + CC cperf_options_parsing.o + CC cperf_test_vectors.o + CC cperf_test_throughput.o + CC cperf_test_latency.o + CC cperf_test_pmd_cyclecount.o + CC cperf_test_verify.o + CC cperf_test_vector_parsing.o + CC cperf_test_common.o + LD dpdk-test-crypto-perf + INSTALL-APP dpdk-test-crypto-perf + INSTALL-MAP dpdk-test-crypto-perf.map + == Build app/test-eventdev + CC evt_main.o + CC evt_options.o + CC evt_test.o + CC parser.o + CC test_order_common.o + CC test_order_queue.o + CC test_order_atq.o + CC test_perf_common.o + CC test_perf_queue.o + CC test_perf_atq.o + LD dpdk-test-eventdev + INSTALL-APP dpdk-test-eventdev + INSTALL-MAP dpdk-test-eventdev.map + Build complete [x86_64-native-linuxapp-gcc] + Installation cannot run with T defined and DESTDIR undefined + ------------------------------------------------------------------------------ + RTE_TARGET exported as x86_64-native-linuxapp-gcc + ------------------------------------------------------------------------------ + + Press enter to continue ... + +Check the VF PCI information: -.. _`CMK operator manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/operator.md +:: + root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM + PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1 + PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0 -Depoly CMK from yaml files +Exit from pod back to host to find which VFs are assigned to this pod by check the pci address: :: - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-rbac-rules.yaml - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-serviceaccount.yaml - root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml + [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn* + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2 -Verify that the cmk cluster init completed and the pods for nodereport and webhook deployment are up and running + [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn* + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1 + lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3 -:: + [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01 + [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02 - root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# kubectl get pods --all-namespaces |grep cmk - default cmk-cluster-init-pod 0/1 Completed 0 11m - default cmk-init-install-discover-pod-oran-aio 0/2 Completed 0 10m - default cmk-reconcile-nodereport-ds-oran-aio-qbdqb 2/2 Running 0 10m - default cmk-webhook-deployment-6f9dd7dfb6-2lj2p 1/1 Running 0 10m -- For detail usage, please refer to `CMK user manual`_ +Get back to pod and run the test_pmd -.. _`CMK user manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/user.md +:: + cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/ + + taskset -p --cpu-list 1 + pid 1's current affinity list: 2,3,18,19 + + ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto- + start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth- + peer=1,"9e:fd:e6:dd:c1:01" + EAL: Detected 32 lcore(s) + EAL: No free hugepages reported in hugepages-2048kB + EAL: Probing VFIO support... + EAL: VFIO support initialized + EAL: PCI device 0000:05:11.0 on NUMA socket 0 + EAL: probe driver: 8086:10ed net_ixgbe_vf + EAL: using IOMMU type 1 (Type 1) + EAL: PCI device 0000:05:11.1 on NUMA socket 0 + EAL: probe driver: 8086:10ed net_ixgbe_vf + Auto-start selected + Ports to start sending a burst of packets first + Warning: lsc_interrupt needs to be off when using tx_first. Disabling. + USER1: create a new mbuf pool : n=155456, size=2176, socket=0 + Configuring Port 0 (socket 0) + Port 0: 9E:FD:E6:DD:C1:01 + Configuring Port 1 (socket 0) + Port 1: 9E:FD:E6:DD:C1:02 + Checking link statuses... + Port0 Link Up. speed 10000 Mbps- full-duplex + Port1 Link Up. speed 10000 Mbps- full-duplex + Done + No commandline core given, start packet forwarding + io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages + disabled + Logical Core 3 (socket 0) forwards packets on 2 streams: + RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01 + RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02 + + io packet forwarding packets/burst=32 + nb forwarding cores=1 - nb forwarding ports=2 + port 0: + CRC stripping enabled + RX queues=1 - RX desc=128 - RX free threshold=32 + RX threshold registers: pthresh=8 hthresh=8 wthresh=0 + TX queues=1 - TX desc=512 - TX free threshold=32 + TX threshold registers: pthresh=32 hthresh=0 wthresh=0 + TX RS bit threshold=32 - TXQ flags=0xf01 + port 1: + CRC stripping enabled + RX queues=1 - RX desc=128 - RX free threshold=32 + RX threshold registers: pthresh=8 hthresh=8 wthresh=0 + TX queues=1 - TX desc=512 - TX free threshold=32 + TX threshold registers: pthresh=32 hthresh=0 wthresh=0 + TX RS bit threshold=32 - TXQ flags=0xf01 + + Port statistics ==================================== + ######################## NIC statistics for port 0 ######################## + RX-packets: 56 RX-missed: 0 RX-bytes: 4096 + RX-errors: 0 + RX-nombuf: 0 + TX-packets: 64 TX-errors: 0 TX-bytes: 4096 + + Throughput (since last show) + Rx-pps: 0 + Tx-pps: 0 + ############################################################################ + + ######################## NIC statistics for port 1 ######################## + RX-packets: 432 RX-missed: 0 RX-bytes: 27712 + RX-errors: 0 + RX-nombuf: 0 + TX-packets: 461 TX-errors: 0 TX-bytes: 30080 + + Throughput (since last show) + Rx-pps: 0 + Tx-pps: 0 + ############################################################################ + + Port statistics ==================================== + ######################## NIC statistics for port 0 ######################## + RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344 + RX-errors: 0 + RX-nombuf: 0 + TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376 + + Throughput (since last show) + Rx-pps: 7068409 + Tx-pps: 7091206 + ############################################################################ + + + + + + References ---------- + +- `StarlingX`_ -- `Flannel`_ -- `Doc for dashboard`_ -- `Multus-CNI quick start`_ -- `CMK operator manual`_ -- `CMK user manual`_ - -.. _`Flannel`: https://github.com/coreos/flannel/blob/master/README.md +.. _`StarlingX`: https://docs.starlingx.io/ +