1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. Copyright (C) 2019 Wind River Systems, Inc.
16 This document describes how to install O-RAN INF image, example configuration (All-in-one Duplex)
17 for better real time performance, and example deployment of Kubernetes cluster and plugins.
19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20 and container technology.
26 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
32 Following minimum hardware requirements must be met for installation of O-RAN INF image with AIO-DX:
34 +-------------------+----------------------------------------------------------------------------+
35 | **HW Aspect** | **Requirement** |
37 +-------------------+----------------------------------------------------------------------------+
38 | **# of servers** | 2 |
39 +-------------------+----------------------------------------------------------------------------+
40 | **CPU** | Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket |
42 | | Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) |
43 +-------------------+----------------------------------------------------------------------------+
46 +-------------------+----------------------------------------------------------------------------+
47 | **Disk** | * Disk 1: 500G(It's better to be SSD) |
48 | | * Disk 2: 1 or more 500 GB for Ceph OSD |
49 +-------------------+----------------------------------------------------------------------------+
50 | **NICs** | * OAM: 1x1GE |
51 | | * Data: 1 or more x 10GE (optional) |
52 +-------------------+----------------------------------------------------------------------------+
53 | **BIOS settings** | * Hyper-Threading technology enabled |
54 | | * Virtualization technology enabled |
55 | | * VT for directed I/O enabled |
56 | | * CPU power and performance policy set to performance |
57 | | * CPU C state control disabled |
58 | | * Plug & play BMC detection disabled |
59 +-------------------+----------------------------------------------------------------------------+
61 ORAN INF E Release tested on HP ProLiant DL380p Gen8
62 ****************************************************
64 1. Installation for the first server from the O-RAN INF ISO image
65 -----------------------------------------------------------------
67 - Please see the README.md file for how to build the image.
68 - The Image is a live ISO image with CLI installer:
69 inf-image-aio-installer-intel-corei7-64.iso
71 1.1 Burn the image to the USB device
72 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
74 - Assume the the usb device is /dev/sdX here
78 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
80 1.2 Install the first server (controller-0)
81 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
83 - Reboot the target from the USB device.
85 - Select “All-in-one Graphics console” or “All-in-one Serial console
86 install” and press ENTER
88 - Start the auto installation
90 - It will reboot aotumatically after installation
92 2. Configuration and initialize the bootstrap
93 ---------------------------------------------
95 2.1 First Login with sysadmin/sysadmin and change password
96 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
98 2.2 Set OAM network before bootstrap
99 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
104 export CONTROLLER0_OAM_CIDR=128.224.210.110/24
105 export DEFAULT_OAM_GATEWAY=128.224.210.1
106 sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
107 sudo ip link set up dev $OAM_DEV
108 sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
110 2.3 Login the server through SSH with sysadmin
111 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
113 2.4 Prepare the localhost.yml for bootstrap
114 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
118 cat << EOF > localhost.yml
120 management_subnet: 192.168.18.0/24
121 management_start_address: 192.168.18.2
122 management_end_address: 192.168.18.50
123 management_gateway_address: 192.168.18.1
124 external_oam_subnet: 128.224.210.0/24
125 external_oam_gateway_address: 128.224.210.1
126 external_oam_floating_address: 128.224.210.110
127 external_oam_node_0_address: 128.224.210.111
128 external_oam_node_1_address: 128.224.210.112
131 2.5 Run the ansible bootstrap
132 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
136 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
138 After the bootstrap successfully finish, it will show as following:
142 PLAY RECAP *************************************************************************************************************
143 localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
145 2.6 Congiure controller-0
146 ~~~~~~~~~~~~~~~~~~~~~~~~~
148 Acquire admin credentials:
152 controller-0:~$ source /etc/platform/openrc
153 [sysadmin@controller-0 ~(keystone_admin)]$
155 Configure the OAM and MGMT interfaces of controller-0 and specify the
162 system host-if-modify controller-0 lo -c none
163 IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
164 for UUID in $IFNET_UUIDS; do
165 system interface-network-remove ${UUID}
168 system host-if-modify controller-0 $OAM_IF -n oam0
169 system host-if-modify controller-0 $MGMT_IF -n pxeboot0
171 system host-if-modify controller-0 oam0 -c platform
172 system interface-network-assign controller-0 oam0 oam
174 system host-if-modify controller-0 pxeboot0 -c platform
175 system interface-network-assign controller-0 pxeboot0 pxeboot
177 system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
178 system interface-network-assign controller-0 mgmt0 mgmt
180 system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
181 system interface-network-assign controller-0 cluster0 cluster-host
187 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
188 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
189 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
190 +-----------------+--------------------------------------+
192 +-----------------+--------------------------------------+
196 | imac | 00:00:00:00:00:00 |
200 | schedpolicy | None |
201 | txhashpolicy | None |
202 | uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c |
203 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
207 | created_at | 2021-11-17T00:30:45.265032+00:00 |
208 | updated_at | 2021-11-17T01:03:39.031612+00:00 |
210 | sriov_vf_driver | None |
211 +-----------------+--------------------------------------+
212 [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
213 [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
214 > system interface-network-remove ${UUID}
216 Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
217 Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
219 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
220 +-----------------+--------------------------------------+
222 +-----------------+--------------------------------------+
224 | iftype | ethernet |
225 | ports | [u'eno3'] |
226 | imac | 24:6e:96:5d:0c:b2 |
230 | schedpolicy | None |
231 | txhashpolicy | None |
232 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
233 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
237 | created_at | 2021-11-17T00:28:32.365863+00:00 |
238 | updated_at | 2021-11-17T01:03:45.090904+00:00 |
240 | sriov_vf_driver | None |
241 | accelerated | [True] |
242 +-----------------+--------------------------------------+
243 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
244 +-----------------+--------------------------------------+
246 +-----------------+--------------------------------------+
247 | ifname | pxeboot0 |
248 | iftype | ethernet |
249 | ports | [u'eno1'] |
250 | imac | 24:6e:96:5d:0c:92 |
254 | schedpolicy | None |
255 | txhashpolicy | None |
256 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
257 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
261 | created_at | 2021-11-17T00:28:32.612230+00:00 |
262 | updated_at | 2021-11-17T01:03:47.341003+00:00 |
264 | sriov_vf_driver | None |
265 | accelerated | [True] |
266 +-----------------+--------------------------------------+
267 [sysadmin@controller-0 ~(keystone_admin)]$
268 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
269 +-----------------+--------------------------------------+
271 +-----------------+--------------------------------------+
273 | iftype | ethernet |
274 | ports | [u'eno3'] |
275 | imac | 24:6e:96:5d:0c:b2 |
277 | ifclass | platform |
279 | schedpolicy | None |
280 | txhashpolicy | None |
281 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
282 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
286 | created_at | 2021-11-17T00:28:32.365863+00:00 |
287 | updated_at | 2021-11-17T01:03:49.368879+00:00 |
289 | sriov_vf_driver | None |
290 | accelerated | [True] |
291 +-----------------+--------------------------------------+
292 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
293 +--------------+--------------------------------------+
295 +--------------+--------------------------------------+
296 | hostname | controller-0 |
297 | uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
299 | network_name | oam |
300 +--------------+--------------------------------------+
301 [sysadmin@controller-0 ~(keystone_admin)]$
302 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
303 +-----------------+--------------------------------------+
305 +-----------------+--------------------------------------+
306 | ifname | pxeboot0 |
307 | iftype | ethernet |
308 | ports | [u'eno1'] |
309 | imac | 24:6e:96:5d:0c:92 |
311 | ifclass | platform |
313 | schedpolicy | None |
314 | txhashpolicy | None |
315 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
316 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
320 | created_at | 2021-11-17T00:28:32.612230+00:00 |
321 | updated_at | 2021-11-17T01:03:53.143795+00:00 |
323 | sriov_vf_driver | None |
324 | accelerated | [True] |
325 +-----------------+--------------------------------------+
326 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot
327 +--------------+--------------------------------------+
329 +--------------+--------------------------------------+
330 | hostname | controller-0 |
331 | uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
332 | ifname | pxeboot0 |
333 | network_name | pxeboot |
334 +--------------+--------------------------------------+
335 [sysadmin@controller-0 ~(keystone_admin)]$
336 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
337 +-----------------+--------------------------------------+
339 +-----------------+--------------------------------------+
343 | imac | 24:6e:96:5d:0c:92 |
347 | schedpolicy | None |
348 | txhashpolicy | None |
349 | uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
350 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
352 | uses | [u'pxeboot0'] |
354 | created_at | 2021-11-17T01:03:57.303000+00:00 |
355 | updated_at | None |
357 | sriov_vf_driver | None |
358 | accelerated | [True] |
359 +-----------------+--------------------------------------+
360 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
361 +--------------+--------------------------------------+
363 +--------------+--------------------------------------+
364 | hostname | controller-0 |
365 | uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
367 | network_name | mgmt |
368 +--------------+--------------------------------------+
369 [sysadmin@controller-0 ~(keystone_admin)]$
370 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
371 +-----------------+--------------------------------------+
373 +-----------------+--------------------------------------+
374 | ifname | cluster0 |
377 | imac | 24:6e:96:5d:0c:92 |
381 | schedpolicy | None |
382 | txhashpolicy | None |
383 | uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
384 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
386 | uses | [u'pxeboot0'] |
388 | created_at | 2021-11-17T01:04:02.613518+00:00 |
389 | updated_at | None |
391 | sriov_vf_driver | None |
392 | accelerated | [True] |
393 +-----------------+--------------------------------------+
394 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
395 +--------------+--------------------------------------+
397 +--------------+--------------------------------------+
398 | hostname | controller-0 |
399 | uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
400 | ifname | cluster0 |
401 | network_name | cluster-host |
402 +--------------+--------------------------------------+
403 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
404 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
405 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
406 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
407 | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
408 | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
409 | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
410 | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
411 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
413 Configure NTP Servers for network time synchronization:
417 system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
423 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
424 +--------------+--------------------------------------+
426 +--------------+--------------------------------------+
427 | uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
428 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
429 | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
430 | created_at | 2021-11-17T00:27:23.529571+00:00 |
431 | updated_at | None |
432 +--------------+--------------------------------------+
434 Add an OSD on controller-0 for Ceph:
438 system host-disk-list controller-0
439 system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
440 system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
441 system host-stor-list controller-0
447 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
448 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
449 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
450 | | de | num | type | gib | gib | | | |
451 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
452 | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
453 | | | | | 75 | | | | |
455 | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
456 | | | | | 25 | | | | |
458 | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
459 | | | | | 25 | | | | |
461 | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
462 | | | | | 25 | | | | |
464 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
465 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
466 +------------------+-------------------------------------------------------+
468 +------------------+-------------------------------------------------------+
471 | state | configuring-on-unlock |
472 | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
473 | journal_size_gib | 1024 |
474 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
475 | journal_node | /dev/sdb2 |
476 | uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
477 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
478 | idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 |
479 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
480 | tier_name | storage |
481 | created_at | 2021-11-17T01:05:04.063823+00:00 |
482 | updated_at | None |
483 +------------------+-------------------------------------------------------+
484 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
485 +------------------+-------------------------------------------------------+
487 +------------------+-------------------------------------------------------+
490 | state | configuring-on-unlock |
491 | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
492 | journal_size_gib | 1024 |
493 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
494 | journal_node | /dev/sdc2 |
495 | uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
496 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
497 | idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 |
498 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
499 | tier_name | storage |
500 | created_at | 2021-11-17T01:05:06.939798+00:00 |
501 | updated_at | None |
502 +------------------+-------------------------------------------------------+
503 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
504 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
505 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
506 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
507 | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
508 | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
509 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
511 2.7 Unlock controller-0
512 ~~~~~~~~~~~~~~~~~~~~~~~
516 system host-unlock controller-0
522 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
523 +-----------------------+-------------------------------------------------+
525 +-----------------------+-------------------------------------------------+
527 | administrative | locked |
528 | availability | online |
531 | bm_username | None |
532 | boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
533 | capabilities | {u'stor_function': u'monitor'} |
534 | clock_synchronization | ntp |
535 | config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 |
536 | config_status | None |
537 | config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 |
538 | console | ttyS0,115200 |
539 | created_at | 2021-11-17T00:28:01.983673+00:00 |
540 | hostname | controller-0 |
542 | install_output | graphical |
543 | install_state | None |
544 | install_state_info | None |
545 | inv_state | inventoried |
546 | invprovision | provisioning |
548 | mgmt_ip | 192.168.18.3 |
549 | mgmt_mac | 24:6e:96:5d:0c:92 |
550 | operational | disabled |
551 | personality | controller |
553 | rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
555 | software_load | 21.05 |
556 | subfunction_avail | online |
557 | subfunction_oper | disabled |
558 | subfunctions | controller,worker,lowlatency |
562 | updated_at | 2021-11-17T01:05:07.015414+00:00 |
564 | uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
565 | vim_progress_status | None |
566 +-----------------------+-------------------------------------------------+
568 Controller-0 will reboot to apply configuration changes and come into
569 service. This can take 5-10 minutes, depending on the performance of the
572 Once the controller comes back up, check the status of controller-0
576 controller-0:~$ source /etc/platform/openrc
577 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
578 +----+--------------+-------------+----------------+-------------+--------------+
579 | id | hostname | personality | administrative | operational | availability |
580 +----+--------------+-------------+----------------+-------------+--------------+
581 | 1 | controller-0 | controller | unlocked | enabled | available |
582 +----+--------------+-------------+----------------+-------------+--------------+
584 2. Installation for the second server (controller-1)
585 ----------------------------------------------------
587 2.1 Power on the controller-1 server and force it to network boot
588 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
590 2.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node
591 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
593 2.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)
594 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
598 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
599 +----+--------------+-------------+----------------+-------------+--------------+
600 | id | hostname | personality | administrative | operational | availability |
601 +----+--------------+-------------+----------------+-------------+--------------+
602 | 1 | controller-0 | controller | unlocked | enabled | degraded |
603 | 2 | None | None | locked | disabled | offline |
604 +----+--------------+-------------+----------------+-------------+--------------+
606 2.4 Using the host id, set the personality of this host to 'controller’:
607 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
611 [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
612 +-----------------------+--------------------------------------+
614 +-----------------------+--------------------------------------+
616 | administrative | locked |
617 | availability | offline |
620 | bm_username | None |
621 | boot_device | /dev/sda |
622 | capabilities | {} |
623 | clock_synchronization | ntp |
624 | config_applied | None |
625 | config_status | None |
626 | config_target | None |
627 | console | ttyS0,115200 |
628 | created_at | 2021-11-17T10:17:44.387813+00:00 |
629 | hostname | controller-1 |
631 | install_output | text |
632 | install_state | None |
633 | install_state_info | None |
635 | invprovision | None |
637 | mgmt_ip | 192.168.18.4 |
638 | mgmt_mac | 24:6e:96:5d:38:ee |
639 | operational | disabled |
640 | personality | controller |
642 | rootfs_device | /dev/sda |
644 | software_load | 21.05 |
645 | subfunction_avail | not-installed |
646 | subfunction_oper | disabled |
647 | subfunctions | controller,worker,lowlatency |
651 | updated_at | None |
653 | uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 |
654 | vim_progress_status | None |
655 +-----------------------+--------------------------------------+
657 2.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
658 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
660 This can take 5-10 minutes, depending on the performance of the host
665 [root@controller-0 hieradata(keystone_admin)]$ system host-list
666 +----+--------------+-------------+----------------+-------------+--------------+
667 | id | hostname | personality | administrative | operational | availability |
668 +----+--------------+-------------+----------------+-------------+--------------+
669 | 1 | controller-0 | controller | unlocked | enabled | available |
670 | 2 | controller-1 | controller | locked | disabled | online |
671 +----+--------------+-------------+----------------+-------------+--------------+
673 2.6 Configure controller-1
674 ~~~~~~~~~~~~~~~~~~~~~~~~~~
680 system host-if-modify controller-1 $OAM_IF -n oam0
681 system host-if-modify controller-1 oam0 -c platform
682 system interface-network-assign controller-1 oam0 oam
684 system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
685 system interface-network-assign controller-1 cluster0 cluster-host
687 system host-if-list controller-1
689 system host-disk-list controller-1
690 system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
691 system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
692 system host-stor-list controller-1
698 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
699 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
700 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
701 +-----------------+--------------------------------------+
703 +-----------------+--------------------------------------+
705 | iftype | ethernet |
706 | ports | [u'eno3'] |
707 | imac | 24:6e:96:5d:39:0e |
711 | schedpolicy | None |
712 | txhashpolicy | None |
713 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
714 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
718 | created_at | 2021-11-22T14:01:32.365863+00:00 |
719 | updated_at | 2021-11-22T15:04:45.090904+00:00 |
721 | sriov_vf_driver | None |
722 | accelerated | [True] |
723 +-----------------+--------------------------------------+
724 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
725 +-----------------+--------------------------------------+
727 +-----------------+--------------------------------------+
729 | iftype | ethernet |
730 | ports | [u'eno3'] |
731 | imac | 24:6e:96:5d:39:0e |
733 | ifclass | platform |
735 | schedpolicy | None |
736 | txhashpolicy | None |
737 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
738 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
742 | created_at | 2021-11-22T14:05:16.052229+00:00 |
743 | updated_at | 2021-11-22T15:08:35.324634+00:00 |
745 | sriov_vf_driver | None |
746 | accelerated | [True] |
747 +-----------------+--------------------------------------+
748 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
749 +--------------+--------------------------------------+
751 +--------------+--------------------------------------+
752 | hostname | controller-1 |
753 | uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
755 | network_name | oam |
756 +--------------+--------------------------------------+
757 [sysadmin@controller-0 ~(keystone_admin)]$
759 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
760 +-----------------+--------------------------------------+
762 +-----------------+--------------------------------------+
763 | ifname | cluster0 |
766 | imac | 24:6e:96:5d:38:ee |
770 | schedpolicy | None |
771 | txhashpolicy | None |
772 | uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 |
773 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
775 | uses | [u'pxeboot0'] |
777 | created_at | 2021-11-22T15:08:43.932209+00:00 |
778 | updated_at | None |
780 | sriov_vf_driver | None |
781 | accelerated | [True] |
782 +-----------------+--------------------------------------+
784 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
785 +--------------+--------------------------------------+
787 +--------------+--------------------------------------+
788 | hostname | controller-1 |
789 | uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
790 | ifname | cluster0 |
791 | network_name | cluster-host |
792 +--------------+--------------------------------------+
793 [sysadmin@controller-0 ~(keystone_admin)]$
794 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
795 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
796 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
797 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
798 | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
799 | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
800 | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
801 | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
802 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
803 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
804 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
805 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
806 | | de | num | type | gib | gib | | | |
807 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
808 | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
809 | | | | | 75 | | | | |
811 | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
812 | | | | | 25 | | | | |
814 | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
815 | | | | | 25 | | | | |
817 | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
818 | | | | | 25 | | | | |
820 | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 |
821 | | | | | 57 | | | | |
823 | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 |
824 | | | | | 57 | | | | |
826 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
827 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
828 +------------------+-------------------------------------------------------+
830 +------------------+-------------------------------------------------------+
833 | state | configuring-on-unlock |
834 | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
835 | journal_size_gib | 1024 |
836 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
837 | journal_node | /dev/sdb2 |
838 | uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
839 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
840 | idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 |
841 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
842 | tier_name | storage |
843 | created_at | 2021-11-22T15:11:55.641193+00:00 |
844 | updated_at | None |
845 +------------------+-------------------------------------------------------+
847 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
848 +------------------+-------------------------------------------------------+
850 +------------------+-------------------------------------------------------+
853 | state | configuring-on-unlock |
854 | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
855 | journal_size_gib | 1024 |
856 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
857 | journal_node | /dev/sdc2 |
858 | uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
859 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
860 | idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae |
861 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
862 | tier_name | storage |
863 | created_at | 2021-11-22T15:12:04.274839+00:00 |
864 | updated_at | None |
865 +------------------+-------------------------------------------------------+
867 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
868 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
869 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
870 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
871 | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
872 | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
873 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
875 2.7 Unlock controller-1
876 ~~~~~~~~~~~~~~~~~~~~~~~
878 Unlock controller-1 in order to bring it into service:
882 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
883 +-----------------------+--------------------------------------+
885 +-----------------------+--------------------------------------+
887 | administrative | locked |
888 | availability | online |
891 | bm_username | None |
892 | boot_device | /dev/sda |
893 | capabilities | {u'stor_function': u'monitor'} |
894 | clock_synchronization | ntp |
895 | config_applied | None |
896 | config_status | Config out-of-date |
897 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
898 | console | ttyS0,115200 |
899 | created_at | 2021-11-22T12:58:11.630526+00:00 |
900 | hostname | controller-1 |
902 | install_output | text |
903 | install_state | None |
904 | install_state_info | None |
905 | inv_state | inventoried |
906 | invprovision | unprovisioned |
908 | mgmt_ip | 192.168.18.4 |
909 | mgmt_mac | 24:6e:96:5d:38:ee |
910 | operational | disabled |
911 | personality | controller |
913 | rootfs_device | /dev/sda |
915 | software_load | 21.05 |
916 | subfunction_avail | online |
917 | subfunction_oper | disabled |
918 | subfunctions | controller,worker,lowlatency |
922 | updated_at | 2021-11-22T15:13:09.716324+00:00 |
924 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
925 | vim_progress_status | None |
926 +-----------------------+--------------------------------------+
928 Controller-1 will reboot in order to apply configuration changes and
929 come into service. This can take 5-10 minutes, depending on the
930 performance of the host machine.
934 [root@controller-0 hieradata(keystone_admin)]$ system host-list
935 +----+--------------+-------------+----------------+-------------+--------------+
936 | id | hostname | personality | administrative | operational | availability |
937 +----+--------------+-------------+----------------+-------------+--------------+
938 | 1 | controller-0 | controller | unlocked | enabled | available |
939 | 2 | controller-1 | controller | unlocked | enabled | available |
940 +----+--------------+-------------+----------------+-------------+--------------+
942 [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
943 +-----------------------+-----------------------------------------------------------------------+
945 +-----------------------+-----------------------------------------------------------------------+
947 | administrative | unlocked |
948 | availability | available |
951 | bm_username | None |
952 | boot_device | /dev/sda |
953 | capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
954 | clock_synchronization | ntp |
955 | config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac |
956 | config_status | None |
957 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
958 | console | ttyS0,115200 |
959 | created_at | 2021-11-22T12:58:11.630526+00:00 |
960 | hostname | controller-1 |
962 | install_output | text |
963 | install_state | None |
964 | install_state_info | None |
965 | inv_state | inventoried |
966 | invprovision | provisioned |
968 | mgmt_ip | 192.168.18.4 |
969 | mgmt_mac | 24:6e:96:5d:38:ee |
970 | operational | enabled |
971 | personality | controller |
973 | rootfs_device | /dev/sda |
975 | software_load | 21.05 |
976 | subfunction_avail | available |
977 | subfunction_oper | enabled |
978 | subfunctions | controller,worker,lowlatency |
982 | updated_at | 2021-11-22T23:59:07.787759+00:00 |
984 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
985 | vim_progress_status | services-enabled |
986 +-----------------------+-----------------------------------------------------------------------+
989 3. Simple use case for sriov
990 ````````````````````````````
992 3.1 After controller-0 is rebooted and up running, download the DPDK
993 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
997 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
998 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
1000 --2021-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
1001 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
1002 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
1004 HTTP request sent, awaiting response... 200 OK
1005 Length: 10251680 (9.8M) [application/octet-stream]
1006 Saving to: ‘dpdk-17.11.10.tar.xz’
1008 dpdk-17.11.10.tar.xz 100%
1009 [========================================================================================>] 9.78M
1012 2021-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
1014 sudo tar xvf dpdk-17.11.10.tar.xz
1016 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
1018 3.2 Prepare the yaml file for the network assignment container
1019 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1021 The following the exmaple of the yaml file:
1025 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
1026 > apiVersion: "k8s.cni.cncf.io/v1"
1027 > kind: NetworkAttachmentDefinition
1029 > name: sriov-data-dpdk-0
1031 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
1035 > "name": "sriov-data-dpdk-0"
1039 > apiVersion: "k8s.cni.cncf.io/v1"
1040 > kind: NetworkAttachmentDefinition
1042 > name: sriov-data-dpdk-1
1044 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
1048 > "name": "sriov-data-dpdk-1"
1052 3.3 Run the network assignent container for the 2 VFs
1053 '''''''''''''''''''''''''''''''''''''''''''''''''''''
1057 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
1058 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
1059 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
1061 3.4 Prepare the VF container yaml file
1062 ''''''''''''''''''''''''''''''''''''''
1066 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
1071 > name: pod-with-dpdk-vfs-0
1073 > k8s.v1.cni.cncf.io/networks: '[
1074 > { "name": "sriov-data-dpdk-0" },
1075 { "name": "sriov-data-dpdk-1" }
1076 > { "name": "sriov-data-dpdk-1" }
1079 > restartPolicy: Never
1081 > - name: pod-with-dpdk-vfs-0
1082 > image: wrsnfv/ubuntu-dpdk-build:v0.3
1085 > value: "/usr/src/dpdk"
1100 > intel.com/pci_sriov_net_physnet0: '1'
1101 > intel.com/pci_sriov_net_physnet1: '1'
1104 > hugepages-1Gi: 2Gi
1106 > intel.com/pci_sriov_net_physnet0: '1'
1107 > intel.com/pci_sriov_net_physnet1: '1'
1109 > - mountPath: /mnt/huge-1048576kB
1111 > - name: dpdk-volume
1112 > mountPath: /usr/src/dpdk
1113 > - name: lib-volume
1114 > mountPath: /lib/modules
1115 > - name: src-volume
1116 > mountPath: /usr/src/
1121 > - name: dpdk-volume
1123 > path: /opt/dpdk-stable/
1124 > - name: lib-volume
1126 > path: /lib/modules
1127 > - name: src-volume
1132 3.5 Run the VF container
1133 ''''''''''''''''''''''''
1135 Start the VF container.
1139 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
1140 pod/pod-with-dpdk-vfs-0 created
1142 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
1143 NAME READY STATUS RESTARTS AGE
1144 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
1146 Login the VF container
1150 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
1156 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
1158 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
1159 HOSTCC scripts/basic/fixdep
1160 HOSTCC scripts/kconfig/conf.o
1161 HOSTCC scripts/kconfig/confdata.o
1162 HOSTCC scripts/kconfig/expr.o
1163 HOSTCC scripts/kconfig/symbol.o
1164 HOSTCC scripts/kconfig/preprocess.o
1165 HOSTCC scripts/kconfig/zconf.lex.o
1166 HOSTCC scripts/kconfig/zconf.tab.o
1167 HOSTLD scripts/kconfig/conf
1168 scripts/kconfig/conf --syncconfig Kconfig
1169 HOSTCC arch/x86/tools/relocs_32.o
1170 HOSTCC arch/x86/tools/relocs_64.o
1171 HOSTCC arch/x86/tools/relocs_common.o
1172 HOSTLD arch/x86/tools/relocs
1173 HOSTCC scripts/genksyms/genksyms.o
1174 YACC scripts/genksyms/parse.tab.c
1175 HOSTCC scripts/genksyms/parse.tab.o
1176 LEX scripts/genksyms/lex.lex.c
1177 YACC scripts/genksyms/parse.tab.h
1178 HOSTCC scripts/genksyms/lex.lex.o
1179 HOSTLD scripts/genksyms/genksyms
1180 HOSTCC scripts/bin2c
1181 HOSTCC scripts/kallsyms
1182 HOSTCC scripts/conmakehash
1183 HOSTCC scripts/recordmcount
1184 HOSTCC scripts/sortextable
1185 HOSTCC scripts/asn1_compiler
1186 HOSTCC scripts/sign-file
1187 HOSTCC scripts/extract-cert
1188 CC scripts/mod/empty.o
1189 HOSTCC scripts/mod/mk_elfconfig
1190 MKELF scripts/mod/elfconfig.h
1191 HOSTCC scripts/mod/modpost.o
1192 CC scripts/mod/devicetable-offsets.s
1193 UPD scripts/mod/devicetable-offsets.h
1194 HOSTCC scripts/mod/file2alias.o
1195 HOSTCC scripts/mod/sumversion.o
1196 HOSTLD scripts/mod/modpost
1198 CC arch/x86/kernel/asm-offsets.s
1199 CALL scripts/checksyscalls.sh
1201 Build the test_pmd application
1206 ./usertools/dpdk-setup.sh
1220 INSTALL-MAP testpmd.map
1221 == Build app/proc_info
1224 INSTALL-APP dpdk-procinfo
1225 INSTALL-MAP dpdk-procinfo.map
1229 INSTALL-APP dpdk-pdump
1230 INSTALL-MAP dpdk-pdump.map
1231 == Build app/test-crypto-perf
1234 CC cperf_options_parsing.o
1235 CC cperf_test_vectors.o
1236 CC cperf_test_throughput.o
1237 CC cperf_test_latency.o
1238 CC cperf_test_pmd_cyclecount.o
1239 CC cperf_test_verify.o
1240 CC cperf_test_vector_parsing.o
1241 CC cperf_test_common.o
1242 LD dpdk-test-crypto-perf
1243 INSTALL-APP dpdk-test-crypto-perf
1244 INSTALL-MAP dpdk-test-crypto-perf.map
1245 == Build app/test-eventdev
1250 CC test_order_common.o
1251 CC test_order_queue.o
1253 CC test_perf_common.o
1254 CC test_perf_queue.o
1256 LD dpdk-test-eventdev
1257 INSTALL-APP dpdk-test-eventdev
1258 INSTALL-MAP dpdk-test-eventdev.map
1259 Build complete [x86_64-native-linuxapp-gcc]
1260 Installation cannot run with T defined and DESTDIR undefined
1261 ------------------------------------------------------------------------------
1262 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1263 ------------------------------------------------------------------------------
1265 Press enter to continue ...
1267 Check the VF PCI information:
1271 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1272 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1273 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1275 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
1279 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1280 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1281 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1282 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1283 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1284 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1285 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1287 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1288 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1289 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1290 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1291 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1292 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1293 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1295 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1296 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1299 Get back to pod and run the test_pmd
1303 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1305 taskset -p --cpu-list 1
1306 pid 1's current affinity list: 2,3,18,19
1308 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1309 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1310 peer=1,"9e:fd:e6:dd:c1:01"
1311 EAL: Detected 32 lcore(s)
1312 EAL: No free hugepages reported in hugepages-2048kB
1313 EAL: Probing VFIO support...
1314 EAL: VFIO support initialized
1315 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1316 EAL: probe driver: 8086:10ed net_ixgbe_vf
1317 EAL: using IOMMU type 1 (Type 1)
1318 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1319 EAL: probe driver: 8086:10ed net_ixgbe_vf
1321 Ports to start sending a burst of packets first
1322 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1323 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1324 Configuring Port 0 (socket 0)
1325 Port 0: 9E:FD:E6:DD:C1:01
1326 Configuring Port 1 (socket 0)
1327 Port 1: 9E:FD:E6:DD:C1:02
1328 Checking link statuses...
1329 Port0 Link Up. speed 10000 Mbps- full-duplex
1330 Port1 Link Up. speed 10000 Mbps- full-duplex
1332 No commandline core given, start packet forwarding
1333 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1335 Logical Core 3 (socket 0) forwards packets on 2 streams:
1336 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1337 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1339 io packet forwarding packets/burst=32
1340 nb forwarding cores=1 - nb forwarding ports=2
1342 CRC stripping enabled
1343 RX queues=1 - RX desc=128 - RX free threshold=32
1344 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1345 TX queues=1 - TX desc=512 - TX free threshold=32
1346 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1347 TX RS bit threshold=32 - TXQ flags=0xf01
1349 CRC stripping enabled
1350 RX queues=1 - RX desc=128 - RX free threshold=32
1351 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1352 TX queues=1 - TX desc=512 - TX free threshold=32
1353 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1354 TX RS bit threshold=32 - TXQ flags=0xf01
1356 Port statistics ====================================
1357 ######################## NIC statistics for port 0 ########################
1358 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1361 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1363 Throughput (since last show)
1366 ############################################################################
1368 ######################## NIC statistics for port 1 ########################
1369 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1372 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1374 Throughput (since last show)
1377 ############################################################################
1379 Port statistics ====================================
1380 ######################## NIC statistics for port 0 ########################
1381 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1384 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1386 Throughput (since last show)
1389 ############################################################################
1401 .. _`StarlingX`: https://docs.starlingx.io/