1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. Copyright (C) 2019 Wind River Systems, Inc.
16 This document describes how to install O-RAN INF image, example configuration for better
17 real time performance, and example deployment of Kubernetes cluster and plugins.
19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20 and container technology.
24 +--------------------+--------------------+--------------------+--------------------+
25 | **Date** | **Ver.** | **Author** | **Comment** |
27 +--------------------+--------------------+--------------------+--------------------+
28 | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version |
30 +--------------------+--------------------+--------------------+--------------------+
31 | 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version |
33 +--------------------+--------------------+--------------------+--------------------+
34 | 2020-11-23 | 3.0.0 (Cherry) | Xiaohua Zhang | Cherry version |
36 +--------------------+--------------------+--------------------+--------------------+
42 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
48 Following minimum hardware requirements must be met for installation of O-RAN INF image:
50 +--------------------+----------------------------------------------------+
51 | **HW Aspect** | **Requirement** |
53 +--------------------+----------------------------------------------------+
54 | **# of servers** | 1 |
55 +--------------------+----------------------------------------------------+
58 +--------------------+----------------------------------------------------+
61 +--------------------+----------------------------------------------------+
62 | **Disk** | 500G * 2 |
64 +--------------------+----------------------------------------------------+
67 +--------------------+----------------------------------------------------+
69 If want to enable the HA mode which is available in Cherry release, 2 servers are needed.
72 ORAN INF Cherry test on Dell R740
73 =================================
75 1. Installation for the first server from the O-RAN INF ISO image
76 -----------------------------------------------------------------
78 - Please see the README.md file for how to build the image.
79 - The Image is a live ISO image with CLI installer:
80 inf-image-aio-installer-intel-corei7-64.iso
82 1.1 Burn the image to the USB device
83 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
85 - Assume the the usb device is /dev/sdX here
89 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
91 1.2 Install the first Dell-R740 server (controller-0)
92 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94 - Reboot the target from the USB device.
96 - Select “All-in-one Graphics console” or “All-in-one Serial console
97 install” and press ENTER
99 - Start the auto installation
101 - It will reboot aotumatically after installation
103 2. Configuration and initialize the bootstrap
104 ---------------------------------------------
106 2.1 First Login with sysadmin/sysadmin and change password
107 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
109 2.2 Set OAM network before bootstrap
110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
115 export CONTROLLER0_OAM_CIDR=128.224.210.110/24
116 export DEFAULT_OAM_GATEWAY=128.224.210.1
117 sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
118 sudo ip link set up dev $OAM_DEV
119 sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
121 2.3 Login the server through SSH with sysadmin
122 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
124 2.4 Prepare the localhost.yml for bootstrap
125 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129 cat << EOF > localhost.yml
131 management_subnet: 192.168.18.0/24
132 management_start_address: 192.168.18.2
133 management_end_address: 192.168.18.50
134 management_gateway_address: 192.168.18.1
135 external_oam_subnet: 128.224.210.0/24
136 external_oam_gateway_address: 128.224.210.1
137 external_oam_floating_address: 128.224.210.110
138 external_oam_node_0_address: 128.224.210.111
139 external_oam_node_1_address: 128.224.210.112
142 2.5 Run the ansible bootstrap
143 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
147 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
149 After the bootstrap successfully finish, it will show as following:
153 PLAY RECAP *************************************************************************************************************
154 localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
156 2.6 Congiure controller-0
157 ~~~~~~~~~~~~~~~~~~~~~~~~~
159 Acquire admin credentials:
163 controller-0:~$ source /etc/platform/openrc
164 [sysadmin@controller-0 ~(keystone_admin)]$
166 Configure the OAM and MGMT interfaces of controller-0 and specify the
173 system host-if-modify controller-0 lo -c none
174 IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
175 for UUID in $IFNET_UUIDS; do
176 system interface-network-remove ${UUID}
179 system host-if-modify controller-0 $OAM_IF -n oam0
180 system host-if-modify controller-0 $MGMT_IF -n pxeboot0
182 system host-if-modify controller-0 oam0 -c platform
183 system interface-network-assign controller-0 oam0 oam
185 system host-if-modify controller-0 pxeboot0 -c platform
186 system interface-network-assign controller-0 pxeboot0 pxeboot
188 system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
189 system interface-network-assign controller-0 mgmt0 mgmt
191 system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
192 system interface-network-assign controller-0 cluster0 cluster-host
198 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
199 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
200 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
201 +-----------------+--------------------------------------+
203 +-----------------+--------------------------------------+
207 | imac | 00:00:00:00:00:00 |
211 | schedpolicy | None |
212 | txhashpolicy | None |
213 | uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c |
214 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
218 | created_at | 2020-11-17T00:30:45.265032+00:00 |
219 | updated_at | 2020-11-17T01:03:39.031612+00:00 |
221 | sriov_vf_driver | None |
222 +-----------------+--------------------------------------+
223 [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
224 [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
225 > system interface-network-remove ${UUID}
227 Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
228 Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
230 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
231 +-----------------+--------------------------------------+
233 +-----------------+--------------------------------------+
235 | iftype | ethernet |
236 | ports | [u'eno3'] |
237 | imac | 24:6e:96:5d:0c:b2 |
241 | schedpolicy | None |
242 | txhashpolicy | None |
243 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
244 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
248 | created_at | 2020-11-17T00:28:32.365863+00:00 |
249 | updated_at | 2020-11-17T01:03:45.090904+00:00 |
251 | sriov_vf_driver | None |
252 | accelerated | [True] |
253 +-----------------+--------------------------------------+
254 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
255 +-----------------+--------------------------------------+
257 +-----------------+--------------------------------------+
258 | ifname | pxeboot0 |
259 | iftype | ethernet |
260 | ports | [u'eno1'] |
261 | imac | 24:6e:96:5d:0c:92 |
265 | schedpolicy | None |
266 | txhashpolicy | None |
267 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
268 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
272 | created_at | 2020-11-17T00:28:32.612230+00:00 |
273 | updated_at | 2020-11-17T01:03:47.341003+00:00 |
275 | sriov_vf_driver | None |
276 | accelerated | [True] |
277 +-----------------+--------------------------------------+
278 [sysadmin@controller-0 ~(keystone_admin)]$
279 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
280 +-----------------+--------------------------------------+
282 +-----------------+--------------------------------------+
284 | iftype | ethernet |
285 | ports | [u'eno3'] |
286 | imac | 24:6e:96:5d:0c:b2 |
288 | ifclass | platform |
290 | schedpolicy | None |
291 | txhashpolicy | None |
292 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
293 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
297 | created_at | 2020-11-17T00:28:32.365863+00:00 |
298 | updated_at | 2020-11-17T01:03:49.368879+00:00 |
300 | sriov_vf_driver | None |
301 | accelerated | [True] |
302 +-----------------+--------------------------------------+
303 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
304 +--------------+--------------------------------------+
306 +--------------+--------------------------------------+
307 | hostname | controller-0 |
308 | uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
310 | network_name | oam |
311 +--------------+--------------------------------------+
312 [sysadmin@controller-0 ~(keystone_admin)]$
313 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
314 +-----------------+--------------------------------------+
316 +-----------------+--------------------------------------+
317 | ifname | pxeboot0 |
318 | iftype | ethernet |
319 | ports | [u'eno1'] |
320 | imac | 24:6e:96:5d:0c:92 |
322 | ifclass | platform |
324 | schedpolicy | None |
325 | txhashpolicy | None |
326 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
327 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
331 | created_at | 2020-11-17T00:28:32.612230+00:00 |
332 | updated_at | 2020-11-17T01:03:53.143795+00:00 |
334 | sriov_vf_driver | None |
335 | accelerated | [True] |
336 +-----------------+--------------------------------------+
337 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot
338 +--------------+--------------------------------------+
340 +--------------+--------------------------------------+
341 | hostname | controller-0 |
342 | uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
343 | ifname | pxeboot0 |
344 | network_name | pxeboot |
345 +--------------+--------------------------------------+
346 [sysadmin@controller-0 ~(keystone_admin)]$
347 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
348 +-----------------+--------------------------------------+
350 +-----------------+--------------------------------------+
354 | imac | 24:6e:96:5d:0c:92 |
358 | schedpolicy | None |
359 | txhashpolicy | None |
360 | uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
361 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
363 | uses | [u'pxeboot0'] |
365 | created_at | 2020-11-17T01:03:57.303000+00:00 |
366 | updated_at | None |
368 | sriov_vf_driver | None |
369 | accelerated | [True] |
370 +-----------------+--------------------------------------+
371 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
372 +--------------+--------------------------------------+
374 +--------------+--------------------------------------+
375 | hostname | controller-0 |
376 | uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
378 | network_name | mgmt |
379 +--------------+--------------------------------------+
380 [sysadmin@controller-0 ~(keystone_admin)]$
381 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
382 +-----------------+--------------------------------------+
384 +-----------------+--------------------------------------+
385 | ifname | cluster0 |
388 | imac | 24:6e:96:5d:0c:92 |
392 | schedpolicy | None |
393 | txhashpolicy | None |
394 | uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
395 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
397 | uses | [u'pxeboot0'] |
399 | created_at | 2020-11-17T01:04:02.613518+00:00 |
400 | updated_at | None |
402 | sriov_vf_driver | None |
403 | accelerated | [True] |
404 +-----------------+--------------------------------------+
405 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
406 +--------------+--------------------------------------+
408 +--------------+--------------------------------------+
409 | hostname | controller-0 |
410 | uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
411 | ifname | cluster0 |
412 | network_name | cluster-host |
413 +--------------+--------------------------------------+
414 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
415 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
416 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
417 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
418 | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
419 | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
420 | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
421 | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
422 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
424 Configure NTP Servers for network time synchronization:
428 system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
434 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
435 +--------------+--------------------------------------+
437 +--------------+--------------------------------------+
438 | uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
439 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
440 | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
441 | created_at | 2020-11-17T00:27:23.529571+00:00 |
442 | updated_at | None |
443 +--------------+--------------------------------------+
445 Add an OSD on controller-0 for Ceph:
449 system host-disk-list controller-0
450 system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
451 system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
452 system host-stor-list controller-0
458 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
459 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
460 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
461 | | de | num | type | gib | gib | | | |
462 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
463 | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
464 | | | | | 75 | | | | |
466 | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
467 | | | | | 25 | | | | |
469 | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
470 | | | | | 25 | | | | |
472 | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
473 | | | | | 25 | | | | |
475 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
476 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
477 +------------------+-------------------------------------------------------+
479 +------------------+-------------------------------------------------------+
482 | state | configuring-on-unlock |
483 | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
484 | journal_size_gib | 1024 |
485 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
486 | journal_node | /dev/sdb2 |
487 | uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
488 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
489 | idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 |
490 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
491 | tier_name | storage |
492 | created_at | 2020-11-17T01:05:04.063823+00:00 |
493 | updated_at | None |
494 +------------------+-------------------------------------------------------+
495 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
496 +------------------+-------------------------------------------------------+
498 +------------------+-------------------------------------------------------+
501 | state | configuring-on-unlock |
502 | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
503 | journal_size_gib | 1024 |
504 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
505 | journal_node | /dev/sdc2 |
506 | uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
507 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
508 | idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 |
509 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
510 | tier_name | storage |
511 | created_at | 2020-11-17T01:05:06.939798+00:00 |
512 | updated_at | None |
513 +------------------+-------------------------------------------------------+
514 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
515 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
516 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
517 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
518 | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
519 | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
520 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
522 2.7 Unlock controller-0
523 ~~~~~~~~~~~~~~~~~~~~~~~
527 system host-unlock controller-0
533 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
534 +-----------------------+-------------------------------------------------+
536 +-----------------------+-------------------------------------------------+
538 | administrative | locked |
539 | availability | online |
542 | bm_username | None |
543 | boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
544 | capabilities | {u'stor_function': u'monitor'} |
545 | clock_synchronization | ntp |
546 | config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 |
547 | config_status | None |
548 | config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 |
549 | console | ttyS0,115200 |
550 | created_at | 2020-11-17T00:28:01.983673+00:00 |
551 | hostname | controller-0 |
553 | install_output | graphical |
554 | install_state | None |
555 | install_state_info | None |
556 | inv_state | inventoried |
557 | invprovision | provisioning |
559 | mgmt_ip | 192.168.18.3 |
560 | mgmt_mac | 24:6e:96:5d:0c:92 |
561 | operational | disabled |
562 | personality | controller |
564 | rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
566 | software_load | 19.12 |
567 | subfunction_avail | online |
568 | subfunction_oper | disabled |
569 | subfunctions | controller,worker,lowlatency |
573 | updated_at | 2020-11-17T01:05:07.015414+00:00 |
575 | uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
576 | vim_progress_status | None |
577 +-----------------------+-------------------------------------------------+
579 Controller-0 will reboot to apply configuration changes and come into
580 service. This can take 5-10 minutes, depending on the performance of the
583 Once the controller comes back up, check the status of controller-0
587 controller-0:~$ source /etc/platform/openrc
588 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
589 +----+--------------+-------------+----------------+-------------+--------------+
590 | id | hostname | personality | administrative | operational | availability |
591 +----+--------------+-------------+----------------+-------------+--------------+
592 | 1 | controller-0 | controller | unlocked | enabled | available |
593 +----+--------------+-------------+----------------+-------------+--------------+
595 2. Installation for the second server (controller-1)
596 ----------------------------------------------------
598 2.1 Power on the controller-1 server and force it to network boot
599 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
601 2.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node
602 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
604 2.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)
605 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
609 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
610 +----+--------------+-------------+----------------+-------------+--------------+
611 | id | hostname | personality | administrative | operational | availability |
612 +----+--------------+-------------+----------------+-------------+--------------+
613 | 1 | controller-0 | controller | unlocked | enabled | degraded |
614 | 2 | None | None | locked | disabled | offline |
615 +----+--------------+-------------+----------------+-------------+--------------+
617 2.4 Using the host id, set the personality of this host to 'controller’:
618 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
622 [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
623 +-----------------------+--------------------------------------+
625 +-----------------------+--------------------------------------+
627 | administrative | locked |
628 | availability | offline |
631 | bm_username | None |
632 | boot_device | /dev/sda |
633 | capabilities | {} |
634 | clock_synchronization | ntp |
635 | config_applied | None |
636 | config_status | None |
637 | config_target | None |
638 | console | ttyS0,115200 |
639 | created_at | 2020-11-17T10:17:44.387813+00:00 |
640 | hostname | controller-1 |
642 | install_output | text |
643 | install_state | None |
644 | install_state_info | None |
646 | invprovision | None |
648 | mgmt_ip | 192.168.18.4 |
649 | mgmt_mac | 24:6e:96:5d:38:ee |
650 | operational | disabled |
651 | personality | controller |
653 | rootfs_device | /dev/sda |
655 | software_load | 19.12 |
656 | subfunction_avail | not-installed |
657 | subfunction_oper | disabled |
658 | subfunctions | controller,worker,lowlatency |
662 | updated_at | None |
664 | uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 |
665 | vim_progress_status | None |
666 +-----------------------+--------------------------------------+
668 2.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
669 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
671 This can take 5-10 minutes, depending on the performance of the host
676 [root@controller-0 hieradata(keystone_admin)]$ system host-list
677 +----+--------------+-------------+----------------+-------------+--------------+
678 | id | hostname | personality | administrative | operational | availability |
679 +----+--------------+-------------+----------------+-------------+--------------+
680 | 1 | controller-0 | controller | unlocked | enabled | available |
681 | 2 | controller-1 | controller | locked | disabled | online |
682 +----+--------------+-------------+----------------+-------------+--------------+
684 2.6 Configure controller-1
685 ~~~~~~~~~~~~~~~~~~~~~~~~~~
691 system host-if-modify controller-1 $OAM_IF -n oam0
692 system host-if-modify controller-1 oam0 -c platform
693 system interface-network-assign controller-1 oam0 oam
695 system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
696 system interface-network-assign controller-1 cluster0 cluster-host
698 system host-if-list controller-1
700 system host-disk-list controller-1
701 system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
702 system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
703 system host-stor-list controller-1
709 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
710 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
711 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
712 +-----------------+--------------------------------------+
714 +-----------------+--------------------------------------+
716 | iftype | ethernet |
717 | ports | [u'eno3'] |
718 | imac | 24:6e:96:5d:39:0e |
722 | schedpolicy | None |
723 | txhashpolicy | None |
724 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
725 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
729 | created_at | 2020-11-22T14:01:32.365863+00:00 |
730 | updated_at | 2020-11-22T15:04:45.090904+00:00 |
732 | sriov_vf_driver | None |
733 | accelerated | [True] |
734 +-----------------+--------------------------------------+
735 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
736 +-----------------+--------------------------------------+
738 +-----------------+--------------------------------------+
740 | iftype | ethernet |
741 | ports | [u'eno3'] |
742 | imac | 24:6e:96:5d:39:0e |
744 | ifclass | platform |
746 | schedpolicy | None |
747 | txhashpolicy | None |
748 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
749 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
753 | created_at | 2020-11-22T14:05:16.052229+00:00 |
754 | updated_at | 2020-11-22T15:08:35.324634+00:00 |
756 | sriov_vf_driver | None |
757 | accelerated | [True] |
758 +-----------------+--------------------------------------+
759 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
760 +--------------+--------------------------------------+
762 +--------------+--------------------------------------+
763 | hostname | controller-1 |
764 | uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
766 | network_name | oam |
767 +--------------+--------------------------------------+
768 [sysadmin@controller-0 ~(keystone_admin)]$
770 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
771 +-----------------+--------------------------------------+
773 +-----------------+--------------------------------------+
774 | ifname | cluster0 |
777 | imac | 24:6e:96:5d:38:ee |
781 | schedpolicy | None |
782 | txhashpolicy | None |
783 | uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 |
784 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
786 | uses | [u'pxeboot0'] |
788 | created_at | 2020-11-22T15:08:43.932209+00:00 |
789 | updated_at | None |
791 | sriov_vf_driver | None |
792 | accelerated | [True] |
793 +-----------------+--------------------------------------+
795 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
796 +--------------+--------------------------------------+
798 +--------------+--------------------------------------+
799 | hostname | controller-1 |
800 | uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
801 | ifname | cluster0 |
802 | network_name | cluster-host |
803 +--------------+--------------------------------------+
804 [sysadmin@controller-0 ~(keystone_admin)]$
805 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
806 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
807 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
808 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
809 | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
810 | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
811 | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
812 | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
813 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
814 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
815 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
816 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
817 | | de | num | type | gib | gib | | | |
818 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
819 | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
820 | | | | | 75 | | | | |
822 | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
823 | | | | | 25 | | | | |
825 | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
826 | | | | | 25 | | | | |
828 | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
829 | | | | | 25 | | | | |
831 | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 |
832 | | | | | 57 | | | | |
834 | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 |
835 | | | | | 57 | | | | |
837 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
838 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
839 +------------------+-------------------------------------------------------+
841 +------------------+-------------------------------------------------------+
844 | state | configuring-on-unlock |
845 | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
846 | journal_size_gib | 1024 |
847 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
848 | journal_node | /dev/sdb2 |
849 | uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
850 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
851 | idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 |
852 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
853 | tier_name | storage |
854 | created_at | 2020-11-22T15:11:55.641193+00:00 |
855 | updated_at | None |
856 +------------------+-------------------------------------------------------+
858 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
859 +------------------+-------------------------------------------------------+
861 +------------------+-------------------------------------------------------+
864 | state | configuring-on-unlock |
865 | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
866 | journal_size_gib | 1024 |
867 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
868 | journal_node | /dev/sdc2 |
869 | uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
870 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
871 | idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae |
872 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
873 | tier_name | storage |
874 | created_at | 2020-11-22T15:12:04.274839+00:00 |
875 | updated_at | None |
876 +------------------+-------------------------------------------------------+
878 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
879 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
880 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
881 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
882 | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
883 | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
884 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
886 2.7 Unlock controller-1
887 ~~~~~~~~~~~~~~~~~~~~~~~
889 Unlock controller-1 in order to bring it into service:
893 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
894 +-----------------------+--------------------------------------+
896 +-----------------------+--------------------------------------+
898 | administrative | locked |
899 | availability | online |
902 | bm_username | None |
903 | boot_device | /dev/sda |
904 | capabilities | {u'stor_function': u'monitor'} |
905 | clock_synchronization | ntp |
906 | config_applied | None |
907 | config_status | Config out-of-date |
908 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
909 | console | ttyS0,115200 |
910 | created_at | 2020-11-22T12:58:11.630526+00:00 |
911 | hostname | controller-1 |
913 | install_output | text |
914 | install_state | None |
915 | install_state_info | None |
916 | inv_state | inventoried |
917 | invprovision | unprovisioned |
919 | mgmt_ip | 192.168.18.4 |
920 | mgmt_mac | 24:6e:96:5d:38:ee |
921 | operational | disabled |
922 | personality | controller |
924 | rootfs_device | /dev/sda |
926 | software_load | 19.12 |
927 | subfunction_avail | online |
928 | subfunction_oper | disabled |
929 | subfunctions | controller,worker,lowlatency |
933 | updated_at | 2020-11-22T15:13:09.716324+00:00 |
935 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
936 | vim_progress_status | None |
937 +-----------------------+--------------------------------------+
939 Controller-1 will reboot in order to apply configuration changes and
940 come into service. This can take 5-10 minutes, depending on the
941 performance of the host machine.
945 [root@controller-0 hieradata(keystone_admin)]$ system host-list
946 +----+--------------+-------------+----------------+-------------+--------------+
947 | id | hostname | personality | administrative | operational | availability |
948 +----+--------------+-------------+----------------+-------------+--------------+
949 | 1 | controller-0 | controller | unlocked | enabled | available |
950 | 2 | controller-1 | controller | unlocked | enabled | available |
951 +----+--------------+-------------+----------------+-------------+--------------+
953 [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
954 +-----------------------+-----------------------------------------------------------------------+
956 +-----------------------+-----------------------------------------------------------------------+
958 | administrative | unlocked |
959 | availability | available |
962 | bm_username | None |
963 | boot_device | /dev/sda |
964 | capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
965 | clock_synchronization | ntp |
966 | config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac |
967 | config_status | None |
968 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
969 | console | ttyS0,115200 |
970 | created_at | 2020-11-22T12:58:11.630526+00:00 |
971 | hostname | controller-1 |
973 | install_output | text |
974 | install_state | None |
975 | install_state_info | None |
976 | inv_state | inventoried |
977 | invprovision | provisioned |
979 | mgmt_ip | 192.168.18.4 |
980 | mgmt_mac | 24:6e:96:5d:38:ee |
981 | operational | enabled |
982 | personality | controller |
984 | rootfs_device | /dev/sda |
986 | software_load | 19.12 |
987 | subfunction_avail | available |
988 | subfunction_oper | enabled |
989 | subfunctions | controller,worker,lowlatency |
993 | updated_at | 2020-11-22T23:59:07.787759+00:00 |
995 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
996 | vim_progress_status | services-enabled |
997 +-----------------------+-----------------------------------------------------------------------+
1000 3. Simple use case for sriov
1001 ````````````````````````````
1003 3.1 After controller-0 is rebooted and up running, download the DPDK
1004 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1008 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
1009 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
1011 --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
1012 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
1013 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
1015 HTTP request sent, awaiting response... 200 OK
1016 Length: 10251680 (9.8M) [application/octet-stream]
1017 Saving to: ‘dpdk-17.11.10.tar.xz’
1019 dpdk-17.11.10.tar.xz 100%
1020 [========================================================================================>] 9.78M
1023 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
1025 sudo tar xvf dpdk-17.11.10.tar.xz
1027 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
1029 3.2 Prepare the yaml file for the network assignment container
1030 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
1032 The following the exmaple of the yaml file:
1036 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
1037 > apiVersion: "k8s.cni.cncf.io/v1"
1038 > kind: NetworkAttachmentDefinition
1040 > name: sriov-data-dpdk-0
1042 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
1046 > "name": "sriov-data-dpdk-0"
1050 > apiVersion: "k8s.cni.cncf.io/v1"
1051 > kind: NetworkAttachmentDefinition
1053 > name: sriov-data-dpdk-1
1055 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
1059 > "name": "sriov-data-dpdk-1"
1063 3.3 Run the network assignent container for the 2 VFs
1064 '''''''''''''''''''''''''''''''''''''''''''''''''''''
1068 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
1069 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
1070 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
1072 3.4 Prepare the VF container yaml file
1073 ''''''''''''''''''''''''''''''''''''''
1077 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
1082 > name: pod-with-dpdk-vfs-0
1084 > k8s.v1.cni.cncf.io/networks: '[
1085 > { "name": "sriov-data-dpdk-0" },
1086 { "name": "sriov-data-dpdk-1" }
1087 > { "name": "sriov-data-dpdk-1" }
1090 > restartPolicy: Never
1092 > - name: pod-with-dpdk-vfs-0
1093 > image: wrsnfv/ubuntu-dpdk-build:v0.3
1096 > value: "/usr/src/dpdk"
1111 > intel.com/pci_sriov_net_physnet0: '1'
1112 > intel.com/pci_sriov_net_physnet1: '1'
1115 > hugepages-1Gi: 2Gi
1117 > intel.com/pci_sriov_net_physnet0: '1'
1118 > intel.com/pci_sriov_net_physnet1: '1'
1120 > - mountPath: /mnt/huge-1048576kB
1122 > - name: dpdk-volume
1123 > mountPath: /usr/src/dpdk
1124 > - name: lib-volume
1125 > mountPath: /lib/modules
1126 > - name: src-volume
1127 > mountPath: /usr/src/
1132 > - name: dpdk-volume
1134 > path: /opt/dpdk-stable/
1135 > - name: lib-volume
1137 > path: /lib/modules
1138 > - name: src-volume
1143 3.5 Run the VF container
1144 ''''''''''''''''''''''''
1146 Start the VF container.
1150 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
1151 pod/pod-with-dpdk-vfs-0 created
1153 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
1154 NAME READY STATUS RESTARTS AGE
1155 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
1157 Login the VF container
1161 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
1167 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
1169 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
1170 HOSTCC scripts/basic/fixdep
1171 HOSTCC scripts/kconfig/conf.o
1172 HOSTCC scripts/kconfig/confdata.o
1173 HOSTCC scripts/kconfig/expr.o
1174 HOSTCC scripts/kconfig/symbol.o
1175 HOSTCC scripts/kconfig/preprocess.o
1176 HOSTCC scripts/kconfig/zconf.lex.o
1177 HOSTCC scripts/kconfig/zconf.tab.o
1178 HOSTLD scripts/kconfig/conf
1179 scripts/kconfig/conf --syncconfig Kconfig
1180 HOSTCC arch/x86/tools/relocs_32.o
1181 HOSTCC arch/x86/tools/relocs_64.o
1182 HOSTCC arch/x86/tools/relocs_common.o
1183 HOSTLD arch/x86/tools/relocs
1184 HOSTCC scripts/genksyms/genksyms.o
1185 YACC scripts/genksyms/parse.tab.c
1186 HOSTCC scripts/genksyms/parse.tab.o
1187 LEX scripts/genksyms/lex.lex.c
1188 YACC scripts/genksyms/parse.tab.h
1189 HOSTCC scripts/genksyms/lex.lex.o
1190 HOSTLD scripts/genksyms/genksyms
1191 HOSTCC scripts/bin2c
1192 HOSTCC scripts/kallsyms
1193 HOSTCC scripts/conmakehash
1194 HOSTCC scripts/recordmcount
1195 HOSTCC scripts/sortextable
1196 HOSTCC scripts/asn1_compiler
1197 HOSTCC scripts/sign-file
1198 HOSTCC scripts/extract-cert
1199 CC scripts/mod/empty.o
1200 HOSTCC scripts/mod/mk_elfconfig
1201 MKELF scripts/mod/elfconfig.h
1202 HOSTCC scripts/mod/modpost.o
1203 CC scripts/mod/devicetable-offsets.s
1204 UPD scripts/mod/devicetable-offsets.h
1205 HOSTCC scripts/mod/file2alias.o
1206 HOSTCC scripts/mod/sumversion.o
1207 HOSTLD scripts/mod/modpost
1209 CC arch/x86/kernel/asm-offsets.s
1210 CALL scripts/checksyscalls.sh
1212 Build the test_pmd application
1217 ./usertools/dpdk-setup.sh
1231 INSTALL-MAP testpmd.map
1232 == Build app/proc_info
1235 INSTALL-APP dpdk-procinfo
1236 INSTALL-MAP dpdk-procinfo.map
1240 INSTALL-APP dpdk-pdump
1241 INSTALL-MAP dpdk-pdump.map
1242 == Build app/test-crypto-perf
1245 CC cperf_options_parsing.o
1246 CC cperf_test_vectors.o
1247 CC cperf_test_throughput.o
1248 CC cperf_test_latency.o
1249 CC cperf_test_pmd_cyclecount.o
1250 CC cperf_test_verify.o
1251 CC cperf_test_vector_parsing.o
1252 CC cperf_test_common.o
1253 LD dpdk-test-crypto-perf
1254 INSTALL-APP dpdk-test-crypto-perf
1255 INSTALL-MAP dpdk-test-crypto-perf.map
1256 == Build app/test-eventdev
1261 CC test_order_common.o
1262 CC test_order_queue.o
1264 CC test_perf_common.o
1265 CC test_perf_queue.o
1267 LD dpdk-test-eventdev
1268 INSTALL-APP dpdk-test-eventdev
1269 INSTALL-MAP dpdk-test-eventdev.map
1270 Build complete [x86_64-native-linuxapp-gcc]
1271 Installation cannot run with T defined and DESTDIR undefined
1272 ------------------------------------------------------------------------------
1273 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1274 ------------------------------------------------------------------------------
1276 Press enter to continue ...
1278 Check the VF PCI information:
1282 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1283 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1284 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1286 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
1290 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1291 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1292 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1293 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1294 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1295 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1296 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1298 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1299 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1300 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1301 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1302 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1303 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1304 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1306 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1307 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1310 Get back to pod and run the test_pmd
1314 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1316 taskset -p --cpu-list 1
1317 pid 1's current affinity list: 2,3,18,19
1319 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1320 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1321 peer=1,"9e:fd:e6:dd:c1:01"
1322 EAL: Detected 32 lcore(s)
1323 EAL: No free hugepages reported in hugepages-2048kB
1324 EAL: Probing VFIO support...
1325 EAL: VFIO support initialized
1326 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1327 EAL: probe driver: 8086:10ed net_ixgbe_vf
1328 EAL: using IOMMU type 1 (Type 1)
1329 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1330 EAL: probe driver: 8086:10ed net_ixgbe_vf
1332 Ports to start sending a burst of packets first
1333 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1334 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1335 Configuring Port 0 (socket 0)
1336 Port 0: 9E:FD:E6:DD:C1:01
1337 Configuring Port 1 (socket 0)
1338 Port 1: 9E:FD:E6:DD:C1:02
1339 Checking link statuses...
1340 Port0 Link Up. speed 10000 Mbps- full-duplex
1341 Port1 Link Up. speed 10000 Mbps- full-duplex
1343 No commandline core given, start packet forwarding
1344 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1346 Logical Core 3 (socket 0) forwards packets on 2 streams:
1347 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1348 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1350 io packet forwarding packets/burst=32
1351 nb forwarding cores=1 - nb forwarding ports=2
1353 CRC stripping enabled
1354 RX queues=1 - RX desc=128 - RX free threshold=32
1355 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1356 TX queues=1 - TX desc=512 - TX free threshold=32
1357 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1358 TX RS bit threshold=32 - TXQ flags=0xf01
1360 CRC stripping enabled
1361 RX queues=1 - RX desc=128 - RX free threshold=32
1362 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1363 TX queues=1 - TX desc=512 - TX free threshold=32
1364 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1365 TX RS bit threshold=32 - TXQ flags=0xf01
1367 Port statistics ====================================
1368 ######################## NIC statistics for port 0 ########################
1369 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1372 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1374 Throughput (since last show)
1377 ############################################################################
1379 ######################## NIC statistics for port 1 ########################
1380 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1383 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1385 Throughput (since last show)
1388 ############################################################################
1390 Port statistics ====================================
1391 ######################## NIC statistics for port 0 ########################
1392 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1395 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1397 Throughput (since last show)
1400 ############################################################################
1412 .. _`StarlingX`: https://docs.starlingx.io/