1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. Copyright (C) 2019 Wind River Systems, Inc.
16 This document describes how to install O-RAN INF image, example configuration for better
17 real time performance, and example deployment of Kubernetes cluster and plugins.
19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20 and container technology.
24 +--------------------+--------------------+--------------------+--------------------+
25 | **Date** | **Ver.** | **Author** | **Comment** |
27 +--------------------+--------------------+--------------------+--------------------+
28 | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version |
30 +--------------------+--------------------+--------------------+--------------------+
31 | 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version |
33 +--------------------+--------------------+--------------------+--------------------+
36 +--------------------+--------------------+--------------------+--------------------+
42 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
48 Following minimum hardware requirements must be met for installation of O-RAN INF image:
50 +--------------------+----------------------------------------------------+
51 | **HW Aspect** | **Requirement** |
53 +--------------------+----------------------------------------------------+
54 | **# of servers** | 1 |
55 +--------------------+----------------------------------------------------+
58 +--------------------+----------------------------------------------------+
61 +--------------------+----------------------------------------------------+
62 | **Disk** | 500G * 2 |
64 +--------------------+----------------------------------------------------+
67 +--------------------+----------------------------------------------------+
71 Software Installation and Deployment
72 ------------------------------------
74 1. Installation from the O-RAN INF ISO image
75 ````````````````````````````````````````````
77 - Please see the README.md file for how to build the image.
78 - The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso
80 1.1 Burn the image to the USB device
81 ''''''''''''''''''''''''''''''''''''
83 - Assume the the usb device is /dev/sdX here
87 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
89 1.2 Install the server
90 ''''''''''''''''''''''
92 - Reboot the target from the USB device.
94 - Select "Graphics console install" or "Serial console install" and press ENTER
96 - Remove the USB device and press ENTER to reboot
99 2. Configuration and initialize the bootstrap
100 `````````````````````````````````````````````
108 ifconfig eno1 128.224.180.14/24 up
110 ip route add default via 128.224.180.1
114 2.3 Login the server through SSH
115 ''''''''''''''''''''''''''''''''
117 2.4 Prepare the localhost.yml for bootstrap
118 '''''''''''''''''''''''''''''''''''''''''''
126 external_oam_subnet: 128.224.180.0/24
128 external_oam_gateway_address: 128.224.180.1
130 external_oam_floating_address: 128.224.180.14
132 admin_username: admin
134 admin_password: Wind123!
136 ansible_become_pass: Wind123!
138 docker_http_proxy: http://xxx.xxx.xxx.xxx:8080
140 docker_https_proxy: https://xxx.xxx.xxx.xxx:8080
142 2.5 Run the bootstrap
143 '''''''''''''''''''''
145 Use the following command:
149 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
151 After the bootstrap successfully finish, it will show as following:
155 PLAY RECAP ******************************************************************************************************************************************
156 localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0
158 2.6 Configure the OAM network
159 '''''''''''''''''''''''''''''
165 controller-0:~$ source /etc/platform/openrc
167 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1
169 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform
171 +-----------------+--------------------------------------+
173 +-----------------+--------------------------------------+
175 +-----------------+--------------------------------------+
176 | iftype | ethernet |
177 +-----------------+--------------------------------------+
178 | ports | [u'eno1'] |
179 +-----------------+--------------------------------------+
180 | imac | 3c:a8:2a:1d:31:d0 |
181 +-----------------+--------------------------------------+
183 +-----------------+--------------------------------------+
184 | ifclass | platform |
185 +-----------------+--------------------------------------+
187 +-----------------+--------------------------------------+
188 | schedpolicy | None |
189 +-----------------+--------------------------------------+
190 | txhashpolicy | None |
191 +-----------------+--------------------------------------+
192 | uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 |
193 +-----------------+--------------------------------------+
194 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
195 +-----------------+--------------------------------------+
197 +-----------------+--------------------------------------+
199 +-----------------+--------------------------------------+
201 +-----------------+--------------------------------------+
202 | created_at | 2020-06-03T08:21:17.526303+00:00 |
203 +-----------------+--------------------------------------+
204 | updated_at | 2020-06-03T10:02:54.682501+00:00 |
205 +-----------------+--------------------------------------+
207 +-----------------+--------------------------------------+
208 | sriov_vf_driver | None |
209 +-----------------+--------------------------------------+
210 | accelerated | [False] |
211 +-----------------+--------------------------------------+
213 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam
215 +--------------+--------------------------------------+
217 +--------------+--------------------------------------+
218 | hostname | controller-0 |
219 +--------------+--------------------------------------+
220 | uuid | 32592cc3-9d66-4961-a51e-23686da00290 |
221 +--------------+--------------------------------------+
223 +--------------+--------------------------------------+
224 | network_name | oam |
225 +--------------+--------------------------------------+
228 [sysadmin@controller-0 ~(keystone_admin)]$
229 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
231 +--------------+--------------------------------------+--------+--------------+
232 | hostname | uuid | ifname | network_name |
233 +--------------+--------------------------------------+--------+--------------+
234 | controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt |
235 +--------------+--------------------------------------+--------+--------------+
236 | controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam |
237 +--------------+--------------------------------------+--------+--------------+
238 | controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host |
239 +--------------+--------------------------------------+--------+--------------+
241 2.7 Configure the NTP server
242 ''''''''''''''''''''''''''''
248 [sysadmin@controller-0 ~(keystone_admin)]$
249 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
251 +--------------+--------------------------------------+
253 +--------------+--------------------------------------+
254 | uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 |
255 +--------------+--------------------------------------+
256 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
257 +--------------+--------------------------------------+
258 | isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 |
259 +--------------+--------------------------------------+
260 | created_at | 2020-06-03T08:19:47.812249+00:00 |
261 +--------------+--------------------------------------+
262 | updated_at | None |
263 +--------------+--------------------------------------+
265 2.8 Configure the Storage by using the second hard drive as ceph backend
266 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
272 [sysadmin@controller-0 ~(keystone_admin)]$
274 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
276 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
277 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path |
278 | | de | num | type | gib | gib | | id | |
279 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
280 | 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
281 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
282 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 |
283 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
285 | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
286 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
287 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 |
288 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
290 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
292 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
294 +------------------+-------------------------------------------------------+
296 +------------------+-------------------------------------------------------+
298 +------------------+-------------------------------------------------------+
300 +------------------+-------------------------------------------------------+
301 | state | configuring-on-unlock |
302 +------------------+-------------------------------------------------------+
303 | journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
304 +------------------+-------------------------------------------------------+
305 | journal_size_gib | 1024 |
306 +------------------+-------------------------------------------------------+
307 | journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 |
308 +------------------+-------------------------------------------------------+
309 | journal_node | /dev/sdb2 |
310 +------------------+-------------------------------------------------------+
311 | uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
312 +------------------+-------------------------------------------------------+
313 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
314 +------------------+-------------------------------------------------------+
315 | idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 |
316 +------------------+-------------------------------------------------------+
317 | tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d |
318 +------------------+-------------------------------------------------------+
319 | tier_name | storage |
320 +------------------+-------------------------------------------------------+
321 | created_at | 2020-06-03T10:03:55.645744+00:00 |
322 +------------------+-------------------------------------------------------+
323 | updated_at | None |
324 +------------------+-------------------------------------------------------+
326 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
328 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
329 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
330 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
331 | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage |
332 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
334 2.9 Configure the SR-IOV enabled network
335 ''''''''''''''''''''''''''''''''''''''''
338 - assign the label of SR-IOV
339 - change the CPU policy to static
340 - change the hugepage setting
341 - create the data network
342 - assign the SR-IOV interface to data network
348 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
350 [sysadmin@controller-0 ~(keystone_admin)]$
352 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled
354 +-------------+--------------------------------------+
356 +-------------+--------------------------------------+
357 | uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b |
358 +-------------+--------------------------------------+
359 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
360 +-------------+--------------------------------------+
361 | label_key | sriovdp |
362 +-------------+--------------------------------------+
363 | label_value | enabled |
364 +-------------+--------------------------------------+
366 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static
368 +-------------+--------------------------------------+
370 +-------------+--------------------------------------+
371 | uuid | 4936f32b-6939-4115-b643-846bd50e7738 |
372 +-------------+--------------------------------------+
373 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
374 +-------------+--------------------------------------+
375 | label_key | kube-cpu-mgr-policy |
376 +-------------+--------------------------------------+
377 | label_value | static |
378 +-------------+--------------------------------------+
381 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0
383 +-------------------------------------+--------------------------------------+
385 +-------------------------------------+--------------------------------------+
386 | Memory: Usable Total (MiB) | 0 |
387 +-------------------------------------+--------------------------------------+
388 | Platform (MiB) | 7000 |
389 +-------------------------------------+--------------------------------------+
390 | Available (MiB) | 0 |
391 +-------------------------------------+--------------------------------------+
392 | Huge Pages Configured | True |
393 +-------------------------------------+--------------------------------------+
394 | vSwitch Huge Pages: Size (MiB) | 1024 |
395 +-------------------------------------+--------------------------------------+
397 +-------------------------------------+--------------------------------------+
399 +-------------------------------------+--------------------------------------+
401 +-------------------------------------+--------------------------------------+
402 | Application Pages (4K): Total | None |
403 +-------------------------------------+--------------------------------------+
404 | Application Huge Pages (2M): Total | 0 |
405 +-------------------------------------+--------------------------------------+
407 +-------------------------------------+--------------------------------------+
408 | Application Huge Pages (1G): Total | 0 |
409 +-------------------------------------+--------------------------------------+
410 | Total Pending | 7 |
411 +-------------------------------------+--------------------------------------+
413 +-------------------------------------+--------------------------------------+
414 | uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 |
415 +-------------------------------------+--------------------------------------+
416 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
417 +-------------------------------------+--------------------------------------+
418 | inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 |
419 +-------------------------------------+--------------------------------------+
420 | created_at | 2020-06-03T08:21:27.468357+00:00 |
421 +-------------------------------------+--------------------------------------+
422 | updated_at | 2020-06-03T10:10:49.159080+00:00 |
423 +-------------------------------------+--------------------------------------+
426 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1
428 +-------------------------------------+--------------------------------------+
430 +-------------------------------------+--------------------------------------+
431 | Memory: Usable Total (MiB) | 0 |
432 +-------------------------------------+--------------------------------------+
433 | Platform (MiB) | 1000 |
434 +-------------------------------------+--------------------------------------+
435 | Available (MiB) | 0 |
436 +-------------------------------------+--------------------------------------+
437 | Huge Pages Configured | True |
438 +-------------------------------------+--------------------------------------+
439 | vSwitch Huge Pages: Size (MiB) | 1024 |
440 +-------------------------------------+--------------------------------------+
442 +-------------------------------------+--------------------------------------+
444 +-------------------------------------+--------------------------------------+
446 +-------------------------------------+--------------------------------------+
447 | Application Pages (4K): Total | None |
448 +-------------------------------------+--------------------------------------+
449 | Application Huge Pages (2M): Total | 0 |
450 +-------------------------------------+--------------------------------------+
452 +-------------------------------------+--------------------------------------+
453 | Application Huge Pages (1G): Total | 0 |
454 +-------------------------------------+--------------------------------------+
455 | Total Pending | 7 |
456 +-------------------------------------+--------------------------------------+
458 +-------------------------------------+--------------------------------------+
459 | uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 |
460 +-------------------------------------+--------------------------------------+
461 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
462 +-------------------------------------+--------------------------------------+
463 | inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f |
464 +-------------------------------------+--------------------------------------+
465 | created_at | 2020-06-03T08:21:27.566279+00:00 |
466 +-------------------------------------+--------------------------------------+
467 | updated_at | 2020-06-03T10:10:49.220907+00:00 |
468 +-------------------------------------+--------------------------------------+
471 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0'
473 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1'
475 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
477 [sysadmin@controller-0 ~(keystone_admin)]$
479 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat
481 +--------------+--------------------------------------+
483 +--------------+--------------------------------------+
485 +--------------+--------------------------------------+
486 | uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b |
487 +--------------+--------------------------------------+
489 +--------------+--------------------------------------+
490 | network_type | flat |
491 +--------------+--------------------------------------+
493 +--------------+--------------------------------------+
494 | description | None |
495 +--------------+--------------------------------------+
497 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat
499 +--------------+--------------------------------------+
501 +--------------+--------------------------------------+
503 +--------------+--------------------------------------+
504 | uuid | 899ca8ba-6800-4d62-aed4-3770878da738 |
505 +--------------+--------------------------------------+
507 +--------------+--------------------------------------+
508 | network_type | flat |
509 +--------------+--------------------------------------+
511 +--------------+--------------------------------------+
512 | description | None |
513 +--------------+--------------------------------------+
516 [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0
518 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
519 | uuid | name | type | pci address | device | processor | accelerated | device type |
520 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
521 | 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
522 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
523 | 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
524 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
525 | e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
526 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
527 | f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
528 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
529 | 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
530 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
531 | 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
532 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
533 | e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
534 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
535 | 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
536 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
539 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0
541 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
543 +-----------------+--------------------------------------+
545 +-----------------+--------------------------------------+
547 +-----------------+--------------------------------------+
548 | iftype | ethernet |
549 +-----------------+--------------------------------------+
550 | ports | [u'ens2f0'] |
551 +-----------------+--------------------------------------+
552 | imac | 00:1b:21:b3:46:2c |
553 +-----------------+--------------------------------------+
555 +-----------------+--------------------------------------+
556 | ifclass | pci-sriov |
557 +-----------------+--------------------------------------+
559 +-----------------+--------------------------------------+
560 | schedpolicy | None |
561 +-----------------+--------------------------------------+
562 | txhashpolicy | None |
563 +-----------------+--------------------------------------+
564 | uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 |
565 +-----------------+--------------------------------------+
566 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
567 +-----------------+--------------------------------------+
569 +-----------------+--------------------------------------+
571 +-----------------+--------------------------------------+
573 +-----------------+--------------------------------------+
574 | created_at | 2020-06-03T12:34:11.633254+00:00 |
575 +-----------------+--------------------------------------+
576 | updated_at | 2020-06-04T02:01:28.168127+00:00 |
577 +-----------------+--------------------------------------+
579 +-----------------+--------------------------------------+
580 | sriov_vf_driver | vfio |
581 +-----------------+--------------------------------------+
582 | accelerated | [True] |
583 +-----------------+--------------------------------------+
585 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0}
587 +------------------+--------------------------------------+
589 +------------------+--------------------------------------+
590 | hostname | controller-0 |
591 +------------------+--------------------------------------+
592 | uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 |
593 +------------------+--------------------------------------+
595 +------------------+--------------------------------------+
596 | datanetwork_name | physnet0 |
597 +------------------+--------------------------------------+
601 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1
603 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
605 +-----------------+--------------------------------------+
607 +-----------------+--------------------------------------+
609 +-----------------+--------------------------------------+
610 | iftype | ethernet |
611 +-----------------+--------------------------------------+
612 | ports | [u'ens2f1'] |
613 +-----------------+--------------------------------------+
614 | imac | 00:1b:21:b3:46:2d |
615 +-----------------+--------------------------------------+
617 +-----------------+--------------------------------------+
618 | ifclass | pci-sriov |
619 +-----------------+--------------------------------------+
621 +-----------------+--------------------------------------+
622 | schedpolicy | None |
623 +-----------------+--------------------------------------+
624 | txhashpolicy | None |
625 +-----------------+--------------------------------------+
626 | uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 |
627 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
631 | created_at | 2020-06-03T12:34:12.072624+00:00 |
632 | updated_at | 2020-06-04T02:03:51.405322+00:00 |
634 | sriov_vf_driver | vfio |
635 | accelerated | [True] |
636 +-----------------+--------------------------------------+
638 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1}
640 +------------------+--------------------------------------+
642 +------------------+--------------------------------------+
643 | hostname | controller-0 |
644 +------------------+--------------------------------------+
645 | uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f |
646 +------------------+--------------------------------------+
648 +------------------+--------------------------------------+
649 | datanetwork_name | physnet1 |
650 +------------------+--------------------------------------+
652 2.10 Unlock the server
653 ''''''''''''''''''''''
659 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
661 +-----------------------+-------------------------------------------------+
663 +-----------------------+-------------------------------------------------+
665 +-----------------------+-------------------------------------------------+
666 | administrative | locked |
667 +-----------------------+-------------------------------------------------+
668 | availability | online |
669 +-----------------------+-------------------------------------------------+
671 +-----------------------+-------------------------------------------------+
673 +-----------------------+-------------------------------------------------+
674 | bm_username | None |
675 +-----------------------+-------------------------------------------------+
676 | boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
677 +-----------------------+-------------------------------------------------+
678 | capabilities | {u'stor_function': u'monitor'} |
679 +-----------------------+-------------------------------------------------+
680 | clock_synchronization | ntp |
681 +-----------------------+-------------------------------------------------+
682 | config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea |
683 +-----------------------+-------------------------------------------------+
684 | config_status | Config out-of-date |
685 +-----------------------+-------------------------------------------------+
686 | config_target | bdaf20c7-c032-4aa2-839d-93da92051bea |
687 +-----------------------+-------------------------------------------------+
689 +-----------------------+-------------------------------------------------+
690 | created_at | 2020-06-03T08:20:59.372602+00:00 |
691 +-----------------------+-------------------------------------------------+
692 | hostname | controller-0 |
693 +-----------------------+-------------------------------------------------+
695 +-----------------------+-------------------------------------------------+
696 | install_output | graphical |
697 +-----------------------+-------------------------------------------------+
698 | install_state | None |
699 +-----------------------+-------------------------------------------------+
700 | install_state_info | None |
701 +-----------------------+-------------------------------------------------+
702 | inv_state | inventoried |
703 +-----------------------+-------------------------------------------------+
704 | invprovision | provisioning |
705 +-----------------------+-------------------------------------------------+
707 +-----------------------+-------------------------------------------------+
708 | mgmt_ip | 192.168.204.2 |
709 +-----------------------+-------------------------------------------------+
710 | mgmt_mac | 00:00:00:00:00:00 |
711 +-----------------------+-------------------------------------------------+
712 | operational | disabled |
713 +-----------------------+-------------------------------------------------+
714 | personality | controller |
715 +-----------------------+-------------------------------------------------+
717 +-----------------------+-------------------------------------------------+
718 | rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
719 +-----------------------+-------------------------------------------------+
721 +-----------------------+-------------------------------------------------+
722 | software_load | 19.12 |
723 +-----------------------+-------------------------------------------------+
724 | subfunction_avail | online |
725 +-----------------------+-------------------------------------------------+
726 | subfunction_oper | disabled |
727 +-----------------------+-------------------------------------------------+
728 | subfunctions | controller,worker |
729 +-----------------------+-------------------------------------------------+
731 +-----------------------+-------------------------------------------------+
733 +-----------------------+-------------------------------------------------+
735 +-----------------------+-------------------------------------------------+
736 | updated_at | 2020-06-04T02:04:59.514854+00:00 |
737 +-----------------------+-------------------------------------------------+
739 +-----------------------+-------------------------------------------------+
740 | uuid | 08c23843-afa4-4b80-a285-be126365276e |
741 +-----------------------+-------------------------------------------------+
742 | vim_progress_status | None |
743 +-----------------------+-------------------------------------------------+
745 3. Simple use case for sriov
746 ````````````````````````````
748 3.1 After controller-0 is rebooted and up running, download the DPDK
749 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
753 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
754 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
756 --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
757 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
758 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
760 HTTP request sent, awaiting response... 200 OK
761 Length: 10251680 (9.8M) [application/octet-stream]
762 Saving to: ‘dpdk-17.11.10.tar.xz’
764 dpdk-17.11.10.tar.xz 100%
765 [========================================================================================>] 9.78M
768 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
770 sudo tar xvf dpdk-17.11.10.tar.xz
772 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
774 3.2 Prepare the yaml file for the network assignment container
775 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
777 The following the exmaple of the yaml file:
781 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
782 > apiVersion: "k8s.cni.cncf.io/v1"
783 > kind: NetworkAttachmentDefinition
785 > name: sriov-data-dpdk-0
787 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
791 > "name": "sriov-data-dpdk-0"
795 > apiVersion: "k8s.cni.cncf.io/v1"
796 > kind: NetworkAttachmentDefinition
798 > name: sriov-data-dpdk-1
800 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
804 > "name": "sriov-data-dpdk-1"
808 3.3 Run the network assignent container for the 2 VFs
809 '''''''''''''''''''''''''''''''''''''''''''''''''''''
813 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
814 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
815 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
817 3.4 Prepare the VF container yaml file
818 ''''''''''''''''''''''''''''''''''''''
822 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
827 > name: pod-with-dpdk-vfs-0
829 > k8s.v1.cni.cncf.io/networks: '[
830 > { "name": "sriov-data-dpdk-0" },
831 { "name": "sriov-data-dpdk-1" }
832 > { "name": "sriov-data-dpdk-1" }
835 > restartPolicy: Never
837 > - name: pod-with-dpdk-vfs-0
838 > image: wrsnfv/ubuntu-dpdk-build:v0.3
841 > value: "/usr/src/dpdk"
856 > intel.com/pci_sriov_net_physnet0: '1'
857 > intel.com/pci_sriov_net_physnet1: '1'
862 > intel.com/pci_sriov_net_physnet0: '1'
863 > intel.com/pci_sriov_net_physnet1: '1'
865 > - mountPath: /mnt/huge-1048576kB
867 > - name: dpdk-volume
868 > mountPath: /usr/src/dpdk
870 > mountPath: /lib/modules
872 > mountPath: /usr/src/
877 > - name: dpdk-volume
879 > path: /opt/dpdk-stable/
888 3.5 Run the VF container
889 ''''''''''''''''''''''''
891 Start the VF container.
895 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
896 pod/pod-with-dpdk-vfs-0 created
898 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
899 NAME READY STATUS RESTARTS AGE
900 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
902 Login the VF container
906 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
912 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
914 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
915 HOSTCC scripts/basic/fixdep
916 HOSTCC scripts/kconfig/conf.o
917 HOSTCC scripts/kconfig/confdata.o
918 HOSTCC scripts/kconfig/expr.o
919 HOSTCC scripts/kconfig/symbol.o
920 HOSTCC scripts/kconfig/preprocess.o
921 HOSTCC scripts/kconfig/zconf.lex.o
922 HOSTCC scripts/kconfig/zconf.tab.o
923 HOSTLD scripts/kconfig/conf
924 scripts/kconfig/conf --syncconfig Kconfig
925 HOSTCC arch/x86/tools/relocs_32.o
926 HOSTCC arch/x86/tools/relocs_64.o
927 HOSTCC arch/x86/tools/relocs_common.o
928 HOSTLD arch/x86/tools/relocs
929 HOSTCC scripts/genksyms/genksyms.o
930 YACC scripts/genksyms/parse.tab.c
931 HOSTCC scripts/genksyms/parse.tab.o
932 LEX scripts/genksyms/lex.lex.c
933 YACC scripts/genksyms/parse.tab.h
934 HOSTCC scripts/genksyms/lex.lex.o
935 HOSTLD scripts/genksyms/genksyms
937 HOSTCC scripts/kallsyms
938 HOSTCC scripts/conmakehash
939 HOSTCC scripts/recordmcount
940 HOSTCC scripts/sortextable
941 HOSTCC scripts/asn1_compiler
942 HOSTCC scripts/sign-file
943 HOSTCC scripts/extract-cert
944 CC scripts/mod/empty.o
945 HOSTCC scripts/mod/mk_elfconfig
946 MKELF scripts/mod/elfconfig.h
947 HOSTCC scripts/mod/modpost.o
948 CC scripts/mod/devicetable-offsets.s
949 UPD scripts/mod/devicetable-offsets.h
950 HOSTCC scripts/mod/file2alias.o
951 HOSTCC scripts/mod/sumversion.o
952 HOSTLD scripts/mod/modpost
954 CC arch/x86/kernel/asm-offsets.s
955 CALL scripts/checksyscalls.sh
957 Build the test_pmd application
962 ./usertools/dpdk-setup.sh
976 INSTALL-MAP testpmd.map
977 == Build app/proc_info
980 INSTALL-APP dpdk-procinfo
981 INSTALL-MAP dpdk-procinfo.map
985 INSTALL-APP dpdk-pdump
986 INSTALL-MAP dpdk-pdump.map
987 == Build app/test-crypto-perf
990 CC cperf_options_parsing.o
991 CC cperf_test_vectors.o
992 CC cperf_test_throughput.o
993 CC cperf_test_latency.o
994 CC cperf_test_pmd_cyclecount.o
995 CC cperf_test_verify.o
996 CC cperf_test_vector_parsing.o
997 CC cperf_test_common.o
998 LD dpdk-test-crypto-perf
999 INSTALL-APP dpdk-test-crypto-perf
1000 INSTALL-MAP dpdk-test-crypto-perf.map
1001 == Build app/test-eventdev
1006 CC test_order_common.o
1007 CC test_order_queue.o
1009 CC test_perf_common.o
1010 CC test_perf_queue.o
1012 LD dpdk-test-eventdev
1013 INSTALL-APP dpdk-test-eventdev
1014 INSTALL-MAP dpdk-test-eventdev.map
1015 Build complete [x86_64-native-linuxapp-gcc]
1016 Installation cannot run with T defined and DESTDIR undefined
1017 ------------------------------------------------------------------------------
1018 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1019 ------------------------------------------------------------------------------
1021 Press enter to continue ...
1023 Check the VF PCI information:
1027 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1028 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1029 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1031 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
1035 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1036 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1037 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1038 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1039 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1040 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1041 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1043 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1044 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1045 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1046 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1047 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1048 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1049 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1051 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1052 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1055 Get back to pod and run the test_pmd
1059 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1061 taskset -p --cpu-list 1
1062 pid 1's current affinity list: 2,3,18,19
1064 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1065 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1066 peer=1,"9e:fd:e6:dd:c1:01"
1067 EAL: Detected 32 lcore(s)
1068 EAL: No free hugepages reported in hugepages-2048kB
1069 EAL: Probing VFIO support...
1070 EAL: VFIO support initialized
1071 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1072 EAL: probe driver: 8086:10ed net_ixgbe_vf
1073 EAL: using IOMMU type 1 (Type 1)
1074 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1075 EAL: probe driver: 8086:10ed net_ixgbe_vf
1077 Ports to start sending a burst of packets first
1078 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1079 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1080 Configuring Port 0 (socket 0)
1081 Port 0: 9E:FD:E6:DD:C1:01
1082 Configuring Port 1 (socket 0)
1083 Port 1: 9E:FD:E6:DD:C1:02
1084 Checking link statuses...
1085 Port0 Link Up. speed 10000 Mbps- full-duplex
1086 Port1 Link Up. speed 10000 Mbps- full-duplex
1088 No commandline core given, start packet forwarding
1089 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1091 Logical Core 3 (socket 0) forwards packets on 2 streams:
1092 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1093 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1095 io packet forwarding packets/burst=32
1096 nb forwarding cores=1 - nb forwarding ports=2
1098 CRC stripping enabled
1099 RX queues=1 - RX desc=128 - RX free threshold=32
1100 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1101 TX queues=1 - TX desc=512 - TX free threshold=32
1102 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1103 TX RS bit threshold=32 - TXQ flags=0xf01
1105 CRC stripping enabled
1106 RX queues=1 - RX desc=128 - RX free threshold=32
1107 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1108 TX queues=1 - TX desc=512 - TX free threshold=32
1109 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1110 TX RS bit threshold=32 - TXQ flags=0xf01
1112 Port statistics ====================================
1113 ######################## NIC statistics for port 0 ########################
1114 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1117 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1119 Throughput (since last show)
1122 ############################################################################
1124 ######################## NIC statistics for port 1 ########################
1125 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1128 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1130 Throughput (since last show)
1133 ############################################################################
1135 Port statistics ====================================
1136 ######################## NIC statistics for port 0 ########################
1137 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1140 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1142 Throughput (since last show)
1145 ############################################################################
1157 .. _`StarlingX`: https://docs.starlingx.io/