1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. Copyright (C) 2019 Wind River Systems, Inc.
16 This document describes how to install O-RAN INF image, example configuration for better
17 real time performance, and example deployment of Kubernetes cluster and plugins.
19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20 and container technology.
24 +--------------------+--------------------+--------------------+--------------------+
25 | **Date** | **Ver.** | **Author** | **Comment** |
27 +--------------------+--------------------+--------------------+--------------------+
28 | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version |
30 +--------------------+--------------------+--------------------+--------------------+
31 | 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version |
33 +--------------------+--------------------+--------------------+--------------------+
36 +--------------------+--------------------+--------------------+--------------------+
42 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
48 Following minimum hardware requirements must be met for installation of O-RAN INF image:
50 +--------------------+----------------------------------------------------+
51 | **HW Aspect** | **Requirement** |
53 +--------------------+----------------------------------------------------+
54 | **# of servers** | 1 |
55 +--------------------+----------------------------------------------------+
58 +--------------------+----------------------------------------------------+
61 +--------------------+----------------------------------------------------+
62 | **Disk** | 500G * 2 |
64 +--------------------+----------------------------------------------------+
67 +--------------------+----------------------------------------------------+
71 Software Installation and Deployment
72 ------------------------------------
74 1. Installation from the O-RAN INF ISO image
75 ````````````````````````````````````````````
77 - Please see the README.md file for how to build the image.
78 - The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso
80 1.1 Burn the image to the USB device
81 ''''''''''''''''''''''''''''''''''''
83 - Assume the the usb device is /dev/sdX here
87 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
89 1.2 Install the server
90 ''''''''''''''''''''''
92 - Reboot the target from the USB device.
94 - Select "Graphics console install" or "Serial console install" and press ENTER
96 - Remove the USB device and press ENTER to reboot
99 2. Configuration and initialize the bootstrap
100 `````````````````````````````````````````````
108 ifconfig eno1 128.224.180.14/24 up
110 ip route add default via 128.224.180.1
114 2.3 Login the server through SSH
115 ````````````````````````````````
117 2.4 Prepare the localhost.yml for bootstrap
118 ```````````````````````````````````````````
126 external_oam_subnet: 128.224.180.0/24
128 external_oam_gateway_address: 128.224.180.1
130 external_oam_floating_address: 128.224.180.14
132 admin_username: admin
134 admin_password: Wind123!
136 ansible_become_pass: Wind123!
138 docker_http_proxy: http://xxx.xxx.xxx.xxx:8080
140 docker_https_proxy: https://xxx.xxx.xxx.xxx:8080
142 2.5 Run the bootstrap
143 `````````````````````
145 Use the following command:
149 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
151 After the bootstrap successfully finish, it will show as following:
155 PLAY RECAP ******************************************************************************************************************************************
156 localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0
158 2.6 Configure the OAM network
159 `````````````````````````````
165 controller-0:~$ source /etc/platform/openrc
167 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1
169 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform
171 +-----------------+--------------------------------------+
173 +-----------------+--------------------------------------+
175 +-----------------+--------------------------------------+
176 | iftype | ethernet |
177 +-----------------+--------------------------------------+
178 | ports | [u'eno1'] |
179 +-----------------+--------------------------------------+
180 | imac | 3c:a8:2a:1d:31:d0 |
181 +-----------------+--------------------------------------+
183 +-----------------+--------------------------------------+
184 | ifclass | platform |
185 +-----------------+--------------------------------------+
187 +-----------------+--------------------------------------+
188 | schedpolicy | None |
189 +-----------------+--------------------------------------+
190 | txhashpolicy | None |
191 +-----------------+--------------------------------------+
192 | uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 |
193 +-----------------+--------------------------------------+
194 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
195 +-----------------+--------------------------------------+
197 +-----------------+--------------------------------------+
199 +-----------------+--------------------------------------+
201 +-----------------+--------------------------------------+
202 | created_at | 2020-06-03T08:21:17.526303+00:00 |
203 +-----------------+--------------------------------------+
204 | updated_at | 2020-06-03T10:02:54.682501+00:00 |
205 +-----------------+--------------------------------------+
207 +-----------------+--------------------------------------+
208 | sriov_vf_driver | None |
209 +-----------------+--------------------------------------+
210 | accelerated | [False] |
211 +-----------------+--------------------------------------+
213 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam
215 +--------------+--------------------------------------+
217 +--------------+--------------------------------------+
218 | hostname | controller-0 |
219 +--------------+--------------------------------------+
220 | uuid | 32592cc3-9d66-4961-a51e-23686da00290 |
221 +--------------+--------------------------------------+
223 +--------------+--------------------------------------+
224 | network_name | oam |
225 +--------------+--------------------------------------+
228 [sysadmin@controller-0 ~(keystone_admin)]$
229 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
231 +--------------+--------------------------------------+--------+--------------+
232 | hostname | uuid | ifname | network_name |
233 +--------------+--------------------------------------+--------+--------------+
234 | controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt |
235 +--------------+--------------------------------------+--------+--------------+
236 | controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam |
237 +--------------+--------------------------------------+--------+--------------+
238 | controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host |
239 +--------------+--------------------------------------+--------+--------------+
241 2.7 Configure the NTP server
242 ````````````````````````````
248 [sysadmin@controller-0 ~(keystone_admin)]$
249 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
251 +--------------+--------------------------------------+
253 +--------------+--------------------------------------+
254 | uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 |
255 +--------------+--------------------------------------+
256 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
257 +--------------+--------------------------------------+
258 | isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 |
259 +--------------+--------------------------------------+
260 | created_at | 2020-06-03T08:19:47.812249+00:00 |
261 +--------------+--------------------------------------+
262 | updated_at | None |
263 +--------------+--------------------------------------+
265 2.8 Configure the Storage by using the second hard drive as ceph backend
266 ````````````````````````````````````````````````````````````````````````
272 [sysadmin@controller-0 ~(keystone_admin)]$
274 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
276 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
277 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path |
278 | | de | num | type | gib | gib | | id | |
279 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
280 | 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
281 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
282 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 |
283 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
285 | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
286 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
287 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 |
288 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
290 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
292 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
294 +------------------+-------------------------------------------------------+
296 +------------------+-------------------------------------------------------+
298 +------------------+-------------------------------------------------------+
300 +------------------+-------------------------------------------------------+
301 | state | configuring-on-unlock |
302 +------------------+-------------------------------------------------------+
303 | journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
304 +------------------+-------------------------------------------------------+
305 | journal_size_gib | 1024 |
306 +------------------+-------------------------------------------------------+
307 | journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 |
308 +------------------+-------------------------------------------------------+
309 | journal_node | /dev/sdb2 |
310 +------------------+-------------------------------------------------------+
311 | uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
312 +------------------+-------------------------------------------------------+
313 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
314 +------------------+-------------------------------------------------------+
315 | idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 |
316 +------------------+-------------------------------------------------------+
317 | tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d |
318 +------------------+-------------------------------------------------------+
319 | tier_name | storage |
320 +------------------+-------------------------------------------------------+
321 | created_at | 2020-06-03T10:03:55.645744+00:00 |
322 +------------------+-------------------------------------------------------+
323 | updated_at | None |
324 +------------------+-------------------------------------------------------+
326 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
328 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
329 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
330 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
331 | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage |
332 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
334 2.9 Configure the SR-IOV enabled network
335 ````````````````````````````````````````
337 - assign the label of SR-IOV
338 - change the CPU policy to static
339 - change the hugepage setting
340 - create the data network
341 - assign the SR-IOV interface to data network
347 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
349 [sysadmin@controller-0 ~(keystone_admin)]$
351 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled
353 +-------------+--------------------------------------+
355 +-------------+--------------------------------------+
356 | uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b |
357 +-------------+--------------------------------------+
358 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
359 +-------------+--------------------------------------+
360 | label_key | sriovdp |
361 +-------------+--------------------------------------+
362 | label_value | enabled |
363 +-------------+--------------------------------------+
365 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static
367 +-------------+--------------------------------------+
369 +-------------+--------------------------------------+
370 | uuid | 4936f32b-6939-4115-b643-846bd50e7738 |
371 +-------------+--------------------------------------+
372 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
373 +-------------+--------------------------------------+
374 | label_key | kube-cpu-mgr-policy |
375 +-------------+--------------------------------------+
376 | label_value | static |
377 +-------------+--------------------------------------+
380 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0
382 +-------------------------------------+--------------------------------------+
384 +-------------------------------------+--------------------------------------+
385 | Memory: Usable Total (MiB) | 0 |
386 +-------------------------------------+--------------------------------------+
387 | Platform (MiB) | 7000 |
388 +-------------------------------------+--------------------------------------+
389 | Available (MiB) | 0 |
390 +-------------------------------------+--------------------------------------+
391 | Huge Pages Configured | True |
392 +-------------------------------------+--------------------------------------+
393 | vSwitch Huge Pages: Size (MiB) | 1024 |
394 +-------------------------------------+--------------------------------------+
396 +-------------------------------------+--------------------------------------+
398 +-------------------------------------+--------------------------------------+
400 +-------------------------------------+--------------------------------------+
401 | Application Pages (4K): Total | None |
402 +-------------------------------------+--------------------------------------+
403 | Application Huge Pages (2M): Total | 0 |
404 +-------------------------------------+--------------------------------------+
406 +-------------------------------------+--------------------------------------+
407 | Application Huge Pages (1G): Total | 0 |
408 +-------------------------------------+--------------------------------------+
409 | Total Pending | 7 |
410 +-------------------------------------+--------------------------------------+
412 +-------------------------------------+--------------------------------------+
413 | uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 |
414 +-------------------------------------+--------------------------------------+
415 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
416 +-------------------------------------+--------------------------------------+
417 | inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 |
418 +-------------------------------------+--------------------------------------+
419 | created_at | 2020-06-03T08:21:27.468357+00:00 |
420 +-------------------------------------+--------------------------------------+
421 | updated_at | 2020-06-03T10:10:49.159080+00:00 |
422 +-------------------------------------+--------------------------------------+
425 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1
427 +-------------------------------------+--------------------------------------+
429 +-------------------------------------+--------------------------------------+
430 | Memory: Usable Total (MiB) | 0 |
431 +-------------------------------------+--------------------------------------+
432 | Platform (MiB) | 1000 |
433 +-------------------------------------+--------------------------------------+
434 | Available (MiB) | 0 |
435 +-------------------------------------+--------------------------------------+
436 | Huge Pages Configured | True |
437 +-------------------------------------+--------------------------------------+
438 | vSwitch Huge Pages: Size (MiB) | 1024 |
439 +-------------------------------------+--------------------------------------+
441 +-------------------------------------+--------------------------------------+
443 +-------------------------------------+--------------------------------------+
445 +-------------------------------------+--------------------------------------+
446 | Application Pages (4K): Total | None |
447 +-------------------------------------+--------------------------------------+
448 | Application Huge Pages (2M): Total | 0 |
449 +-------------------------------------+--------------------------------------+
451 +-------------------------------------+--------------------------------------+
452 | Application Huge Pages (1G): Total | 0 |
453 +-------------------------------------+--------------------------------------+
454 | Total Pending | 7 |
455 +-------------------------------------+--------------------------------------+
457 +-------------------------------------+--------------------------------------+
458 | uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 |
459 +-------------------------------------+--------------------------------------+
460 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
461 +-------------------------------------+--------------------------------------+
462 | inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f |
463 +-------------------------------------+--------------------------------------+
464 | created_at | 2020-06-03T08:21:27.566279+00:00 |
465 +-------------------------------------+--------------------------------------+
466 | updated_at | 2020-06-03T10:10:49.220907+00:00 |
467 +-------------------------------------+--------------------------------------+
470 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0'
472 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1'
474 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
476 [sysadmin@controller-0 ~(keystone_admin)]$
478 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat
480 +--------------+--------------------------------------+
482 +--------------+--------------------------------------+
484 +--------------+--------------------------------------+
485 | uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b |
486 +--------------+--------------------------------------+
488 +--------------+--------------------------------------+
489 | network_type | flat |
490 +--------------+--------------------------------------+
492 +--------------+--------------------------------------+
493 | description | None |
494 +--------------+--------------------------------------+
496 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat
498 +--------------+--------------------------------------+
500 +--------------+--------------------------------------+
502 +--------------+--------------------------------------+
503 | uuid | 899ca8ba-6800-4d62-aed4-3770878da738 |
504 +--------------+--------------------------------------+
506 +--------------+--------------------------------------+
507 | network_type | flat |
508 +--------------+--------------------------------------+
510 +--------------+--------------------------------------+
511 | description | None |
512 +--------------+--------------------------------------+
515 [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0
517 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
518 | uuid | name | type | pci address | device | processor | accelerated | device type |
519 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
520 | 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
521 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
522 | 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
523 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
524 | e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
525 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
526 | f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
527 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
528 | 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
529 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
530 | 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
531 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
532 | e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
533 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
534 | 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
535 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
538 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0
540 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
542 +-----------------+--------------------------------------+
544 +-----------------+--------------------------------------+
546 +-----------------+--------------------------------------+
547 | iftype | ethernet |
548 +-----------------+--------------------------------------+
549 | ports | [u'ens2f0'] |
550 +-----------------+--------------------------------------+
551 | imac | 00:1b:21:b3:46:2c |
552 +-----------------+--------------------------------------+
554 +-----------------+--------------------------------------+
555 | ifclass | pci-sriov |
556 +-----------------+--------------------------------------+
558 +-----------------+--------------------------------------+
559 | schedpolicy | None |
560 +-----------------+--------------------------------------+
561 | txhashpolicy | None |
562 +-----------------+--------------------------------------+
563 | uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 |
564 +-----------------+--------------------------------------+
565 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
566 +-----------------+--------------------------------------+
568 +-----------------+--------------------------------------+
570 +-----------------+--------------------------------------+
572 +-----------------+--------------------------------------+
573 | created_at | 2020-06-03T12:34:11.633254+00:00 |
574 +-----------------+--------------------------------------+
575 | updated_at | 2020-06-04T02:01:28.168127+00:00 |
576 +-----------------+--------------------------------------+
578 +-----------------+--------------------------------------+
579 | sriov_vf_driver | vfio |
580 +-----------------+--------------------------------------+
581 | accelerated | [True] |
582 +-----------------+--------------------------------------+
584 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0}
586 +------------------+--------------------------------------+
588 +------------------+--------------------------------------+
589 | hostname | controller-0 |
590 +------------------+--------------------------------------+
591 | uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 |
592 +------------------+--------------------------------------+
594 +------------------+--------------------------------------+
595 | datanetwork_name | physnet0 |
596 +------------------+--------------------------------------+
600 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1
602 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
604 +-----------------+--------------------------------------+
606 +-----------------+--------------------------------------+
608 +-----------------+--------------------------------------+
609 | iftype | ethernet |
610 +-----------------+--------------------------------------+
611 | ports | [u'ens2f1'] |
612 +-----------------+--------------------------------------+
613 | imac | 00:1b:21:b3:46:2d |
614 +-----------------+--------------------------------------+
616 +-----------------+--------------------------------------+
617 | ifclass | pci-sriov |
618 +-----------------+--------------------------------------+
620 +-----------------+--------------------------------------+
621 | schedpolicy | None |
622 +-----------------+--------------------------------------+
623 | txhashpolicy | None |
624 +-----------------+--------------------------------------+
625 | uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 |
626 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
630 | created_at | 2020-06-03T12:34:12.072624+00:00 |
631 | updated_at | 2020-06-04T02:03:51.405322+00:00 |
633 | sriov_vf_driver | vfio |
634 | accelerated | [True] |
635 +-----------------+--------------------------------------+
637 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1}
639 +------------------+--------------------------------------+
641 +------------------+--------------------------------------+
642 | hostname | controller-0 |
643 +------------------+--------------------------------------+
644 | uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f |
645 +------------------+--------------------------------------+
647 +------------------+--------------------------------------+
648 | datanetwork_name | physnet1 |
649 +------------------+--------------------------------------+
651 2.10 Unlock the server
652 ``````````````````````
658 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
660 +-----------------------+-------------------------------------------------+
662 +-----------------------+-------------------------------------------------+
664 +-----------------------+-------------------------------------------------+
665 | administrative | locked |
666 +-----------------------+-------------------------------------------------+
667 | availability | online |
668 +-----------------------+-------------------------------------------------+
670 +-----------------------+-------------------------------------------------+
672 +-----------------------+-------------------------------------------------+
673 | bm_username | None |
674 +-----------------------+-------------------------------------------------+
675 | boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
676 +-----------------------+-------------------------------------------------+
677 | capabilities | {u'stor_function': u'monitor'} |
678 +-----------------------+-------------------------------------------------+
679 | clock_synchronization | ntp |
680 +-----------------------+-------------------------------------------------+
681 | config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea |
682 +-----------------------+-------------------------------------------------+
683 | config_status | Config out-of-date |
684 +-----------------------+-------------------------------------------------+
685 | config_target | bdaf20c7-c032-4aa2-839d-93da92051bea |
686 +-----------------------+-------------------------------------------------+
688 +-----------------------+-------------------------------------------------+
689 | created_at | 2020-06-03T08:20:59.372602+00:00 |
690 +-----------------------+-------------------------------------------------+
691 | hostname | controller-0 |
692 +-----------------------+-------------------------------------------------+
694 +-----------------------+-------------------------------------------------+
695 | install_output | graphical |
696 +-----------------------+-------------------------------------------------+
697 | install_state | None |
698 +-----------------------+-------------------------------------------------+
699 | install_state_info | None |
700 +-----------------------+-------------------------------------------------+
701 | inv_state | inventoried |
702 +-----------------------+-------------------------------------------------+
703 | invprovision | provisioning |
704 +-----------------------+-------------------------------------------------+
706 +-----------------------+-------------------------------------------------+
707 | mgmt_ip | 192.168.204.2 |
708 +-----------------------+-------------------------------------------------+
709 | mgmt_mac | 00:00:00:00:00:00 |
710 +-----------------------+-------------------------------------------------+
711 | operational | disabled |
712 +-----------------------+-------------------------------------------------+
713 | personality | controller |
714 +-----------------------+-------------------------------------------------+
716 +-----------------------+-------------------------------------------------+
717 | rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
718 +-----------------------+-------------------------------------------------+
720 +-----------------------+-------------------------------------------------+
721 | software_load | 19.12 |
722 +-----------------------+-------------------------------------------------+
723 | subfunction_avail | online |
724 +-----------------------+-------------------------------------------------+
725 | subfunction_oper | disabled |
726 +-----------------------+-------------------------------------------------+
727 | subfunctions | controller,worker |
728 +-----------------------+-------------------------------------------------+
730 +-----------------------+-------------------------------------------------+
732 +-----------------------+-------------------------------------------------+
734 +-----------------------+-------------------------------------------------+
735 | updated_at | 2020-06-04T02:04:59.514854+00:00 |
736 +-----------------------+-------------------------------------------------+
738 +-----------------------+-------------------------------------------------+
739 | uuid | 08c23843-afa4-4b80-a285-be126365276e |
740 +-----------------------+-------------------------------------------------+
741 | vim_progress_status | None |
742 +-----------------------+-------------------------------------------------+
744 3. Simple use case for sriov
745 ````````````````````````````
747 3.1 After controller-0 is rebooted and up running, download the DPDK
748 ````````````````````````````````````````````````````````````````````
752 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
753 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
755 --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
756 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
757 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
759 HTTP request sent, awaiting response... 200 OK
760 Length: 10251680 (9.8M) [application/octet-stream]
761 Saving to: ‘dpdk-17.11.10.tar.xz’
763 dpdk-17.11.10.tar.xz 100%
764 [========================================================================================>] 9.78M
767 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
769 sudo tar xvf dpdk-17.11.10.tar.xz
771 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
773 3.1 Prepare the yaml file for the network assignment container
774 ``````````````````````````````````````````````````````````````
776 The following the exmaple of the yaml file:
780 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
781 > apiVersion: "k8s.cni.cncf.io/v1"
782 > kind: NetworkAttachmentDefinition
784 > name: sriov-data-dpdk-0
786 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
790 > "name": "sriov-data-dpdk-0"
794 > apiVersion: "k8s.cni.cncf.io/v1"
795 > kind: NetworkAttachmentDefinition
797 > name: sriov-data-dpdk-1
799 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
803 > "name": "sriov-data-dpdk-1"
807 3.2 Run the network assignent container for the 2 VFs
808 `````````````````````````````````````````````````````
812 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
813 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
814 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
816 3.3 Prepare the VF container yaml file
817 ``````````````````````````````````````
821 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
826 > name: pod-with-dpdk-vfs-0
828 > k8s.v1.cni.cncf.io/networks: '[
829 > { "name": "sriov-data-dpdk-0" },
830 { "name": "sriov-data-dpdk-1" }
831 > { "name": "sriov-data-dpdk-1" }
834 > restartPolicy: Never
836 > - name: pod-with-dpdk-vfs-0
837 > image: wrsnfv/ubuntu-dpdk-build:v0.3
840 > value: "/usr/src/dpdk"
855 > intel.com/pci_sriov_net_physnet0: '1'
856 > intel.com/pci_sriov_net_physnet1: '1'
861 > intel.com/pci_sriov_net_physnet0: '1'
862 > intel.com/pci_sriov_net_physnet1: '1'
864 > - mountPath: /mnt/huge-1048576kB
866 > - name: dpdk-volume
867 > mountPath: /usr/src/dpdk
869 > mountPath: /lib/modules
871 > mountPath: /usr/src/
876 > - name: dpdk-volume
878 > path: /opt/dpdk-stable/
887 3.4 Run the VF container
888 ````````````````````````
890 Start the VF container.
894 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
895 pod/pod-with-dpdk-vfs-0 created
897 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
898 NAME READY STATUS RESTARTS AGE
899 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
901 Login the VF container
905 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
911 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
913 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
914 HOSTCC scripts/basic/fixdep
915 HOSTCC scripts/kconfig/conf.o
916 HOSTCC scripts/kconfig/confdata.o
917 HOSTCC scripts/kconfig/expr.o
918 HOSTCC scripts/kconfig/symbol.o
919 HOSTCC scripts/kconfig/preprocess.o
920 HOSTCC scripts/kconfig/zconf.lex.o
921 HOSTCC scripts/kconfig/zconf.tab.o
922 HOSTLD scripts/kconfig/conf
923 scripts/kconfig/conf --syncconfig Kconfig
924 HOSTCC arch/x86/tools/relocs_32.o
925 HOSTCC arch/x86/tools/relocs_64.o
926 HOSTCC arch/x86/tools/relocs_common.o
927 HOSTLD arch/x86/tools/relocs
928 HOSTCC scripts/genksyms/genksyms.o
929 YACC scripts/genksyms/parse.tab.c
930 HOSTCC scripts/genksyms/parse.tab.o
931 LEX scripts/genksyms/lex.lex.c
932 YACC scripts/genksyms/parse.tab.h
933 HOSTCC scripts/genksyms/lex.lex.o
934 HOSTLD scripts/genksyms/genksyms
936 HOSTCC scripts/kallsyms
937 HOSTCC scripts/conmakehash
938 HOSTCC scripts/recordmcount
939 HOSTCC scripts/sortextable
940 HOSTCC scripts/asn1_compiler
941 HOSTCC scripts/sign-file
942 HOSTCC scripts/extract-cert
943 CC scripts/mod/empty.o
944 HOSTCC scripts/mod/mk_elfconfig
945 MKELF scripts/mod/elfconfig.h
946 HOSTCC scripts/mod/modpost.o
947 CC scripts/mod/devicetable-offsets.s
948 UPD scripts/mod/devicetable-offsets.h
949 HOSTCC scripts/mod/file2alias.o
950 HOSTCC scripts/mod/sumversion.o
951 HOSTLD scripts/mod/modpost
953 CC arch/x86/kernel/asm-offsets.s
954 CALL scripts/checksyscalls.sh
956 Build the test_pmd application
961 ./usertools/dpdk-setup.sh
975 INSTALL-MAP testpmd.map
976 == Build app/proc_info
979 INSTALL-APP dpdk-procinfo
980 INSTALL-MAP dpdk-procinfo.map
984 INSTALL-APP dpdk-pdump
985 INSTALL-MAP dpdk-pdump.map
986 == Build app/test-crypto-perf
989 CC cperf_options_parsing.o
990 CC cperf_test_vectors.o
991 CC cperf_test_throughput.o
992 CC cperf_test_latency.o
993 CC cperf_test_pmd_cyclecount.o
994 CC cperf_test_verify.o
995 CC cperf_test_vector_parsing.o
996 CC cperf_test_common.o
997 LD dpdk-test-crypto-perf
998 INSTALL-APP dpdk-test-crypto-perf
999 INSTALL-MAP dpdk-test-crypto-perf.map
1000 == Build app/test-eventdev
1005 CC test_order_common.o
1006 CC test_order_queue.o
1008 CC test_perf_common.o
1009 CC test_perf_queue.o
1011 LD dpdk-test-eventdev
1012 INSTALL-APP dpdk-test-eventdev
1013 INSTALL-MAP dpdk-test-eventdev.map
1014 Build complete [x86_64-native-linuxapp-gcc]
1015 Installation cannot run with T defined and DESTDIR undefined
1016 ------------------------------------------------------------------------------
1017 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1018 ------------------------------------------------------------------------------
1020 Press enter to continue ...
1022 Check the VF PCI information:
1025 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1026 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1027 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1029 Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
1032 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1033 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1034 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1035 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1036 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1037 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1038 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1040 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1041 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1042 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1043 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1044 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1045 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1046 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1048 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1049 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1052 Get back to pod and run the test_pmd
1056 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1058 taskset -p --cpu-list 1
1059 pid 1's current affinity list: 2,3,18,19
1061 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1062 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1063 peer=1,"9e:fd:e6:dd:c1:01"
1064 EAL: Detected 32 lcore(s)
1065 EAL: No free hugepages reported in hugepages-2048kB
1066 EAL: Probing VFIO support...
1067 EAL: VFIO support initialized
1068 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1069 EAL: probe driver: 8086:10ed net_ixgbe_vf
1070 EAL: using IOMMU type 1 (Type 1)
1071 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1072 EAL: probe driver: 8086:10ed net_ixgbe_vf
1074 Ports to start sending a burst of packets first
1075 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1076 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1077 Configuring Port 0 (socket 0)
1078 Port 0: 9E:FD:E6:DD:C1:01
1079 Configuring Port 1 (socket 0)
1080 Port 1: 9E:FD:E6:DD:C1:02
1081 Checking link statuses...
1082 Port0 Link Up. speed 10000 Mbps- full-duplex
1083 Port1 Link Up. speed 10000 Mbps- full-duplex
1085 No commandline core given, start packet forwarding
1086 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1088 Logical Core 3 (socket 0) forwards packets on 2 streams:
1089 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1090 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1092 io packet forwarding packets/burst=32
1093 nb forwarding cores=1 - nb forwarding ports=2
1095 CRC stripping enabled
1096 RX queues=1 - RX desc=128 - RX free threshold=32
1097 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1098 TX queues=1 - TX desc=512 - TX free threshold=32
1099 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1100 TX RS bit threshold=32 - TXQ flags=0xf01
1102 CRC stripping enabled
1103 RX queues=1 - RX desc=128 - RX free threshold=32
1104 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1105 TX queues=1 - TX desc=512 - TX free threshold=32
1106 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1107 TX RS bit threshold=32 - TXQ flags=0xf01
1109 Port statistics ====================================
1110 ######################## NIC statistics for port 0 ########################
1111 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1114 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1116 Throughput (since last show)
1119 ############################################################################
1121 ######################## NIC statistics for port 1 ########################
1122 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1125 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1127 Throughput (since last show)
1130 ############################################################################
1132 Port statistics ====================================
1133 ######################## NIC statistics for port 0 ########################
1134 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1137 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1139 Throughput (since last show)
1142 ############################################################################
1154 .. _`StarlingX`: https://docs.starlingx.io/