.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 .. Copyright (C) 2019 Wind River Systems, Inc. Installation Guide ================== .. contents:: :depth: 3 :local: Abstract -------- This document describes how to install O-RAN INF image, example configuration for better real time performance, and example deployment of Kubernetes cluster and plugins. The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux and container technology. Version history +--------------------+--------------------+--------------------+--------------------+ | **Date** | **Ver.** | **Author** | **Comment** | | | | | | +--------------------+--------------------+--------------------+--------------------+ | 2019-11-02 | 1.0.0 | Jackie Huang | Initail version | | | | | | +--------------------+--------------------+--------------------+--------------------+ | | | | | | | | | | +--------------------+--------------------+--------------------+--------------------+ | | | | | | | | | | +--------------------+--------------------+--------------------+--------------------+ Preface ------- Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide. Hardware Requirements --------------------- Following minimum hardware requirements must be met for installation of O-RAN INF image: +--------------------+----------------------------------------------------+ | **HW Aspect** | **Requirement** | | | | +--------------------+----------------------------------------------------+ | **# of servers** | 1 | +--------------------+----------------------------------------------------+ | **CPU** | 2 | | | | +--------------------+----------------------------------------------------+ | **RAM** | 4G | | | | +--------------------+----------------------------------------------------+ | **Disk** | 20G | | | | +--------------------+----------------------------------------------------+ | **NICs** | 1 | | | | +--------------------+----------------------------------------------------+ Software Installation and Deployment ------------------------------------ 1. Installation from the O-RAN INF ISO image ```````````````````````````````````````````` - Please see the README.md file for how to build the image. - The Image is a live ISO image with CLI installer: oran-image-inf-host-intel-x86-64.iso 1.1 Burn the image to the USB device '''''''''''''''''''''''''''''''''''' - Assume the the usb device is /dev/sdX here :: $ sudo dd if=/path/to/oran-image-inf-host-intel-x86-64.iso of=/dev/sdX bs=1M 1.2 Insert the USB device in the target to be booted. ''''''''''''''''''''''''''''''''''''''''''''''''''''' 1.3 Reboot the target from the USB device. '''''''''''''''''''''''''''''''''''''''''' 1.4 Select "Graphics console install" or "Serial console install" and press ENTER ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' 1.5 Select the hard disk and press ENTER '''''''''''''''''''''''''''''''''''''''' Notes: In this installer, you can only select which hard disk to install, the whole disk will be used and partitioned automatically. - e.g. insert "sda" and press ENTER 1.6 Remove the USB device and press ENTER to reboot ''''''''''''''''''''''''''''''''''''''''''''''''''' 2. Configuration for better real time performance ````````````````````````````````````````````````` Notes: Some of the tuning options are machine specific or depend on use cases, like the hugepages, isolcpus, rcu_nocbs, kthread_cpus, irqaffinity, nohz_full and so on, please do not just copy and past. - Edit the grub.cfg with the following example tuning options :: # Notes: the grub.cfg file path is different for legacy and UEFI mode # For legacy mode: /boot/grub/grub.cfg # For UEFI mode: /boot/EFI/BOOT/grub.cfg grub_cfg="/boot/grub/grub.cfg" #grub_cfg="/boot/EFI/BOOT/grub.cfg" # In this example, 1-16 cores are isolated for real time processes root@intel-x86-64:~# rt_tuning="crashkernel=auto biosdevname=0 iommu=pt usbcore.autosuspend=-1 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on cgroup_enable=memory skew_tick=1 hugepagesz=1G hugepages=4 default_hugepagesz=1G isolcpus=1-16 rcu_nocbs=1-16 kthread_cpus=0 irqaffinity=0 nohz=on nohz_full=1-16 intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable nosoftlockup idle=poll mce=ignore_ce" # optional to add the console setting root@intel-x86-64:~# console="console=ttyS0,115200" root@intel-x86-64:~# sed -i "/linux / s/$/ $console $rt_tuning/" $grub_cfg - Reboot the target :: root@intel-x86-64:~# reboot 3. Kubernetes cluster and plugins deployment instructions (All-in-one) `````````````````````````````````````````````````````````````````````` This instruction will show you how to deploy kubernetes cluster and plugins in an all-in-one example scenario after the above installation. 3.1 Change the hostname (Optional) '''''''''''''''''''''''''''''''''' :: # Assuming the hostname is oran-aio, ip address is # please DO NOT copy and paste, use your actaul hostname and ip address root@intel-x86-64:~# echo oran-aio > /etc/hostname root@intel-x86-64:~# export AIO_HOST_IP="" root@intel-x86-64:~# echo "$AIO_HOST_IP oran-aio" >> /etc/hosts 3.2 Disable swap for Kubernetes ''''''''''''''''''''''''''''''' :: root@intel-x86-64:~# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab root@intel-x86-64:~# systemctl mask dev-sda4.swap 3.3 Set the proxy for docker (Optional) ''''''''''''''''''''''''''''''''''''''' - If you are under a firewall, you may need to set the proxy for docker to pull images :: root@intel-x86-64:~# HTTP_PROXY="http://:" root@intel-x86-64:~# mkdir /etc/systemd/system/docker.service.d/ root@intel-x86-64:~# cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=$HTTP_PROXY" "NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,$AIO_HOST_IP,10.244.0.0/16" EOF 3.4 Reboot the target ''''''''''''''''''''' :: root@intel-x86-64:~# reboot 3.5 Initialize kubernetes cluster master '''''''''''''''''''''''''''''''''''''''' :: root@oran-aio:~# kubeadm init --kubernetes-version v1.15.2 --pod-network-cidr=10.244.0.0/16 root@oran-aio:~# mkdir -p $HOME/.kube root@oran-aio:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config root@oran-aio:~# chown $(id -u):$(id -g) $HOME/.kube/config 3.6 Make the master also works as a worker node ''''''''''''''''''''''''''''''''''''''''''''''' :: root@oran-aio:~# kubectl taint nodes oran-aio node-role.kubernetes.io/master- 3.7 Deploy flannel '''''''''''''''''' :: root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/flannel/kube-flannel.yml Check that the aio node is ready after flannel is successfully deployed and running :: root@oran-aio:~# kubectl get pods --all-namespaces |grep flannel kube-system kube-flannel-ds-amd64-bwt52 1/1 Running 0 3m24s root@oran-aio:~# kubectl get nodes NAME STATUS ROLES AGE VERSION oran-aio Ready master 3m17s v1.15.2-dirty 3.8 Deploy kubernetes dashboard ''''''''''''''''''''''''''''''' Deploy kubernetes dashboard :: root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard-admin.rbac.yaml root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard.yaml Verify that the dashboard is up and running :: # Check the pod for dashboard root@oran-aio:~# kubectl get pods --all-namespaces |grep dashboard kube-system kubernetes-dashboard-5b67bf4d5f-ghg4f 1/1 Running 0 64s Access the dashboard UI in a web browser with the url: https://:30443 - For detail usage, please refer to `Doc for dashboard`_ .. _`Doc for dashboard`: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/ 3.9 Deploy Multus-CNI ''''''''''''''''''''' :: root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/multus-cni/multus-daemonset.yml Verify that the multus-cni is up and running :: root@oran-aio:~# kubectl get pods --all-namespaces | grep -i multus kube-system kube-multus-ds-amd64-hjpk4 1/1 Running 0 7m34s - For further validating, please refer to the `Multus-CNI quick start`_ .. _`Multus-CNI quick start`: https://github.com/intel/multus-cni/blob/master/doc/quickstart.md 3.10 Deploy NFD (node-feature-discovery) '''''''''''''''''''''''''''''''''''''''' :: root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-master.yaml root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-worker-daemonset.yaml Verify that nfd-master and nfd-worker are up and running :: root@oran-aio:~# kubectl get pods --all-namespaces |grep nfd default nfd-master-7v75k 1/1 Running 0 91s default nfd-worker-xn797 1/1 Running 0 24s Verify that the node is labeled by nfd: :: root@oran-aio:~# kubectl describe nodes|grep feature.node.kubernetes feature.node.kubernetes.io/cpu-cpuid.AESNI=true feature.node.kubernetes.io/cpu-cpuid.AVX=true feature.node.kubernetes.io/cpu-cpuid.AVX2=true (...snip...) 3.11 Deploy CMK (CPU-Manager-for-Kubernetes) '''''''''''''''''''''''''''''''''''''''''''' Build the CMK docker image :: root@oran-aio:~# cd /opt/kubernetes_plugins/cpu-manager-for-kubernetes/ root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# make Verify that the cmk docker images is built successfully :: root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# docker images|grep cmk cmk v1.3.1 3fec5f753b05 44 minutes ago 765MB Edit the template yaml file for your deployment: - The template file is: /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml - The options you may need to change: :: # You can change the value for the following env: env: - name: HOST_LIST # Change this to modify the the host list to be initialized value: "oran-aio" - name: NUM_EXCLUSIVE_CORES # Change this to modify the value passed to `--num-exclusive-cores` flag value: "4" - name: NUM_SHARED_CORES # Change this to modify the value passed to `--num-shared-cores` flag value: "1" - name: CMK_IMG # Change his ONLY if you built the docker images with a different tag name value: "cmk:v1.3.1" Or you can also refer to `CMK operator manual`_ .. _`CMK operator manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/operator.md Depoly CMK from yaml files :: root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-rbac-rules.yaml root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-serviceaccount.yaml root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml Verify that the cmk cluster init completed and the pods for nodereport and webhook deployment are up and running :: root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# kubectl get pods --all-namespaces |grep cmk default cmk-cluster-init-pod 0/1 Completed 0 11m default cmk-init-install-discover-pod-oran-aio 0/2 Completed 0 10m default cmk-reconcile-nodereport-ds-oran-aio-qbdqb 2/2 Running 0 10m default cmk-webhook-deployment-6f9dd7dfb6-2lj2p 1/1 Running 0 10m - For detail usage, please refer to `CMK user manual`_ .. _`CMK user manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/user.md References ---------- - `Flannel`_ - `Doc for dashboard`_ - `Multus-CNI quick start`_ - `CMK operator manual`_ - `CMK user manual`_ .. _`Flannel`: https://github.com/coreos/flannel/blob/master/README.md