5b6a81730670f959900bb3ae75720baa0c019f60
[pti/rtp.git] / installation-guide.rst
1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. Copyright (C) 2019 Wind River Systems, Inc.
4
5
6 Installation Guide
7 ==================
8
9 .. contents::
10    :depth: 3
11    :local:
12
13 Abstract
14 --------
15
16 This document describes how to install O-RAN INF image, example configuration for better
17 real time performance, and example deployment of Kubernetes cluster and plugins. 
18
19 The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20 and container technology.
21
22 Version history
23
24 +--------------------+--------------------+--------------------+--------------------+
25 | **Date**           | **Ver.**           | **Author**         | **Comment**        |
26 |                    |                    |                    |                    |
27 +--------------------+--------------------+--------------------+--------------------+
28 | 2019-11-02         | 1.0.0              | Jackie Huang       | Initail version    |
29 |                    |                    |                    |                    |
30 +--------------------+--------------------+--------------------+--------------------+
31 |                    |                    |                    |                    |
32 |                    |                    |                    |                    |
33 +--------------------+--------------------+--------------------+--------------------+
34 |                    |                    |                    |                    |
35 |                    |                    |                    |                    |
36 +--------------------+--------------------+--------------------+--------------------+
37
38
39 Preface
40 -------
41
42 Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
43
44
45 Hardware Requirements
46 ---------------------
47
48 Following minimum hardware requirements must be met for installation of O-RAN INF image:
49
50 +--------------------+----------------------------------------------------+
51 | **HW Aspect**      | **Requirement**                                    |
52 |                    |                                                    |
53 +--------------------+----------------------------------------------------+
54 | **# of servers**   | 1                                                  |
55 +--------------------+----------------------------------------------------+
56 | **CPU**            | 2                                                  |
57 |                    |                                                    |
58 +--------------------+----------------------------------------------------+
59 | **RAM**            | 4G                                                 |
60 |                    |                                                    |
61 +--------------------+----------------------------------------------------+
62 | **Disk**           | 20G                                                |
63 |                    |                                                    |
64 +--------------------+----------------------------------------------------+
65 | **NICs**           | 1                                                  |
66 |                    |                                                    |
67 +--------------------+----------------------------------------------------+
68
69
70
71 Software Installation and Deployment
72 ------------------------------------
73
74 1. Installation from the O-RAN INF ISO image
75 ````````````````````````````````````````````
76
77 - Please see the README.md file for how to build the image.
78 - The Image is a live ISO image with CLI installer: oran-image-inf-host-intel-x86-64.iso
79
80 1.1 Burn the image to the USB device
81 ''''''''''''''''''''''''''''''''''''
82
83 - Assume the the usb device is /dev/sdX here
84
85 ::
86
87   $ sudo dd if=/path/to/oran-image-inf-host-intel-x86-64.iso of=/dev/sdX bs=1M
88
89 1.2 Insert the USB device in the target to be booted.
90 '''''''''''''''''''''''''''''''''''''''''''''''''''''
91
92 1.3 Reboot the target from the USB device.
93 ''''''''''''''''''''''''''''''''''''''''''
94
95 1.4 Select "Graphics console install" or "Serial console install" and press ENTER
96 '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
97
98 1.5 Select the hard disk and press ENTER
99 ''''''''''''''''''''''''''''''''''''''''
100
101 Notes: In this installer, you can only select which hard disk to install, the whole disk will be used and partitioned automatically.
102
103 - e.g. insert "sda" and press ENTER
104
105 1.6 Remove the USB device and press ENTER to reboot
106 '''''''''''''''''''''''''''''''''''''''''''''''''''
107
108 2. Configuration for better real time performance
109 `````````````````````````````````````````````````
110
111 Notes: Some of the tuning options are machine specific or depend on use cases,
112 like the hugepages, isolcpus, rcu_nocbs, kthread_cpus, irqaffinity, nohz_full and
113 so on, please do not just copy and past.
114
115 - Edit the grub.cfg with the following example tuning options
116
117 ::
118
119   # Notes: the grub.cfg file path is different for legacy and UEFI mode
120   #   For legacy mode: /boot/grub/grub.cfg
121   #   For UEFI mode: /boot/EFI/BOOT/grub.cfg
122
123   grub_cfg="/boot/grub/grub.cfg"
124   #grub_cfg="/boot/EFI/BOOT/grub.cfg"
125
126   # In this example, 1-16 cores are isolated for real time processes
127   root@intel-x86-64:~# rt_tuning="crashkernel=auto biosdevname=0 iommu=pt usbcore.autosuspend=-1 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on cgroup_enable=memory skew_tick=1 hugepagesz=1G hugepages=4 default_hugepagesz=1G isolcpus=1-16 rcu_nocbs=1-16 kthread_cpus=0 irqaffinity=0 nohz=on nohz_full=1-16 intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable nosoftlockup idle=poll mce=ignore_ce"
128
129   # optional to add the console setting
130   root@intel-x86-64:~# console="console=ttyS0,115200"
131
132   root@intel-x86-64:~# sed -i "/linux / s/$/ $console $rt_tuning/" $grub_cfg
133
134
135 - Reboot the target
136
137 ::
138
139   root@intel-x86-64:~# reboot
140
141 3. Kubernetes cluster and plugins deployment instructions (All-in-one)
142 ``````````````````````````````````````````````````````````````````````
143 This instruction will show you how to deploy kubernetes cluster and plugins in an all-in-one example scenario after the above installation.
144
145 3.1 Change the hostname (Optional)
146 ''''''''''''''''''''''''''''''''''
147
148 ::
149
150   # Assuming the hostname is oran-aio, ip address is <aio_host_ip>
151   # please DO NOT copy and paste, use your actaul hostname and ip address
152   root@intel-x86-64:~# echo oran-aio > /etc/hostname
153   root@intel-x86-64:~# export AIO_HOST_IP="<aio_host_ip>"
154   root@intel-x86-64:~# echo "$AIO_HOST_IP oran-aio" >> /etc/hosts
155
156 3.2 Disable swap for Kubernetes
157 '''''''''''''''''''''''''''''''
158
159 ::
160
161   root@intel-x86-64:~# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
162   root@intel-x86-64:~# systemctl mask dev-sda4.swap
163
164 3.3 Set the proxy for docker (Optional)
165 '''''''''''''''''''''''''''''''''''''''
166
167 - If you are under a firewall, you may need to set the proxy for docker to pull images
168
169 ::
170
171   root@intel-x86-64:~# HTTP_PROXY="http://<your_proxy_server_ip>:<port>"
172   root@intel-x86-64:~# mkdir /etc/systemd/system/docker.service.d/
173   root@intel-x86-64:~# cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf
174   [Service]
175   Environment="HTTP_PROXY=$HTTP_PROXY" "NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,$AIO_HOST_IP,10.244.0.0/16"
176   EOF
177
178 3.4 Reboot the target
179 '''''''''''''''''''''
180
181 ::
182
183   root@intel-x86-64:~# reboot
184
185 3.5 Initialize kubernetes cluster master
186 ''''''''''''''''''''''''''''''''''''''''
187
188 ::
189
190   root@oran-aio:~# kubeadm init --kubernetes-version v1.15.2 --pod-network-cidr=10.244.0.0/16
191   root@oran-aio:~# mkdir -p $HOME/.kube
192   root@oran-aio:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
193   root@oran-aio:~# chown $(id -u):$(id -g) $HOME/.kube/config
194
195 3.6 Make the master also works as a worker node
196 '''''''''''''''''''''''''''''''''''''''''''''''
197
198 ::
199
200   root@oran-aio:~# kubectl taint nodes oran-aio node-role.kubernetes.io/master-
201
202 3.7 Deploy flannel
203 ''''''''''''''''''
204
205 ::
206
207   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/flannel/kube-flannel.yml
208
209 Check that the aio node is ready after flannel is successfully deployed and running
210
211 ::
212
213   root@oran-aio:~# kubectl get pods --all-namespaces |grep flannel
214   kube-system   kube-flannel-ds-amd64-bwt52        1/1     Running   0          3m24s
215
216   root@oran-aio:~# kubectl get nodes
217   NAME       STATUS   ROLES    AGE     VERSION
218   oran-aio   Ready    master   3m17s   v1.15.2-dirty
219
220 3.8 Deploy kubernetes dashboard
221 '''''''''''''''''''''''''''''''
222
223 Deploy kubernetes dashboard
224
225 ::
226
227   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard-admin.rbac.yaml
228   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard.yaml
229
230 Verify that the dashboard is up and running
231
232 ::
233
234   # Check the pod for dashboard
235   root@oran-aio:~# kubectl get pods --all-namespaces |grep dashboard
236   kube-system   kubernetes-dashboard-5b67bf4d5f-ghg4f   1/1     Running   0          64s
237
238 Access the dashboard UI in a web browser with the https url, port number is 30443.
239
240 - For detail usage, please refer to `Doc for dashboard`_
241
242 .. _`Doc for dashboard`: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/
243
244 3.9 Deploy Multus-CNI
245 '''''''''''''''''''''
246
247 ::
248
249   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/multus-cni/multus-daemonset.yml
250
251 Verify that the multus-cni is up and running
252
253 ::
254
255   root@oran-aio:~# kubectl get pods --all-namespaces | grep -i multus
256   kube-system   kube-multus-ds-amd64-hjpk4              1/1     Running   0          7m34s
257
258 - For further validating, please refer to the `Multus-CNI quick start`_
259
260 .. _`Multus-CNI quick start`: https://github.com/intel/multus-cni/blob/master/doc/quickstart.md
261
262 3.10 Deploy NFD (node-feature-discovery)
263 ''''''''''''''''''''''''''''''''''''''''
264
265 ::
266
267   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-master.yaml
268   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-worker-daemonset.yaml
269
270 Verify that nfd-master and nfd-worker are up and running
271
272 ::
273
274   root@oran-aio:~# kubectl get pods --all-namespaces |grep nfd
275   default       nfd-master-7v75k                        1/1     Running   0          91s
276   default       nfd-worker-xn797                        1/1     Running   0          24s
277
278 Verify that the node is labeled by nfd:
279
280 ::
281
282   root@oran-aio:~# kubectl describe nodes|grep feature.node.kubernetes
283                      feature.node.kubernetes.io/cpu-cpuid.AESNI=true
284                      feature.node.kubernetes.io/cpu-cpuid.AVX=true
285                      feature.node.kubernetes.io/cpu-cpuid.AVX2=true
286                      (...snip...)
287
288 3.11 Deploy CMK (CPU-Manager-for-Kubernetes)
289 ''''''''''''''''''''''''''''''''''''''''''''
290
291 Build the CMK docker image
292
293 ::
294
295   root@oran-aio:~# cd /opt/kubernetes_plugins/cpu-manager-for-kubernetes/
296   root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# make
297
298 Verify that the cmk docker images is built successfully
299
300 ::
301
302   root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# docker images|grep cmk
303   cmk          v1.3.1              3fec5f753b05        44 minutes ago      765MB
304
305 Edit the template yaml file for your deployment:
306   - The template file is: /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml
307   - The options you may need to change:
308
309 ::
310
311   # You can change the value for the following env:
312   env:
313   - name: HOST_LIST
314     # Change this to modify the the host list to be initialized
315     value: "oran-aio"
316   - name: NUM_EXCLUSIVE_CORES
317     # Change this to modify the value passed to `--num-exclusive-cores` flag
318     value: "4"
319   - name: NUM_SHARED_CORES
320     # Change this to modify the value passed to `--num-shared-cores` flag
321     value: "1"
322   - name: CMK_IMG
323     # Change his ONLY if you built the docker images with a different tag name
324     value: "cmk:v1.3.1"
325
326 Or you can also refer to `CMK operator manual`_
327
328 .. _`CMK operator manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/operator.md
329
330
331 Depoly CMK from yaml files
332
333 ::
334
335   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-rbac-rules.yaml
336   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-serviceaccount.yaml
337   root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml
338
339 Verify that the cmk cluster init completed and the pods for nodereport and webhook deployment are up and running
340
341 ::
342
343   root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# kubectl get pods --all-namespaces |grep cmk
344   default       cmk-cluster-init-pod                         0/1     Completed   0          11m
345   default       cmk-init-install-discover-pod-oran-aio       0/2     Completed   0          10m
346   default       cmk-reconcile-nodereport-ds-oran-aio-qbdqb   2/2     Running     0          10m
347   default       cmk-webhook-deployment-6f9dd7dfb6-2lj2p      1/1     Running     0          10m
348
349 - For detail usage, please refer to `CMK user manual`_
350
351 .. _`CMK user manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/user.md
352
353 References
354 ----------
355
356 - `Flannel`_
357 - `Doc for dashboard`_
358 - `Multus-CNI quick start`_
359 - `CMK operator manual`_
360 - `CMK user manual`_
361
362 .. _`Flannel`: https://github.com/coreos/flannel/blob/master/README.md