Implementing OKD (https://www.okd.io/) as an alternative O-Cloud platform.
Issue-ID: INF-441
Change-Id: I92a0b7dd2057642d50a4c931e78c273a27b6b7d3
Signed-off-by: Chris Wheeler <chwheele@redhat.com>
--- /dev/null
+# Overview
+The purpose of the contained Ansible playbook and roles is to deploy an ORAN-compliant O-Cloud instance.
+
+Currently supported Kubernetes platforms and infrastructure targets are:
+
+## Platform
+- [OKD](https://www.okd.io/)
+
+## Infrastructure
+- KVM/libvirtd virtual machine
+
+# Prerequisites
+The following prerequisites must be installed on the host where the playbook will be run (localhost, by default):
+
+# DNS
+To enable network access to cluster services, DNS address records must be defined for the following endpoints:
+
+* api.<cluster>.<domain> (e.g. api.ocloud.example.com)
+* api-int.<cluster>.<domain> (e.g. api-int.ocloud.example.com)
+* *.apps.<cluster>.<domain> (e.g. *.apps.ocloud.example.com)
+
+In the case of all-in-one topology clusters, all addresses must resolve to the machine network IP assigned to the node.
+
+## Ansible
+
+Install Ansible per [Installing Ansible on specific operating systems](https://docs.ansible.com/ansible/latest/installation_guide/installation_distros.html) documentation.
+
+## libvirt/KVM
+
+If deploying the O-Cloud as a virtual machine, the host must be configured as a libvirt/KVM host.
+Instructions for doing so vary by Linux distribution, for example:
+
+- [Fedora](https://docs.fedoraproject.org/en-US/quick-docs/virtualization-getting-started/)
+- [Ubuntu](https://ubuntu.com/server/docs/virtualization-libvirt)
+
+Ensure that the 'libvirt-devel' package is installed, as it is a dependency for the 'libvirt-python' module.
+
+## Python Modules
+
+Install required python modules by installing via the package manager (e.g. yum, dnf, apt) or running:
+
+```
+pip install -r requirements.txt
+```
+
+## Ansible Collections
+
+Install required Ansible collections by running:
+
+```
+ansible-galaxy collection install -r requirements.yml
+```
+
+## Ansible Variables
+
+### General
+
+#### Optional
+The following variables can be set to override deployment defaults:
+- ocloud_infra [default="vm"]: infrastructure target
+- ocloud_platform [default="okd"]: platform target
+- ocloud_topology [default="aio"]: O-Cloud cluster topology
+- ocloud_cluster_name [default="ocloud-{{ ocloud_infra }}-{{ ocloud_platform }}-{{ ocloud_topology }}"]: O-Cloud cluster name
+- ocloud_domain_name [default="example.com"]: O-Cloud domain name
+- ocloud_net_cidr [default="192.168.123.0/24"]: O-Cloud machine network CIDR
+
+### Infrastructure / VM
+
+#### Optional
+The following variables can be set to override defaults for deploying to a VM infrastructure target:
+
+- ocloud_infra_vm_cpus [default=8]: Number of vCPUs to allocate to the VM
+- ocloud_infra_vm_mem_gb [default=24]: Amount of RAM to allocate to the VM in GB
+- ocloud_infra_vm_disk_gb [default=120]: Amount of disk space to allocate to the VM in GB
+- ocloud_infra_vm_disk_dir [default="/var/lib/libvirt/images"]: directory where VM images are stored
+- ocloud_net_name [default="ocloud"]: virtual network name
+- ocloud_net_bridge [default="ocloud-br"]: virtual network bridge name
+- ocloud_net_mac_prefix [default="52:54:00:01:23"]: virtual network MAC prefix
+
+### Platform / OKD
+
+#### Required
+The following Ansible variables must be defined in group_vars/all.yml:
+
+- ocloud_platform_okd_ssh_pubkey: the SSH public key that will be embedded in the OKD install image and used to access deployed nodes
+
+#### Optional
+Optionally, the following variables can be set to override default settings:
+
+- ocloud_platform_okd_release [default=4.14.0-0.okd-2024-01-26-175629]: OKD release, as defined in [OKD releases](https://github.com/okd-project/okd/releases)
+- ocloud_platform_okd_pull_secret [default=None]: pull secret for use with non-public image registries
+
+# Installation
+
+Execute the playbook from the base directory as follows:
+
+```
+ansible-playbook -i inventory playbooks/ocloud.yml
+```
+
+This will deploy the O-Cloud up through the bootstrap phase.
+Continue to monitor the cluster deployment through completion per the Validation section below.
+
+# Validation
+
+## OKD
+
+Set the KUBECONFIG variable to point to the config generated by the agent-based installer, for example:
+
+```
+export KUBECONFIG=/tmp/ansible.6u4ydu5n/cfg/auth/kubeconfig
+```
+
+Monitor the progress of the installation by running the 'oc get nodes', 'oc get clusteroperators', and
+'oc get clusterversion' commands until all nodes are ready and all cluster operators are available, for example:
+
+```
+$ oc get nodes
+NAME STATUS ROLES AGE VERSION
+master-0 Ready control-plane,master,worker 105m v1.27.9+e36e183
+
+$ oc get clusteroperators
+NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
+authentication 4.14.0-0.okd-2024-01-26-175629 True False False 87m
+baremetal 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+cloud-controller-manager 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+cloud-credential 4.14.0-0.okd-2024-01-26-175629 True False False 116m
+cluster-autoscaler 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+config-operator 4.14.0-0.okd-2024-01-26-175629 True False False 92m
+console 4.14.0-0.okd-2024-01-26-175629 True False False 88m
+control-plane-machine-set 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+csi-snapshot-controller 4.14.0-0.okd-2024-01-26-175629 True False False 96m
+dns 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+etcd 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+image-registry 4.14.0-0.okd-2024-01-26-175629 True False False 89m
+ingress 4.14.0-0.okd-2024-01-26-175629 True False False 96m
+insights 4.14.0-0.okd-2024-01-26-175629 True False False 91m
+kube-apiserver 4.14.0-0.okd-2024-01-26-175629 True False False 92m
+kube-controller-manager 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+kube-scheduler 4.14.0-0.okd-2024-01-26-175629 True False False 91m
+kube-storage-version-migrator 4.14.0-0.okd-2024-01-26-175629 True False False 97m
+machine-api 4.14.0-0.okd-2024-01-26-175629 True False False 91m
+machine-approver 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+machine-config 4.14.0-0.okd-2024-01-26-175629 True False False 96m
+marketplace 4.14.0-0.okd-2024-01-26-175629 True False False 96m
+monitoring 4.14.0-0.okd-2024-01-26-175629 True False False 85m
+network 4.14.0-0.okd-2024-01-26-175629 True False False 98m
+node-tuning 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+openshift-apiserver 4.14.0-0.okd-2024-01-26-175629 True False False 89m
+openshift-controller-manager 4.14.0-0.okd-2024-01-26-175629 True False False 90m
+openshift-samples 4.14.0-0.okd-2024-01-26-175629 True False False 90m
+operator-lifecycle-manager 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+operator-lifecycle-manager-catalog 4.14.0-0.okd-2024-01-26-175629 True False False 94m
+operator-lifecycle-manager-packageserver 4.14.0-0.okd-2024-01-26-175629 True False False 93m
+service-ca 4.14.0-0.okd-2024-01-26-175629 True False False 97m
+storage 4.14.0-0.okd-2024-01-26-175629 True False False 92m
+
+$ oc get clusterversion
+NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
+version 4.14.0-0.okd-2024-01-26-175629 True False 83m Cluster version is 4.14.0-0.okd-2024-01-26-175629
+```
+
+# Troubleshooting
+
+## OKD
+
+Refer to [Troubleshooting installation issues](https://docs.okd.io/4.14/installing/installing-troubleshooting.html) for information
+on diagnosing OKD deployment failures.
+
+# Cleanup
+
+## VM
+
+To cleanup a VM-based deployment due to failure, or to prepare to redeploy, execute the following as root on the libvirt/KVM host:
+
+1. Shut down and remove the virtual machine (note that the VM name may differ if the default is overridden):
+
+ ```
+ virsh destroy master-0
+ virsh undefine master-0
+ ```
+
+2. Disable and remove the virtual network (note that the network name may differ if the default is overridden):
+
+ ```
+ virsh net-destroy ocloud
+ virsh net-undefine ocloud
+ ```
+
+3. Remove virtual disk and boot media:
+
+ ```
+ rm /var/lib/libvirt/images/master-0*.{qcow2,iso}
+ ```
--- /dev/null
+[defaults]
+roles_path = ./roles
+remote_tmp = /tmp
+
+[privilege_escalation]
+become_method = sudo
+become_ask_pass = True
--- /dev/null
+---
+ocloud_platform_okd_ssh_pubkey: ~
+# Uncomment to override default release from ocloud_platform_okd role:
+#ocloud_platform_okd_release: ~
--- /dev/null
+---
+# Encrypt with `ansible-vault encrypt vault.yml` before adding secrets
+# Uncomment to override default pull secret from ocloud_platform_okd role:
+#ocloud_platform_okd_pull_secret: ~
--- /dev/null
+---
+ocloud_infra: vm
+ocloud_platform: okd
+ocloud_topology: aio
--- /dev/null
+deployer:
+ hosts:
+ localhost:
+ ansible_connection: local
+
+kvm:
+ hosts:
+ localhost:
+ ansible_connection: local
+
+ocloud:
+ hosts:
+ master-0:
--- /dev/null
+---
+- name: Deploy O-Cloud
+ hosts: ocloud
+ gather_facts: false
+ roles:
+ - ocloud
--- /dev/null
+netaddr
+libvirt-python
--- /dev/null
+collections:
+ - ansible.posix
+ - ansible.utils
+ - community.general
+ - community.libvirt
+ - kubernetes.core
--- /dev/null
+---
+ocloud_infra: "vm"
+ocloud_platform: "okd"
+ocloud_topology: "aio"
+ocloud_cluster_name: "ocloud-{{ ocloud_infra }}-{{ ocloud_platform }}-{{ ocloud_topology }}"
+ocloud_domain_name: "example.com"
+ocloud_net_cidr: "192.168.123.0/24"
--- /dev/null
+---
+- name: Include platform role - {{ ocloud_platform }}
+ ansible.builtin.import_role:
+ name: "ocloud_platform_okd"
+ delegate_to: "{{ groups['deployer'][0] }}"
+ when: ocloud_platform == "okd"
+
+- name: Include infra role - {{ ocloud_infra }}
+ ansible.builtin.import_role:
+ name: "ocloud_infra_vm"
+ delegate_to: "{{ groups['kvm'][0] }}"
+ when: ocloud_infra == "vm"
--- /dev/null
+---
+ocloud_infra_vm_cpus: 8
+ocloud_infra_vm_mem_gb: 24
+ocloud_infra_vm_disk_gb: 120
+ocloud_infra_vm_disk_dir: "/var/lib/libvirt/images"
+ocloud_infra_vm_disk_path: "{{ ocloud_infra_vm_disk_dir }}/{{ inventory_hostname }}.qcow2"
+ocloud_infra_vm_image: "{{ ocloud_infra_vm_disk_dir }}/{{ inventory_hostname }}-image.iso"
+ocloud_net_name: "ocloud"
+ocloud_net_bridge: "ocloud-br"
+ocloud_net_mac_prefix: "52:54:00:01:23"
--- /dev/null
+---
+- name: Define virtual network - {{ ocloud_net_name }}
+ community.libvirt.virt_net:
+ command: define
+ name: "{{ ocloud_net_name }}"
+ xml: '{{ lookup("template", "virt_net.xml.j2") }}'
+ run_once: true
+ become: true
+
+- name: Activate virtual network - {{ ocloud_net_name }}
+ community.libvirt.virt_net:
+ name: "{{ ocloud_net_name }}"
+ state: active
+ run_once: true
+ become: true
+
+- name: Configure virtual network to auto-start - {{ ocloud_net_name }}
+ community.libvirt.virt_net:
+ name: "{{ ocloud_net_name }}"
+ autostart: true
+ run_once: true
+ become: true
+
+- name: Create virtual machine - {{ inventory_hostname }}
+ community.libvirt.virt:
+ command: define
+ xml: '{{ lookup("template", "virt.xml.j2") }}'
+ register: ocloud_infra_vm_definition
+ become: true
+
+- name: Create VM disk
+ ansible.builtin.command:
+ cmd: "qemu-img create -f qcow2 {{ ocloud_infra_vm_disk_path }} {{ ocloud_infra_vm_disk_gb }}G"
+ when: ocloud_infra_vm_definition.changed
+ become: true
+
+- name: Copy platform boot image
+ ansible.builtin.copy:
+ src: "{{ ocloud_platform_image }}"
+ dest: "{{ ocloud_infra_vm_image }}"
+ remote_src: true
+ when: ocloud_infra_vm_definition.changed
+ become: true
+
+- name: Activate virtual machine - {{ inventory_hostname }}
+ community.libvirt.virt:
+ name: "{{ inventory_hostname }}"
+ state: running
+ notify: monitor_platform_deployment
+ become: true
--- /dev/null
+<domain type='kvm'>
+ <name>{{ inventory_hostname }}</name>
+ <memory unit='KiB'>{{ ocloud_infra_vm_mem_gb * 1048576 }}</memory>
+ <currentMemory unit='KiB'>{{ ocloud_infra_vm_mem_gb * 1048576 }}</currentMemory>
+ <vcpu>{{ ocloud_infra_vm_cpus }}</vcpu>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ </os>
+ <features>
+ <acpi/>
+ <apic/>
+ <vmport state='off'/>
+ </features>
+ <cpu mode='host-model' check='partial'/>
+ <clock offset='utc'>
+ <timer name='rtc' tickpolicy='catchup'/>
+ <timer name='pit' tickpolicy='delay'/>
+ <timer name='hpet' present='no'/>
+ </clock>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <pm>
+ <suspend-to-mem enabled='no'/>
+ <suspend-to-disk enabled='no'/>
+ </pm>
+ <devices>
+ <emulator>/usr/libexec/qemu-kvm</emulator>
+ <disk type='file' device='disk'>
+ <driver name='qemu' type='qcow2'/>
+ <source file='{{ ocloud_infra_vm_disk_path }}'/>
+ <target dev='vda' bus='virtio'/>
+ <boot order='1'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
+ </disk>
+ <disk type='file' device='cdrom'>
+ <driver name='qemu' type='raw'/>
+ <source file='{{ ocloud_infra_vm_image }}'/>
+ <target dev='hdb' bus='ide'/>
+ <readonly/>
+ <boot order='2'/>
+ <address type='drive' controller='0' bus='0' target='0' unit='1'/>
+ </disk>
+ <controller type='usb' index='0' model='ich9-ehci1'>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
+ </controller>
+ <controller type='usb' index='0' model='ich9-uhci1'>
+ <master startport='0'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
+ </controller>
+ <controller type='usb' index='0' model='ich9-uhci2'>
+ <master startport='2'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
+ </controller>
+ <controller type='usb' index='0' model='ich9-uhci3'>
+ <master startport='4'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
+ </controller>
+ <controller type='pci' index='0' model='pci-root'/>
+ <controller type='ide' index='0'>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
+ </controller>
+ <controller type='virtio-serial' index='0'>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
+ </controller>
+ <interface type='network'>
+{% for ocloud_host in groups['ocloud'] %}
+{% if ocloud_host == inventory_hostname %}
+ <mac address='{{ ocloud_net_mac_prefix }}:{{ loop.index + 10 }}'/>
+{% endif %}
+{% endfor %}
+ <source network='{{ ocloud_net_name }}'/>
+ <model type='virtio'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+ </interface>
+ <serial type='pty'>
+ <target type='isa-serial' port='0'>
+ <model name='isa-serial'/>
+ </target>
+ </serial>
+ <console type='pty'>
+ <target type='serial' port='0'/>
+ </console>
+ <input type='tablet' bus='usb'>
+ <address type='usb' bus='0' port='1'/>
+ </input>
+ <input type='mouse' bus='ps2'/>
+ <input type='keyboard' bus='ps2'/>
+ <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
+ <listen type='address' address='127.0.0.1'/>
+ </graphics>
+ <audio id='1' type='none'/>
+ <video>
+ <model type='vga' vram='16384' heads='1' primary='yes'/>
+ <alias name='video0'/>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
+ </video>
+ <memballoon model='virtio'>
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
+ </memballoon>
+ </devices>
+</domain>
--- /dev/null
+<network>
+ <name>{{ ocloud_net_name }}</name>
+ <forward mode='nat'>
+ <nat>
+ <port start='1024' end='65535'/>
+ </nat>
+ </forward>
+ <bridge name='{{ ocloud_net_bridge }}' stp='on' delay='0'/>
+ <mac address='{{ ocloud_net_mac_prefix }}:01'/>
+ <ip address='{{ ocloud_net_cidr | ansible.utils.ipmath(1) }}' netmask='{{ ocloud_net_cidr | ansible.utils.ipaddr('netmask') }}'>
+ <dhcp>
+ <range start='{{ ocloud_net_cidr | ansible.utils.ipmath(100) }}' end='{{ ocloud_net_cidr | ansible.utils.ipmath(250) }}'/>
+{% for ocloud_host in groups['ocloud'] %}
+ <host mac='{{ ocloud_net_mac_prefix }}:{{ loop.index + 10 }}' name='{{ ocloud_host }}' ip='{{ ocloud_net_cidr | ansible.utils.ipmath(loop.index + 10) }}'/>
+{% endfor %}
+ </dhcp>
+ </ip>
+</network>
--- /dev/null
+---
+ocloud_platform_okd_pull_secret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}'
+ocloud_platform_okd_ssh_pubkey: ~
+ocloud_platform_okd_release: "4.14.0-0.okd-2024-01-26-175629"
+ocloud_platform_okd_base_url: "https://github.com/okd-project/okd/releases/download"
--- /dev/null
+---
+- name: Monitor OKD platform bootstrap
+ ansible.builtin.command:
+ cmd: "openshift-install agent wait-for bootstrap-complete --log-level=info --dir {{ ocloud_platform_okd_staging_dir['path'] }}/cfg"
+ environment:
+ PATH: "{{ ocloud_platform_okd_staging_dir['path'] }}/bin:{{ ansible_env.PATH }}"
+ listen: monitor_platform_deployment
--- /dev/null
+---
+- name: Gather facts
+ ansible.builtin.setup:
+ gather_subset: all
+
+- name: Create staging dir for OKD installation
+ ansible.builtin.tempfile:
+ state: directory
+ register: ocloud_platform_okd_staging_dir
+
+- name: Create staging subdirs
+ ansible.builtin.file:
+ path: "{{ ocloud_platform_okd_staging_dir['path'] }}/{{ item }}"
+ state: directory
+ loop:
+ - bin
+ - cfg
+
+- name: Download OKD binaries
+ ansible.builtin.get_url:
+ url: "{{ ocloud_platform_okd_base_url }}/{{ ocloud_platform_okd_release }}/{{ item }}-{{ ocloud_platform_okd_release }}.tar.gz"
+ dest: "{{ ocloud_platform_okd_staging_dir['path'] }}"
+ loop:
+ - openshift-client-linux
+ - openshift-install-linux
+
+- name: Extract OKD binaries
+ ansible.builtin.unarchive:
+ src: "{{ ocloud_platform_okd_staging_dir['path'] }}/{{ item }}-{{ ocloud_platform_okd_release }}.tar.gz"
+ dest: "{{ ocloud_platform_okd_staging_dir['path'] }}/bin"
+ remote_src: true
+ loop:
+ - openshift-client-linux
+ - openshift-install-linux
+
+- name: Template OKD configs
+ ansible.builtin.template:
+ src: "{{ item }}.j2"
+ dest: "{{ ocloud_platform_okd_staging_dir['path'] }}/cfg/{{ item }}"
+ loop:
+ - agent-config.yaml
+ - install-config.yaml
+
+- name: Generate OKD agent-based installer image
+ ansible.builtin.shell:
+ cmd: "openshift-install agent create image --dir {{ ocloud_platform_okd_staging_dir['path'] }}/cfg"
+ environment:
+ PATH: "{{ ocloud_platform_okd_staging_dir['path'] }}/bin:{{ ansible_env.PATH }}"
+
+- debug:
+ msg: |
+ OKD agent-based installer image has been created as {{ ocloud_platform_okd_staging_dir['path'] }}/cfg/agent.x86_64.iso
+ kubeconfig and kubeadmin-password can be found under {{ ocloud_platform_okd_staging_dir['path'] }}/cfg/auth/
+
+- ansible.builtin.set_fact:
+ ocloud_platform_image: "{{ ocloud_platform_okd_staging_dir['path'] }}/cfg/agent.x86_64.iso"
--- /dev/null
+apiVersion: v1beta1
+kind: AgentConfig
+metadata:
+ name: {{ ocloud_cluster_name }}
+rendezvousIP: {{ ocloud_net_cidr | ansible.utils.ipmath(11) }}
--- /dev/null
+apiVersion: v1
+baseDomain: {{ ocloud_domain_name }}
+compute:
+- name: worker
+ replicas: 0
+controlPlane:
+ name: master
+ replicas: 1
+metadata:
+ name: {{ ocloud_cluster_name }}
+networking:
+ clusterNetwork:
+ - cidr: 10.128.0.0/14
+ hostPrefix: 23
+ networkType: OVNKubernetes
+ machineNetwork:
+ - cidr: {{ ocloud_net_cidr }}
+ serviceNetwork:
+ - 172.30.0.0/16
+platform:
+ none: {}
+bootstrapInPlace:
+ installationDisk: /dev/vda
+fips: false
+pullSecret: '{{ ocloud_platform_okd_pull_secret }}'
+sshKey: '{{ ocloud_platform_okd_ssh_pubkey }}'