X-Git-Url: https://gerrit.o-ran-sc.org/r/gitweb?a=blobdiff_plain;f=XTesting%2Fkubespray%2Fdocs%2Fansible.md;fp=XTesting%2Fkubespray%2Fdocs%2Fansible.md;h=7d48a7fdf615762d66b8b468864267af9a99d40c;hb=31af17bb5935b722dcf59d5800aaff9e789cfa93;hp=0000000000000000000000000000000000000000;hpb=c8bda4f07b7e87beb2aa3d8729f9b0b456d4da6f;p=it%2Ftest.git diff --git a/XTesting/kubespray/docs/ansible.md b/XTesting/kubespray/docs/ansible.md new file mode 100644 index 0000000..7d48a7f --- /dev/null +++ b/XTesting/kubespray/docs/ansible.md @@ -0,0 +1,303 @@ +# Ansible + +## Installing Ansible + +Kubespray supports multiple ansible versions and ships different `requirements.txt` files for them. +Depending on your available python version you may be limited in chooding which ansible version to use. + +It is recommended to deploy the ansible version used by kubespray into a python virtual environment. + +```ShellSession +VENVDIR=kubespray-venv +KUBESPRAYDIR=kubespray +ANSIBLE_VERSION=2.12 +virtualenv --python=$(which python3) $VENVDIR +source $VENVDIR/bin/activate +cd $KUBESPRAYDIR +pip install -U -r requirements-$ANSIBLE_VERSION.txt +test -f requirements-$ANSIBLE_VERSION.yml && \ + ansible-galaxy role install -r requirements-$ANSIBLE_VERSION.yml && \ + ansible-galaxy collection -r requirements-$ANSIBLE_VERSION.yml +``` + +### Ansible Python Compatibility + +Based on the table below and the available python version for your ansible host you should choose the appropriate ansible version to use with kubespray. + +| Ansible Version | Python Version | +| --------------- | -------------- | +| 2.11 | 2.7,3.5-3.9 | +| 2.12 | 3.8-3.10 | + +## Inventory + +The inventory is composed of 3 groups: + +* **kube_node** : list of kubernetes nodes where the pods will run. +* **kube_control_plane** : list of servers where kubernetes control plane components (apiserver, scheduler, controller) will run. +* **etcd**: list of servers to compose the etcd server. You should have at least 3 servers for failover purpose. + +Note: do not modify the children of _k8s_cluster_, like putting +the _etcd_ group into the _k8s_cluster_, unless you are certain +to do that and you have it fully contained in the latter: + +```ShellSession +etcd ⊂ k8s_cluster => kube_node ∩ etcd = etcd +``` + +When _kube_node_ contains _etcd_, you define your etcd cluster to be as well schedulable for Kubernetes workloads. +If you want it a standalone, make sure those groups do not intersect. +If you want the server to act both as control-plane and node, the server must be defined +on both groups _kube_control_plane_ and _kube_node_. If you want a standalone and +unschedulable control plane, the server must be defined only in the _kube_control_plane_ and +not _kube_node_. + +There are also two special groups: + +* **calico_rr** : explained for [advanced Calico networking cases](/docs/calico.md) +* **bastion** : configure a bastion host if your nodes are not directly reachable + +Below is a complete inventory example: + +```ini +## Configure 'ip' variable to bind kubernetes services on a +## different ip than the default iface +node1 ansible_host=95.54.0.12 ip=10.3.0.1 +node2 ansible_host=95.54.0.13 ip=10.3.0.2 +node3 ansible_host=95.54.0.14 ip=10.3.0.3 +node4 ansible_host=95.54.0.15 ip=10.3.0.4 +node5 ansible_host=95.54.0.16 ip=10.3.0.5 +node6 ansible_host=95.54.0.17 ip=10.3.0.6 + +[kube_control_plane] +node1 +node2 + +[etcd] +node1 +node2 +node3 + +[kube_node] +node2 +node3 +node4 +node5 +node6 + +[k8s_cluster:children] +kube_node +kube_control_plane +``` + +## Group vars and overriding variables precedence + +The group variables to control main deployment options are located in the directory ``inventory/sample/group_vars``. +Optional variables are located in the `inventory/sample/group_vars/all.yml`. +Mandatory variables that are common for at least one role (or a node group) can be found in the +`inventory/sample/group_vars/k8s_cluster.yml`. +There are also role vars for docker, kubernetes preinstall and control plane roles. +According to the [ansible docs](https://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable), +those cannot be overridden from the group vars. In order to override, one should use +the `-e` runtime flags (most simple way) or other layers described in the docs. + +Kubespray uses only a few layers to override things (or expect them to +be overridden for roles): + +Layer | Comment +------|-------- +**role defaults** | provides best UX to override things for Kubespray deployments +inventory vars | Unused +**inventory group_vars** | Expects users to use ``all.yml``,``k8s_cluster.yml`` etc. to override things +inventory host_vars | Unused +playbook group_vars | Unused +playbook host_vars | Unused +**host facts** | Kubespray overrides for internal roles' logic, like state flags +play vars | Unused +play vars_prompt | Unused +play vars_files | Unused +registered vars | Unused +set_facts | Kubespray overrides those, for some places +**role and include vars** | Provides bad UX to override things! Use extra vars to enforce +block vars (only for tasks in block) | Kubespray overrides for internal roles' logic +task vars (only for the task) | Unused for roles, but only for helper scripts +**extra vars** (always win precedence) | override with ``ansible-playbook -e @foo.yml`` + +## Ansible tags + +The following tags are defined in playbooks: + +| Tag name | Used for +|--------------------------------|--------- +| annotate | Create kube-router annotation +| apps | K8s apps definitions +| asserts | Check tasks for download role +| aws-ebs-csi-driver | Configuring csi driver: aws-ebs +| azure-csi-driver | Configuring csi driver: azure +| bastion | Setup ssh config for bastion +| bootstrap-os | Anything related to host OS configuration +| calico | Network plugin Calico +| calico_rr | Configuring Calico route reflector +| canal | Network plugin Canal +| cephfs-provisioner | Configuring CephFS +| cert-manager | Configuring certificate manager for K8s +| cilium | Network plugin Cilium +| cinder-csi-driver | Configuring csi driver: cinder +| client | Kubernetes clients role +| cloud-provider | Cloud-provider related tasks +| cluster-roles | Configuring cluster wide application (psp ...) +| cni | CNI plugins for Network Plugins +| containerd | Configuring containerd engine runtime for hosts +| container_engine_accelerator | Enable nvidia accelerator for runtimes +| container-engine | Configuring container engines +| container-runtimes | Configuring container runtimes +| coredns | Configuring coredns deployment +| crio | Configuring crio container engine for hosts +| crun | Configuring crun runtime +| csi-driver | Configuring csi driver +| dashboard | Installing and configuring the Kubernetes Dashboard +| dns | Remove dns entries when resetting +| docker | Configuring docker engine runtime for hosts +| download | Fetching container images to a delegate host +| etcd | Configuring etcd cluster +| etcd-secrets | Configuring etcd certs/keys +| etchosts | Configuring /etc/hosts entries for hosts +| external-cloud-controller | Configure cloud controllers +| external-openstack | Cloud controller : openstack +| external-provisioner | Configure external provisioners +| external-vsphere | Cloud controller : vsphere +| facts | Gathering facts and misc check results +| files | Remove files when resetting +| flannel | Network plugin flannel +| gce | Cloud-provider GCP +| gcp-pd-csi-driver | Configuring csi driver: gcp-pd +| gvisor | Configuring gvisor runtime +| helm | Installing and configuring Helm +| ingress-controller | Configure ingress controllers +| ingress_alb | AWS ALB Ingress Controller +| init | Windows kubernetes init nodes +| iptables | Flush and clear iptable when resetting +| k8s-pre-upgrade | Upgrading K8s cluster +| k8s-secrets | Configuring K8s certs/keys +| k8s-gen-tokens | Configuring K8s tokens +| kata-containers | Configuring kata-containers runtime +| krew | Install and manage krew +| kubeadm | Roles linked to kubeadm tasks +| kube-apiserver | Configuring static pod kube-apiserver +| kube-controller-manager | Configuring static pod kube-controller-manager +| kube-vip | Installing and configuring kube-vip +| kubectl | Installing kubectl and bash completion +| kubelet | Configuring kubelet service +| kube-ovn | Network plugin kube-ovn +| kube-router | Network plugin kube-router +| kube-proxy | Configuring static pod kube-proxy +| localhost | Special steps for the localhost (ansible runner) +| local-path-provisioner | Configure External provisioner: local-path +| local-volume-provisioner | Configure External provisioner: local-volume +| macvlan | Network plugin macvlan +| master | Configuring K8s master node role +| metallb | Installing and configuring metallb +| metrics_server | Configuring metrics_server +| netchecker | Installing netchecker K8s app +| network | Configuring networking plugins for K8s +| mounts | Umount kubelet dirs when reseting +| multus | Network plugin multus +| nginx | Configuring LB for kube-apiserver instances +| node | Configuring K8s minion (compute) node role +| nodelocaldns | Configuring nodelocaldns daemonset +| node-label | Tasks linked to labeling of nodes +| node-webhook | Tasks linked to webhook (grating access to resources) +| nvidia_gpu | Enable nvidia accelerator for runtimes +| oci | Cloud provider: oci +| persistent_volumes | Configure csi volumes +| persistent_volumes_aws_ebs_csi | Configuring csi driver: aws-ebs +| persistent_volumes_cinder_csi | Configuring csi driver: cinder +| persistent_volumes_gcp_pd_csi | Configuring csi driver: gcp-pd +| persistent_volumes_openstack | Configuring csi driver: openstack +| policy-controller | Configuring Calico policy controller +| post-remove | Tasks running post-remove operation +| post-upgrade | Tasks running post-upgrade operation +| pre-remove | Tasks running pre-remove operation +| pre-upgrade | Tasks running pre-upgrade operation +| preinstall | Preliminary configuration steps +| registry | Configuring local docker registry +| reset | Tasks running doing the node reset +| resolvconf | Configuring /etc/resolv.conf for hosts/apps +| rbd-provisioner | Configure External provisioner: rdb +| services | Remove services (etcd, kubelet etc...) when resetting +| snapshot | Enabling csi snapshot +| snapshot-controller | Configuring csi snapshot controller +| upgrade | Upgrading, f.e. container images/binaries +| upload | Distributing images/binaries across hosts +| vsphere-csi-driver | Configuring csi driver: vsphere +| weave | Network plugin Weave +| win_nodes | Running windows specific tasks +| youki | Configuring youki runtime + +Note: Use the ``bash scripts/gen_tags.sh`` command to generate a list of all +tags found in the codebase. New tags will be listed with the empty "Used for" +field. + +## Example commands + +Example command to filter and apply only DNS configuration tasks and skip +everything else related to host OS configuration and downloading images of containers: + +```ShellSession +ansible-playbook -i inventory/sample/hosts.ini cluster.yml --tags preinstall,facts --skip-tags=download,bootstrap-os +``` + +And this play only removes the K8s cluster DNS resolver IP from hosts' /etc/resolv.conf files: + +```ShellSession +ansible-playbook -i inventory/sample/hosts.ini -e dns_mode='none' cluster.yml --tags resolvconf +``` + +And this prepares all container images locally (at the ansible runner node) without installing +or upgrading related stuff or trying to upload container to K8s cluster nodes: + +```ShellSession +ansible-playbook -i inventory/sample/hosts.ini cluster.yml \ + -e download_run_once=true -e download_localhost=true \ + --tags download --skip-tags upload,upgrade +``` + +Note: use `--tags` and `--skip-tags` wise and only if you're 100% sure what you're doing. + +## Bastion host + +If you prefer to not make your nodes publicly accessible (nodes with private IPs only), +you can use a so called *bastion* host to connect to your nodes. To specify and use a bastion, +simply add a line to your inventory, where you have to replace x.x.x.x with the public IP of the +bastion host. + +```ShellSession +[bastion] +bastion ansible_host=x.x.x.x +``` + +For more information about Ansible and bastion hosts, read +[Running Ansible Through an SSH Bastion Host](https://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/) + +## Mitogen + +Mitogen support is deprecated, please see [mitogen related docs](/docs/mitogen.md) for useage and reasons for deprecation. + +## Beyond ansible 2.9 + +Ansible project has decided, in order to ease their maintenance burden, to split between +two projects which are now joined under the Ansible umbrella. + +Ansible-base (2.10.x branch) will contain just the ansible language implementation while +ansible modules that were previously bundled into a single repository will be part of the +ansible 3.x package. Pleasee see [this blog post](https://blog.while-true-do.io/ansible-release-3-0-0/) +that explains in detail the need and the evolution plan. + +**Note:** this change means that ansible virtual envs cannot be upgraded with `pip install -U`. +You first need to uninstall your old ansible (pre 2.10) version and install the new one. + +```ShellSession +pip uninstall ansible ansible-base ansible-core +cd kubespray/ +pip install -U . +```