--- /dev/null
+kubespray.io
\ No newline at end of file
--- /dev/null
+# Contributing guidelines
+
+## How to become a contributor and submit your own code
+
+### Environment setup
+
+It is recommended to use filter to manage the GitHub email notification, see [examples for setting filters to Kubernetes Github notifications](https://github.com/kubernetes/community/blob/master/communication/best-practices.md#examples-for-setting-filters-to-kubernetes-github-notifications)
+
+To install development dependencies you can set up a python virtual env with the necessary dependencies:
+
+```ShellSession
+virtualenv venv
+source venv/bin/activate
+pip install -r tests/requirements.txt
+```
+
+#### Linting
+
+Kubespray uses `yamllint` and `ansible-lint`. To run them locally use `yamllint .` and `ansible-lint`. It is a good idea to add call these tools as part of your pre-commit hook and avoid a lot of back end forth on fixing linting issues (<https://support.gitkraken.com/working-with-repositories/githooksexample/>).
+
+#### Molecule
+
+[molecule](https://github.com/ansible-community/molecule) is designed to help the development and testing of Ansible roles. In Kubespray you can run it all for all roles with `./tests/scripts/molecule_run.sh` or for a specific role (that you are working with) with `molecule test` from the role directory (`cd roles/my-role`).
+
+When developing or debugging a role it can be useful to run `molecule create` and `molecule converge` separately. Then you can use `molecule login` to SSH into the test environment.
+
+#### Vagrant
+
+Vagrant with VirtualBox or libvirt driver helps you to quickly spin test clusters to test things end to end. See [README.md#vagrant](README.md)
+
+### Contributing A Patch
+
+1. Submit an issue describing your proposed change to the repo in question.
+2. The [repo owners](OWNERS) will respond to your issue promptly.
+3. Fork the desired repo, develop and test your code changes.
+4. Sign the CNCF CLA (<https://git.k8s.io/community/CLA.md#the-contributor-license-agreement>)
+5. Submit a pull request.
+6. Work with the reviewers on their suggestions.
+7. Ensure to rebase to the HEAD of your target branch and squash un-necessary commits (<https://blog.carbonfive.com/always-squash-and-rebase-your-git-commits/>) before final merger of your contribution.
--- /dev/null
+# Use imutable image tags rather than mutable tags (like ubuntu:20.04)
+FROM ubuntu:focal-20220531
+
+ARG ARCH=amd64
+ARG TZ=Etc/UTC
+RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
+
+RUN apt update -y \
+ && apt install -y \
+ libssl-dev python3-dev sshpass apt-transport-https jq moreutils \
+ ca-certificates curl gnupg2 software-properties-common python3-pip unzip rsync git \
+ && rm -rf /var/lib/apt/lists/*
+RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - \
+ && add-apt-repository \
+ "deb [arch=$ARCH] https://download.docker.com/linux/ubuntu \
+ $(lsb_release -cs) \
+ stable" \
+ && apt update -y && apt-get install --no-install-recommends -y docker-ce \
+ && rm -rf /var/lib/apt/lists/*
+
+# Some tools like yamllint need this
+# Pip needs this as well at the moment to install ansible
+# (and potentially other packages)
+# See: https://github.com/pypa/pip/issues/10219
+ENV LANG=C.UTF-8
+
+WORKDIR /kubespray
+COPY . .
+RUN /usr/bin/python3 -m pip install --no-cache-dir pip -U \
+ && /usr/bin/python3 -m pip install --no-cache-dir -r tests/requirements.txt \
+ && python3 -m pip install --no-cache-dir -r requirements.txt \
+ && update-alternatives --install /usr/bin/python python /usr/bin/python3 1
+
+RUN KUBE_VERSION=$(sed -n 's/^kube_version: //p' roles/kubespray-defaults/defaults/main.yaml) \
+ && curl -LO https://storage.googleapis.com/kubernetes-release/release/$KUBE_VERSION/bin/linux/$ARCH/kubectl \
+ && chmod a+x kubectl \
+ && mv kubectl /usr/local/bin/kubectl
+
+RUN chmod a+x scripts/deploy.sh
+
+CMD scripts/deploy.sh
--- /dev/null
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "{}"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2016 Kubespray
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
--- /dev/null
+mitogen:
+ @echo Mitogen support is deprecated.
+ @echo Please run the following command manually:
+ @echo ansible-playbook -c local mitogen.yml -vv
+clean:
+ rm -rf dist/
+ rm *.retry
--- /dev/null
+# See the OWNERS docs at https://go.k8s.io/owners
+
+approvers:
+ - kubespray-approvers
+reviewers:
+ - kubespray-reviewers
+emeritus_approvers:
+ - kubespray-emeritus_approvers
\ No newline at end of file
--- /dev/null
+aliases:
+ kubespray-approvers:
+ - mattymo
+ - chadswen
+ - mirwan
+ - miouge1
+ - luckysb
+ - floryut
+ - oomichi
+ - cristicalin
+ kubespray-reviewers:
+ - holmsten
+ - bozzo
+ - eppo
+ - oomichi
+ - jayonlau
+ - cristicalin
+ - liupeng0518
+ kubespray-emeritus_approvers:
+ - riverzhang
+ - atoms
+ - ant31
+ - woopstar
--- /dev/null
+# Deploy a Production Ready Kubernetes Cluster
+
+
+
+If you have questions, check the documentation at [kubespray.io](https://kubespray.io) and join us on the [kubernetes slack](https://kubernetes.slack.com), channel **\#kubespray**.
+You can get your invite [here](http://slack.k8s.io/)
+
+- Can be deployed on **[AWS](docs/aws.md), GCE, [Azure](docs/azure.md), [OpenStack](docs/openstack.md), [vSphere](docs/vsphere.md), [Equinix Metal](docs/equinix-metal.md) (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal**
+- **Highly available** cluster
+- **Composable** (Choice of the network plugin for instance)
+- Supports most popular **Linux distributions**
+- **Continuous integration tests**
+
+## Quick Start
+
+To deploy the cluster you can use :
+
+### Ansible
+
+#### Usage
+
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
+then run the following steps:
+
+```ShellSession
+# Copy ``inventory/sample`` as ``inventory/mycluster``
+cp -rfp inventory/sample inventory/mycluster
+
+# Update Ansible inventory file with inventory builder
+declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
+CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+
+# Review and change parameters under ``inventory/mycluster/group_vars``
+cat inventory/mycluster/group_vars/all/all.yml
+cat inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
+
+# Deploy Kubespray with Ansible Playbook - run the playbook as root
+# The option `--become` is required, as for example writing SSL keys in /etc/,
+# installing packages and interacting with various systemd daemons.
+# Without --become the playbook will fail to run!
+ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
+```
+
+Note: When Ansible is already installed via system packages on the control machine, other python packages installed via `sudo pip install -r requirements.txt` will go to a different directory tree (e.g. `/usr/local/lib/python2.7/dist-packages` on Ubuntu) from Ansible's (e.g. `/usr/lib/python2.7/dist-packages/ansible` still on Ubuntu).
+As a consequence, `ansible-playbook` command will fail with:
+
+```raw
+ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.
+```
+
+probably pointing on a task depending on a module present in requirements.txt.
+
+One way of solving this would be to uninstall the Ansible package and then, to install it via pip but it is not always possible.
+A workaround consists of setting `ANSIBLE_LIBRARY` and `ANSIBLE_MODULE_UTILS` environment variables respectively to the `ansible/modules` and `ansible/module_utils` subdirectories of pip packages installation location, which can be found in the Location field of the output of `pip show [package]` before executing `ansible-playbook`.
+
+A simple way to ensure you get all the correct version of Ansible is to use the [pre-built docker image from Quay](https://quay.io/repository/kubespray/kubespray?tab=tags).
+You will then need to use [bind mounts](https://docs.docker.com/storage/bind-mounts/) to get the inventory and ssh key into the container, like this:
+
+```ShellSession
+docker pull quay.io/kubespray/kubespray:v2.19.0
+docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \
+ --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
+ quay.io/kubespray/kubespray:v2.19.0 bash
+# Inside the container you may now run the kubespray playbooks:
+ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
+```
+
+### Vagrant
+
+For Vagrant we need to install python dependencies for provisioning tasks.
+Check if Python and pip are installed:
+
+```ShellSession
+python -V && pip -V
+```
+
+If this returns the version of the software, you're good to go. If not, download and install Python from here <https://www.python.org/downloads/source/>
+
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
+then run the following step:
+
+```ShellSession
+vagrant up
+```
+
+## Documents
+
+- [Requirements](#requirements)
+- [Kubespray vs ...](docs/comparisons.md)
+- [Getting started](docs/getting-started.md)
+- [Setting up your first cluster](docs/setting-up-your-first-cluster.md)
+- [Ansible inventory and tags](docs/ansible.md)
+- [Integration with existing ansible repo](docs/integration.md)
+- [Deployment data variables](docs/vars.md)
+- [DNS stack](docs/dns-stack.md)
+- [HA mode](docs/ha-mode.md)
+- [Network plugins](#network-plugins)
+- [Vagrant install](docs/vagrant.md)
+- [Flatcar Container Linux bootstrap](docs/flatcar.md)
+- [Fedora CoreOS bootstrap](docs/fcos.md)
+- [Debian Jessie setup](docs/debian.md)
+- [openSUSE setup](docs/opensuse.md)
+- [Downloaded artifacts](docs/downloads.md)
+- [Cloud providers](docs/cloud.md)
+- [OpenStack](docs/openstack.md)
+- [AWS](docs/aws.md)
+- [Azure](docs/azure.md)
+- [vSphere](docs/vsphere.md)
+- [Equinix Metal](docs/equinix-metal.md)
+- [Large deployments](docs/large-deployments.md)
+- [Adding/replacing a node](docs/nodes.md)
+- [Upgrades basics](docs/upgrades.md)
+- [Air-Gap installation](docs/offline-environment.md)
+- [NTP](docs/ntp.md)
+- [Hardening](docs/hardening.md)
+- [Roadmap](docs/roadmap.md)
+
+## Supported Linux Distributions
+
+- **Flatcar Container Linux by Kinvolk**
+- **Debian** Bullseye, Buster, Jessie, Stretch
+- **Ubuntu** 16.04, 18.04, 20.04, 22.04
+- **CentOS/RHEL** 7, [8](docs/centos.md#centos-8)
+- **Fedora** 35, 36
+- **Fedora CoreOS** (see [fcos Note](docs/fcos.md))
+- **openSUSE** Leap 15.x/Tumbleweed
+- **Oracle Linux** 7, [8](docs/centos.md#centos-8)
+- **Alma Linux** [8](docs/centos.md#centos-8)
+- **Rocky Linux** [8](docs/centos.md#centos-8)
+- **Kylin Linux Advanced Server V10** (experimental: see [kylin linux notes](docs/kylinlinux.md))
+- **Amazon Linux 2** (experimental: see [amazon linux notes](docs/amazonlinux.md))
+
+Note: Upstart/SysV init based OS types are not supported.
+
+## Supported Components
+
+- Core
+ - [kubernetes](https://github.com/kubernetes/kubernetes) v1.24.4
+ - [etcd](https://github.com/etcd-io/etcd) v3.5.4
+ - [docker](https://www.docker.com/) v20.10 (see note)
+ - [containerd](https://containerd.io/) v1.6.8
+ - [cri-o](http://cri-o.io/) v1.24 (experimental: see [CRI-O Note](docs/cri-o.md). Only on fedora, ubuntu and centos based OS)
+- Network Plugin
+ - [cni-plugins](https://github.com/containernetworking/plugins) v1.1.1
+ - [calico](https://github.com/projectcalico/calico) v3.23.3
+ - [canal](https://github.com/projectcalico/canal) (given calico/flannel versions)
+ - [cilium](https://github.com/cilium/cilium) v1.11.7
+ - [flannel](https://github.com/flannel-io/flannel) v0.18.1
+ - [kube-ovn](https://github.com/alauda/kube-ovn) v1.9.7
+ - [kube-router](https://github.com/cloudnativelabs/kube-router) v1.5.1
+ - [multus](https://github.com/intel/multus-cni) v3.8
+ - [weave](https://github.com/weaveworks/weave) v2.8.1
+ - [kube-vip](https://github.com/kube-vip/kube-vip) v0.4.2
+- Application
+ - [cert-manager](https://github.com/jetstack/cert-manager) v1.9.0
+ - [coredns](https://github.com/coredns/coredns) v1.8.6
+ - [ingress-nginx](https://github.com/kubernetes/ingress-nginx) v1.3.0
+ - [krew](https://github.com/kubernetes-sigs/krew) v0.4.3
+ - [argocd](https://argoproj.github.io/) v2.4.7
+ - [helm](https://helm.sh/) v3.9.2
+ - [metallb](https://metallb.universe.tf/) v0.12.1
+ - [registry](https://github.com/distribution/distribution) v2.8.1
+- Storage Plugin
+ - [cephfs-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.0-k8s1.11
+ - [rbd-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.1-k8s1.11
+ - [aws-ebs-csi-plugin](https://github.com/kubernetes-sigs/aws-ebs-csi-driver) v0.5.0
+ - [azure-csi-plugin](https://github.com/kubernetes-sigs/azuredisk-csi-driver) v1.10.0
+ - [cinder-csi-plugin](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/cinder-csi-plugin/using-cinder-csi-plugin.md) v1.22.0
+ - [gcp-pd-csi-plugin](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver) v1.4.0
+ - [local-path-provisioner](https://github.com/rancher/local-path-provisioner) v0.0.22
+ - [local-volume-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) v2.4.0
+
+## Container Runtime Notes
+
+- The list of available docker version is 18.09, 19.03 and 20.10. The recommended docker version is 20.10. The kubelet might break on docker's non-standard version numbering (it no longer uses semantic versioning). To ensure auto-updates don't break your cluster look into e.g. yum versionlock plugin or apt pin).
+- The cri-o version should be aligned with the respective kubernetes version (i.e. kube_version=1.20.x, crio_version=1.20)
+
+## Requirements
+
+- **Minimum required version of Kubernetes is v1.22**
+- **Ansible v2.11+, Jinja 2.11+ and python-netaddr is installed on the machine that will run Ansible commands**
+- The target servers must have **access to the Internet** in order to pull docker images. Otherwise, additional configuration is required (See [Offline Environment](docs/offline-environment.md))
+- The target servers are configured to allow **IPv4 forwarding**.
+- If using IPv6 for pods and services, the target servers are configured to allow **IPv6 forwarding**.
+- The **firewalls are not managed**, you'll need to implement your own rules the way you used to.
+ in order to avoid any issue during deployment you should disable your firewall.
+- If kubespray is ran from non-root user account, correct privilege escalation method
+ should be configured in the target servers. Then the `ansible_become` flag
+ or command parameters `--become or -b` should be specified.
+
+Hardware:
+These limits are safe guarded by Kubespray. Actual requirements for your workload can differ. For a sizing guide go to the [Building Large Clusters](https://kubernetes.io/docs/setup/cluster-large/#size-of-master-and-master-components) guide.
+
+- Master
+ - Memory: 1500 MB
+- Node
+ - Memory: 1024 MB
+
+## Network Plugins
+
+You can choose between 10 network plugins. (default: `calico`, except Vagrant uses `flannel`)
+
+- [flannel](docs/flannel.md): gre/vxlan (layer 2) networking.
+
+- [Calico](https://docs.projectcalico.org/latest/introduction/) is a networking and network policy provider. Calico supports a flexible set of networking options
+ designed to give you the most efficient networking across a range of situations, including non-overlay
+ and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts,
+ pods, and (if using Istio and Envoy) applications at the service mesh layer.
+
+- [canal](https://github.com/projectcalico/canal): a composition of calico and flannel plugins.
+
+- [cilium](http://docs.cilium.io/en/latest/): layer 3/4 networking (as well as layer 7 to protect and secure application protocols), supports dynamic insertion of BPF bytecode into the Linux kernel to implement security services, networking and visibility logic.
+
+- [weave](docs/weave.md): Weave is a lightweight container overlay network that doesn't require an external K/V database cluster.
+ (Please refer to `weave` [troubleshooting documentation](https://www.weave.works/docs/net/latest/troubleshooting/)).
+
+- [kube-ovn](docs/kube-ovn.md): Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.
+
+- [kube-router](docs/kube-router.md): Kube-router is a L3 CNI for Kubernetes networking aiming to provide operational
+ simplicity and high performance: it uses IPVS to provide Kube Services Proxy (if setup to replace kube-proxy),
+ iptables for network policies, and BGP for ods L3 networking (with optionally BGP peering with out-of-cluster BGP peers).
+ It can also optionally advertise routes to Kubernetes cluster Pods CIDRs, ClusterIPs, ExternalIPs and LoadBalancerIPs.
+
+- [macvlan](docs/macvlan.md): Macvlan is a Linux network driver. Pods have their own unique Mac and Ip address, connected directly the physical (layer 2) network.
+
+- [multus](docs/multus.md): Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.
+
+The choice is defined with the variable `kube_network_plugin`. There is also an
+option to leverage built-in cloud provider networking instead.
+See also [Network checker](docs/netcheck.md).
+
+## Ingress Plugins
+
+- [nginx](https://kubernetes.github.io/ingress-nginx): the NGINX Ingress Controller.
+
+- [metallb](docs/metallb.md): the MetalLB bare-metal service LoadBalancer provider.
+
+## Community docs and resources
+
+- [kubernetes.io/docs/setup/production-environment/tools/kubespray/](https://kubernetes.io/docs/setup/production-environment/tools/kubespray/)
+- [kubespray, monitoring and logging](https://github.com/gregbkr/kubernetes-kargo-logging-monitoring) by @gregbkr
+- [Deploy Kubernetes w/ Ansible & Terraform](https://rsmitty.github.io/Terraform-Ansible-Kubernetes/) by @rsmitty
+- [Deploy a Kubernetes Cluster with Kubespray (video)](https://www.youtube.com/watch?v=CJ5G4GpqDy0)
+
+## Tools and projects on top of Kubespray
+
+- [Digital Rebar Provision](https://github.com/digitalrebar/provision/blob/v4/doc/integrations/ansible.rst)
+- [Terraform Contrib](https://github.com/kubernetes-sigs/kubespray/tree/master/contrib/terraform)
+
+## CI Tests
+
+[](https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/pipelines)
+
+CI/end-to-end tests sponsored by: [CNCF](https://cncf.io), [Equinix Metal](https://metal.equinix.com/), [OVHcloud](https://www.ovhcloud.com/), [ELASTX](https://elastx.se/).
+
+See the [test matrix](docs/test_cases.md) for details.
--- /dev/null
+# Release Process
+
+The Kubespray Project is released on an as-needed basis. The process is as follows:
+
+1. An issue is proposing a new release with a changelog since the last release. Please see [a good sample issue](https://github.com/kubernetes-sigs/kubespray/issues/8325)
+2. At least one of the [approvers](OWNERS_ALIASES) must approve this release
+3. The `kube_version_min_required` variable is set to `n-1`
+4. Remove hashes for [EOL versions](https://github.com/kubernetes/website/blob/main/content/en/releases/patch-releases.md) of kubernetes from `*_checksums` variables.
+5. Create the release note with [Kubernetes Release Notes Generator](https://github.com/kubernetes/release/blob/master/cmd/release-notes/README.md). See the following `Release note creation` section for the details.
+6. An approver creates [new release in GitHub](https://github.com/kubernetes-sigs/kubespray/releases/new) using a version and tag name like `vX.Y.Z` and attaching the release notes
+7. An approver creates a release branch in the form `release-X.Y`
+8. The corresponding version of [quay.io/kubespray/kubespray:vX.Y.Z](https://quay.io/repository/kubespray/kubespray) and [quay.io/kubespray/vagrant:vX.Y.Z](https://quay.io/repository/kubespray/vagrant) container images are built and tagged. See the following `Container image creation` section for the details.
+9. The `KUBESPRAY_VERSION` variable is updated in `.gitlab-ci.yml`
+10. The release issue is closed
+11. An announcement email is sent to `dev@kubernetes.io` with the subject `[ANNOUNCE] Kubespray $VERSION is released`
+12. The topic of the #kubespray channel is updated with `vX.Y.Z is released! | ...`
+
+## Major/minor releases and milestones
+
+* For major releases (vX.Y) Kubespray maintains one branch (`release-X.Y`). Minor releases (vX.Y.Z) are available only as tags.
+
+* Security patches and bugs might be backported.
+
+* Fixes for major releases (vX.Y) and minor releases (vX.Y.Z) are delivered
+ via maintenance releases (vX.Y.Z) and assigned to the corresponding open
+ [GitHub milestone](https://github.com/kubernetes-sigs/kubespray/milestones).
+ That milestone remains open for the major/minor releases support lifetime,
+ which ends once the milestone is closed. Then only a next major or minor release
+ can be done.
+
+* Kubespray major and minor releases are bound to the given `kube_version` major/minor
+ version numbers and other components' arbitrary versions, like etcd or network plugins.
+ Older or newer component versions are not supported and not tested for the given
+ release (even if included in the checksum variables, like `kubeadm_checksums`).
+
+* There is no unstable releases and no APIs, thus Kubespray doesn't follow
+ [semver](https://semver.org/). Every version describes only a stable release.
+ Breaking changes, if any introduced by changed defaults or non-contrib ansible roles'
+ playbooks, shall be described in the release notes. Other breaking changes, if any in
+ the contributed addons or bound versions of Kubernetes and other components, are
+ considered out of Kubespray scope and are up to the components' teams to deal with and
+ document.
+
+* Minor releases can change components' versions, but not the major `kube_version`.
+ Greater `kube_version` requires a new major or minor release. For example, if Kubespray v2.0.0
+ is bound to `kube_version: 1.4.x`, `calico_version: 0.22.0`, `etcd_version: v3.0.6`,
+ then Kubespray v2.1.0 may be bound to only minor changes to `kube_version`, like v1.5.1
+ and *any* changes to other components, like etcd v4, or calico 1.2.3.
+ And Kubespray v3.x.x shall be bound to `kube_version: 2.x.x` respectively.
+
+## Release note creation
+
+You can create a release note with:
+
+```shell
+export GITHUB_TOKEN=<your-github-token>
+export ORG=kubernetes-sigs
+export REPO=kubespray
+release-notes --start-sha <The start commit-id> --end-sha <The end commit-id> --dependencies=false --output=/tmp/kubespray-release-note --required-author=""
+```
+
+If the release note file(/tmp/kubespray-release-note) contains "### Uncategorized" pull requests, those pull requests don't have a valid kind label(`kind/feature`, etc.).
+It is necessary to put a valid label on each pull request and run the above release-notes command again to get a better release note)
+
+## Container image creation
+
+The container image `quay.io/kubespray/kubespray:vX.Y.Z` can be created from Dockerfile of the kubespray root directory:
+
+```shell
+cd kubespray/
+nerdctl build -t quay.io/kubespray/kubespray:vX.Y.Z .
+nerdctl push quay.io/kubespray/kubespray:vX.Y.Z
+```
+
+The container image `quay.io/kubespray/vagrant:vX.Y.Z` can be created from build.sh of test-infra/vagrant-docker/:
+
+```shell
+cd kubespray/test-infra/vagrant-docker/
+./build vX.Y.Z
+```
+
+Please note that the above operation requires the permission to push container images into quay.io/kubespray/.
+If you don't have the permission, please ask it on the #kubespray-dev channel.
--- /dev/null
+# Defined below are the security contacts for this repo.
+#
+# They are the contact point for the Product Security Committee to reach out
+# to for triaging and handling of incoming issues.
+#
+# The below names agree to abide by the
+# [Embargo Policy](https://git.k8s.io/security/private-distributors-list.md#embargo-policy)
+# and will be removed and replaced if they violate that agreement.
+#
+# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
+# INSTRUCTIONS AT https://kubernetes.io/security/
+atoms
+mattymo
--- /dev/null
+# -*- mode: ruby -*-
+# # vi: set ft=ruby :
+
+# For help on using kubespray with vagrant, check out docs/vagrant.md
+
+require 'fileutils'
+
+Vagrant.require_version ">= 2.0.0"
+
+CONFIG = File.join(File.dirname(__FILE__), ENV['KUBESPRAY_VAGRANT_CONFIG'] || 'vagrant/config.rb')
+
+FLATCAR_URL_TEMPLATE = "https://%s.release.flatcar-linux.net/amd64-usr/current/flatcar_production_vagrant.json"
+
+# Uniq disk UUID for libvirt
+DISK_UUID = Time.now.utc.to_i
+
+SUPPORTED_OS = {
+ "flatcar-stable" => {box: "flatcar-stable", user: "core", box_url: FLATCAR_URL_TEMPLATE % ["stable"]},
+ "flatcar-beta" => {box: "flatcar-beta", user: "core", box_url: FLATCAR_URL_TEMPLATE % ["beta"]},
+ "flatcar-alpha" => {box: "flatcar-alpha", user: "core", box_url: FLATCAR_URL_TEMPLATE % ["alpha"]},
+ "flatcar-edge" => {box: "flatcar-edge", user: "core", box_url: FLATCAR_URL_TEMPLATE % ["edge"]},
+ "ubuntu1604" => {box: "generic/ubuntu1604", user: "vagrant"},
+ "ubuntu1804" => {box: "generic/ubuntu1804", user: "vagrant"},
+ "ubuntu2004" => {box: "generic/ubuntu2004", user: "vagrant"},
+ "centos" => {box: "centos/7", user: "vagrant"},
+ "centos-bento" => {box: "bento/centos-7.6", user: "vagrant"},
+ "centos8" => {box: "centos/8", user: "vagrant"},
+ "centos8-bento" => {box: "bento/centos-8", user: "vagrant"},
+ "almalinux8" => {box: "almalinux/8", user: "vagrant"},
+ "almalinux8-bento" => {box: "bento/almalinux-8", user: "vagrant"},
+ "rockylinux8" => {box: "generic/rocky8", user: "vagrant"},
+ "fedora35" => {box: "fedora/35-cloud-base", user: "vagrant"},
+ "fedora36" => {box: "fedora/36-cloud-base", user: "vagrant"},
+ "opensuse" => {box: "opensuse/Leap-15.4.x86_64", user: "vagrant"},
+ "opensuse-tumbleweed" => {box: "opensuse/Tumbleweed.x86_64", user: "vagrant"},
+ "oraclelinux" => {box: "generic/oracle7", user: "vagrant"},
+ "oraclelinux8" => {box: "generic/oracle8", user: "vagrant"},
+ "rhel7" => {box: "generic/rhel7", user: "vagrant"},
+ "rhel8" => {box: "generic/rhel8", user: "vagrant"},
+}
+
+if File.exist?(CONFIG)
+ require CONFIG
+end
+
+# Defaults for config options defined in CONFIG
+$num_instances ||= 3
+$instance_name_prefix ||= "k8s"
+$vm_gui ||= false
+$vm_memory ||= 2048
+$vm_cpus ||= 2
+$shared_folders ||= {}
+$forwarded_ports ||= {}
+$subnet ||= "172.18.8"
+$subnet_ipv6 ||= "fd3c:b398:0698:0756"
+$os ||= "ubuntu1804"
+$network_plugin ||= "flannel"
+# Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni
+$multi_networking ||= "False"
+$download_run_once ||= "True"
+$download_force_cache ||= "False"
+# The first three nodes are etcd servers
+$etcd_instances ||= $num_instances
+# The first two nodes are kube masters
+$kube_master_instances ||= $num_instances == 1 ? $num_instances : ($num_instances - 1)
+# All nodes are kube nodes
+$kube_node_instances ||= $num_instances
+# The following only works when using the libvirt provider
+$kube_node_instances_with_disks ||= false
+$kube_node_instances_with_disks_size ||= "20G"
+$kube_node_instances_with_disks_number ||= 2
+$override_disk_size ||= false
+$disk_size ||= "20GB"
+$local_path_provisioner_enabled ||= "False"
+$local_path_provisioner_claim_root ||= "/opt/local-path-provisioner/"
+$libvirt_nested ||= false
+# boolean or string (e.g. "-vvv")
+$ansible_verbosity ||= false
+$ansible_tags ||= ENV['VAGRANT_ANSIBLE_TAGS'] || ""
+
+$playbook ||= "cluster.yml"
+
+host_vars = {}
+
+$box = SUPPORTED_OS[$os][:box]
+# if $inventory is not set, try to use example
+$inventory = "inventory/sample" if ! $inventory
+$inventory = File.absolute_path($inventory, File.dirname(__FILE__))
+
+# if $inventory has a hosts.ini file use it, otherwise copy over
+# vars etc to where vagrant expects dynamic inventory to be
+if ! File.exist?(File.join(File.dirname($inventory), "hosts.ini"))
+ $vagrant_ansible = File.join(File.dirname(__FILE__), ".vagrant", "provisioners", "ansible")
+ FileUtils.mkdir_p($vagrant_ansible) if ! File.exist?($vagrant_ansible)
+ $vagrant_inventory = File.join($vagrant_ansible,"inventory")
+ FileUtils.rm_f($vagrant_inventory)
+ FileUtils.ln_s($inventory, $vagrant_inventory)
+end
+
+if Vagrant.has_plugin?("vagrant-proxyconf")
+ $no_proxy = ENV['NO_PROXY'] || ENV['no_proxy'] || "127.0.0.1,localhost"
+ (1..$num_instances).each do |i|
+ $no_proxy += ",#{$subnet}.#{i+100}"
+ end
+end
+
+Vagrant.configure("2") do |config|
+
+ config.vm.box = $box
+ if SUPPORTED_OS[$os].has_key? :box_url
+ config.vm.box_url = SUPPORTED_OS[$os][:box_url]
+ end
+ config.ssh.username = SUPPORTED_OS[$os][:user]
+
+ # plugin conflict
+ if Vagrant.has_plugin?("vagrant-vbguest") then
+ config.vbguest.auto_update = false
+ end
+
+ # always use Vagrants insecure key
+ config.ssh.insert_key = false
+
+ if ($override_disk_size)
+ unless Vagrant.has_plugin?("vagrant-disksize")
+ system "vagrant plugin install vagrant-disksize"
+ end
+ config.disksize.size = $disk_size
+ end
+
+ (1..$num_instances).each do |i|
+ config.vm.define vm_name = "%s-%01d" % [$instance_name_prefix, i] do |node|
+
+ node.vm.hostname = vm_name
+
+ if Vagrant.has_plugin?("vagrant-proxyconf")
+ node.proxy.http = ENV['HTTP_PROXY'] || ENV['http_proxy'] || ""
+ node.proxy.https = ENV['HTTPS_PROXY'] || ENV['https_proxy'] || ""
+ node.proxy.no_proxy = $no_proxy
+ end
+
+ ["vmware_fusion", "vmware_workstation"].each do |vmware|
+ node.vm.provider vmware do |v|
+ v.vmx['memsize'] = $vm_memory
+ v.vmx['numvcpus'] = $vm_cpus
+ end
+ end
+
+ node.vm.provider :virtualbox do |vb|
+ vb.memory = $vm_memory
+ vb.cpus = $vm_cpus
+ vb.gui = $vm_gui
+ vb.linked_clone = true
+ vb.customize ["modifyvm", :id, "--vram", "8"] # ubuntu defaults to 256 MB which is a waste of precious RAM
+ vb.customize ["modifyvm", :id, "--audio", "none"]
+ end
+
+ node.vm.provider :libvirt do |lv|
+ lv.nested = $libvirt_nested
+ lv.cpu_mode = "host-model"
+ lv.memory = $vm_memory
+ lv.cpus = $vm_cpus
+ lv.default_prefix = 'kubespray'
+ # Fix kernel panic on fedora 28
+ if $os == "fedora"
+ lv.cpu_mode = "host-passthrough"
+ end
+ end
+
+ if $kube_node_instances_with_disks
+ # Libvirt
+ driverletters = ('a'..'z').to_a
+ node.vm.provider :libvirt do |lv|
+ # always make /dev/sd{a/b/c} so that CI can ensure that
+ # virtualbox and libvirt will have the same devices to use for OSDs
+ (1..$kube_node_instances_with_disks_number).each do |d|
+ lv.storage :file, :device => "hd#{driverletters[d]}", :path => "disk-#{i}-#{d}-#{DISK_UUID}.disk", :size => $kube_node_instances_with_disks_size, :bus => "scsi"
+ end
+ end
+ end
+
+ if $expose_docker_tcp
+ node.vm.network "forwarded_port", guest: 2375, host: ($expose_docker_tcp + i - 1), auto_correct: true
+ end
+
+ $forwarded_ports.each do |guest, host|
+ node.vm.network "forwarded_port", guest: guest, host: host, auto_correct: true
+ end
+
+ if ["rhel7","rhel8"].include? $os
+ # Vagrant synced_folder rsync options cannot be used for RHEL boxes as Rsync package cannot
+ # be installed until the host is registered with a valid Red Hat support subscription
+ node.vm.synced_folder ".", "/vagrant", disabled: false
+ $shared_folders.each do |src, dst|
+ node.vm.synced_folder src, dst
+ end
+ else
+ node.vm.synced_folder ".", "/vagrant", disabled: false, type: "rsync", rsync__args: ['--verbose', '--archive', '--delete', '-z'] , rsync__exclude: ['.git','venv']
+ $shared_folders.each do |src, dst|
+ node.vm.synced_folder src, dst, type: "rsync", rsync__args: ['--verbose', '--archive', '--delete', '-z']
+ end
+ end
+
+ ip = "#{$subnet}.#{i+100}"
+ node.vm.network :private_network, ip: ip,
+ :libvirt__guest_ipv6 => 'yes',
+ :libvirt__ipv6_address => "#{$subnet_ipv6}::#{i+100}",
+ :libvirt__ipv6_prefix => "64",
+ :libvirt__forward_mode => "none",
+ :libvirt__dhcp_enabled => false
+
+ # Disable swap for each vm
+ node.vm.provision "shell", inline: "swapoff -a"
+
+ # ubuntu1804 and ubuntu2004 have IPv6 explicitly disabled. This undoes that.
+ if ["ubuntu1804", "ubuntu2004"].include? $os
+ node.vm.provision "shell", inline: "rm -f /etc/modprobe.d/local.conf"
+ node.vm.provision "shell", inline: "sed -i '/net.ipv6.conf.all.disable_ipv6/d' /etc/sysctl.d/99-sysctl.conf /etc/sysctl.conf"
+ end
+
+ # Disable firewalld on oraclelinux/redhat vms
+ if ["oraclelinux","oraclelinux8","rhel7","rhel8"].include? $os
+ node.vm.provision "shell", inline: "systemctl stop firewalld; systemctl disable firewalld"
+ end
+
+ host_vars[vm_name] = {
+ "ip": ip,
+ "flannel_interface": "eth1",
+ "kube_network_plugin": $network_plugin,
+ "kube_network_plugin_multus": $multi_networking,
+ "download_run_once": $download_run_once,
+ "download_localhost": "False",
+ "download_cache_dir": ENV['HOME'] + "/kubespray_cache",
+ # Make kubespray cache even when download_run_once is false
+ "download_force_cache": $download_force_cache,
+ # Keeping the cache on the nodes can improve provisioning speed while debugging kubespray
+ "download_keep_remote_cache": "False",
+ "docker_rpm_keepcache": "1",
+ # These two settings will put kubectl and admin.config in $inventory/artifacts
+ "kubeconfig_localhost": "True",
+ "kubectl_localhost": "True",
+ "local_path_provisioner_enabled": "#{$local_path_provisioner_enabled}",
+ "local_path_provisioner_claim_root": "#{$local_path_provisioner_claim_root}",
+ "ansible_ssh_user": SUPPORTED_OS[$os][:user]
+ }
+
+ # Only execute the Ansible provisioner once, when all the machines are up and ready.
+ # And limit the action to gathering facts, the full playbook is going to be ran by testcases_run.sh
+ if i == $num_instances
+ node.vm.provision "ansible" do |ansible|
+ ansible.playbook = $playbook
+ ansible.verbose = $ansible_verbosity
+ $ansible_inventory_path = File.join( $inventory, "hosts.ini")
+ if File.exist?($ansible_inventory_path)
+ ansible.inventory_path = $ansible_inventory_path
+ end
+ ansible.become = true
+ ansible.limit = "all,localhost"
+ ansible.host_key_checking = false
+ ansible.raw_arguments = ["--forks=#{$num_instances}", "--flush-cache", "-e ansible_become_pass=vagrant"]
+ ansible.host_vars = host_vars
+ if $ansible_tags != ""
+ ansible.tags = [$ansible_tags]
+ end
+ ansible.groups = {
+ "etcd" => ["#{$instance_name_prefix}-[1:#{$etcd_instances}]"],
+ "kube_control_plane" => ["#{$instance_name_prefix}-[1:#{$kube_master_instances}]"],
+ "kube_node" => ["#{$instance_name_prefix}-[1:#{$kube_node_instances}]"],
+ "k8s_cluster:children" => ["kube_control_plane", "kube_node"],
+ }
+ end
+ end
+
+ end
+ end
+end
--- /dev/null
+---
+theme: jekyll-theme-slate
--- /dev/null
+[ssh_connection]
+pipelining=True
+ansible_ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
+#control_path = ~/.ssh/ansible-%%r@%%h:%%p
+[defaults]
+# https://github.com/ansible/ansible/issues/56930 (to ignore group names with - and .)
+force_valid_group_names = ignore
+
+host_key_checking=False
+gathering = smart
+fact_caching = jsonfile
+fact_caching_connection = /tmp
+fact_caching_timeout = 86400
+stdout_callback = default
+display_skipped_hosts = no
+library = ./library
+callbacks_enabled = profile_tasks,ara_default
+roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles
+deprecation_warnings=False
+inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds, .gpg
+[inventory]
+ignore_patterns = artifacts, credentials
--- /dev/null
+---
+- hosts: localhost
+ gather_facts: false
+ become: no
+ vars:
+ minimal_ansible_version: 2.11.0
+ maximal_ansible_version: 2.13.0
+ ansible_connection: local
+ tags: always
+ tasks:
+ - name: "Check {{ minimal_ansible_version }} <= Ansible version < {{ maximal_ansible_version }}"
+ assert:
+ msg: "Ansible must be between {{ minimal_ansible_version }} and {{ maximal_ansible_version }} exclusive"
+ that:
+ - ansible_version.string is version(minimal_ansible_version, ">=")
+ - ansible_version.string is version(maximal_ansible_version, "<")
+ tags:
+ - check
+
+ - name: "Check that python netaddr is installed"
+ assert:
+ msg: "Python netaddr is not present"
+ that: "'127.0.0.1' | ipaddr"
+ tags:
+ - check
+
+ # CentOS 7 provides too old jinja version
+ - name: "Check that jinja is not too old (install via pip)"
+ assert:
+ msg: "Your Jinja version is too old, install via pip"
+ that: "{% set test %}It works{% endset %}{{ test == 'It works' }}"
+ tags:
+ - check
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"] }
+
+- hosts: k8s_cluster:etcd
+ strategy: linear
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ gather_facts: false
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bootstrap-os, tags: bootstrap-os}
+
+- name: Gather facts
+ tags: always
+ import_playbook: facts.yml
+
+- hosts: k8s_cluster:etcd
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, tags: preinstall }
+ - { role: "container-engine", tags: "container-engine", when: deploy_container_engine }
+ - { role: download, tags: download, when: "not skip_downloads" }
+
+- hosts: etcd
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - role: etcd
+ tags: etcd
+ vars:
+ etcd_cluster_setup: true
+ etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
+ when: etcd_deployment_type != "kubeadm"
+
+- hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - role: etcd
+ tags: etcd
+ vars:
+ etcd_cluster_setup: false
+ etcd_events_cluster_setup: false
+ when: etcd_deployment_type != "kubeadm"
+
+- hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/node, tags: node }
+
+- hosts: kube_control_plane
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/control-plane, tags: master }
+ - { role: kubernetes/client, tags: client }
+ - { role: kubernetes-apps/cluster_roles, tags: cluster-roles }
+
+- hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/kubeadm, tags: kubeadm}
+ - { role: kubernetes/node-label, tags: node-label }
+ - { role: network_plugin, tags: network }
+
+- hosts: calico_rr
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: network_plugin/calico/rr, tags: ['network', 'calico_rr'] }
+
+- hosts: kube_control_plane[0]
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: win_nodes/kubernetes_patch, tags: ["master", "win_nodes"] }
+
+- hosts: kube_control_plane
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes-apps/external_cloud_controller, tags: external-cloud-controller }
+ - { role: kubernetes-apps/network_plugin, tags: network }
+ - { role: kubernetes-apps/policy_controller, tags: policy-controller }
+ - { role: kubernetes-apps/ingress_controller, tags: ingress-controller }
+ - { role: kubernetes-apps/external_provisioner, tags: external-provisioner }
+ - { role: kubernetes-apps, tags: apps }
+
+- name: Apply resolv.conf changes now that cluster DNS is up
+ hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_late: true }
--- /dev/null
+# Kubernetes Community Code of Conduct
+
+Please refer to our [Kubernetes Community Code of Conduct](https://git.k8s.io/community/code-of-conduct.md)
--- /dev/null
+* [Readme](/)
+* [Comparisons](/docs/comparisons.md)
+* [Getting started](/docs/getting-started.md)
+* [Ansible](docs/ansible.md)
+* [Variables](/docs/vars.md)
+* Operations
+ * [Integration](docs/integration.md)
+ * [Upgrades](/docs/upgrades.md)
+ * [HA Mode](docs/ha-mode.md)
+ * [Adding/replacing a node](docs/nodes.md)
+ * [Large deployments](docs/large-deployments.md)
+ * [Air-Gap Installation](docs/offline-environment.md)
+* CNI
+ * [Calico](docs/calico.md)
+ * [Flannel](docs/flannel.md)
+ * [Kube Router](docs/kube-router.md)
+ * [Kube OVN](docs/kube-ovn.md)
+ * [Weave](docs/weave.md)
+ * [Multus](docs/multus.md)
+* Ingress
+ * [kube-vip](docs/kube-vip.md)
+ * [ALB Ingress](docs/ingress_controller/alb_ingress_controller.md)
+ * [MetalLB](docs/metallb.md)
+ * [Nginx Ingress](docs/ingress_controller/ingress_nginx.md)
+* [Cloud providers](docs/cloud.md)
+ * [AWS](docs/aws.md)
+ * [Azure](docs/azure.md)
+ * [OpenStack](/docs/openstack.md)
+ * [Equinix Metal](/docs/equinix-metal.md)
+ * [vSphere](/docs/vsphere.md)
+* [Operating Systems](docs/bootstrap-os.md)
+ * [Debian](docs/debian.md)
+ * [Flatcar Container Linux](docs/flatcar.md)
+ * [Fedora CoreOS](docs/fcos.md)
+ * [OpenSUSE](docs/opensuse.md)
+ * [RedHat Enterprise Linux](docs/rhel.md)
+ * [CentOS/OracleLinux/AlmaLinux/Rocky Linux](docs/centos.md)
+ * [Kylin Linux Advanced Server V10](docs/kylinlinux.md)
+ * [Amazon Linux 2](docs/amazonlinux.md)
+* CRI
+ * [Containerd](docs/containerd.md)
+ * [Docker](docs/docker.md)
+ * [CRI-O](docs/cri-o.md)
+ * [Kata Containers](docs/kata-containers.md)
+ * [gVisor](docs/gvisor.md)
+* Advanced
+ * [Proxy](/docs/proxy.md)
+ * [Downloads](docs/downloads.md)
+ * [Netcheck](docs/netcheck.md)
+ * [Cert Manager](docs/cert_manager.md)
+ * [DNS Stack](docs/dns-stack.md)
+ * [Kubernetes reliability](docs/kubernetes-reliability.md)
+ * [Local Registry](docs/kubernetes-apps/registry.md)
+ * [NTP](docs/ntp.md)
+* External Storage Provisioners
+ * [RBD Provisioner](docs/kubernetes-apps/rbd_provisioner.md)
+ * [CEPHFS Provisioner](docs/kubernetes-apps/cephfs_provisioner.md)
+ * [Local Volume Provisioner](docs/kubernetes-apps/local_volume_provisioner.md)
+* Developers
+ * [Test cases](docs/test_cases.md)
+ * [Vagrant](docs/vagrant.md)
+ * [CI Matrix](docs/ci.md)
+ * [CI Setup](docs/ci-setup.md)
+* [Roadmap](docs/roadmap.md)
--- /dev/null
+# Amazon Linux 2
+
+Amazon Linux is supported with docker,containerd and cri-o runtimes.
+
+**Note:** that Amazon Linux is not currently covered in kubespray CI and
+support for it is currently considered experimental.
+
+Amazon Linux 2, while derrived from the Redhat OS family, does not keep in
+sync with RHEL upstream like CentOS/AlmaLinux/Oracle Linux. In order to use
+Amazon Linux as the ansible host for your kubespray deployments you need to
+manually install `python3` and deploy ansible and kubespray dependencies in
+a python virtual environment or use the official kubespray containers.
+
+There are no special considerations for using Amazon Linux as the target OS
+for Kubespray deployments.
--- /dev/null
+# Ansible
+
+## Installing Ansible
+
+Kubespray supports multiple ansible versions and ships different `requirements.txt` files for them.
+Depending on your available python version you may be limited in chooding which ansible version to use.
+
+It is recommended to deploy the ansible version used by kubespray into a python virtual environment.
+
+```ShellSession
+VENVDIR=kubespray-venv
+KUBESPRAYDIR=kubespray
+ANSIBLE_VERSION=2.12
+virtualenv --python=$(which python3) $VENVDIR
+source $VENVDIR/bin/activate
+cd $KUBESPRAYDIR
+pip install -U -r requirements-$ANSIBLE_VERSION.txt
+test -f requirements-$ANSIBLE_VERSION.yml && \
+ ansible-galaxy role install -r requirements-$ANSIBLE_VERSION.yml && \
+ ansible-galaxy collection -r requirements-$ANSIBLE_VERSION.yml
+```
+
+### Ansible Python Compatibility
+
+Based on the table below and the available python version for your ansible host you should choose the appropriate ansible version to use with kubespray.
+
+| Ansible Version | Python Version |
+| --------------- | -------------- |
+| 2.11 | 2.7,3.5-3.9 |
+| 2.12 | 3.8-3.10 |
+
+## Inventory
+
+The inventory is composed of 3 groups:
+
+* **kube_node** : list of kubernetes nodes where the pods will run.
+* **kube_control_plane** : list of servers where kubernetes control plane components (apiserver, scheduler, controller) will run.
+* **etcd**: list of servers to compose the etcd server. You should have at least 3 servers for failover purpose.
+
+Note: do not modify the children of _k8s_cluster_, like putting
+the _etcd_ group into the _k8s_cluster_, unless you are certain
+to do that and you have it fully contained in the latter:
+
+```ShellSession
+etcd ⊂ k8s_cluster => kube_node ∩ etcd = etcd
+```
+
+When _kube_node_ contains _etcd_, you define your etcd cluster to be as well schedulable for Kubernetes workloads.
+If you want it a standalone, make sure those groups do not intersect.
+If you want the server to act both as control-plane and node, the server must be defined
+on both groups _kube_control_plane_ and _kube_node_. If you want a standalone and
+unschedulable control plane, the server must be defined only in the _kube_control_plane_ and
+not _kube_node_.
+
+There are also two special groups:
+
+* **calico_rr** : explained for [advanced Calico networking cases](/docs/calico.md)
+* **bastion** : configure a bastion host if your nodes are not directly reachable
+
+Below is a complete inventory example:
+
+```ini
+## Configure 'ip' variable to bind kubernetes services on a
+## different ip than the default iface
+node1 ansible_host=95.54.0.12 ip=10.3.0.1
+node2 ansible_host=95.54.0.13 ip=10.3.0.2
+node3 ansible_host=95.54.0.14 ip=10.3.0.3
+node4 ansible_host=95.54.0.15 ip=10.3.0.4
+node5 ansible_host=95.54.0.16 ip=10.3.0.5
+node6 ansible_host=95.54.0.17 ip=10.3.0.6
+
+[kube_control_plane]
+node1
+node2
+
+[etcd]
+node1
+node2
+node3
+
+[kube_node]
+node2
+node3
+node4
+node5
+node6
+
+[k8s_cluster:children]
+kube_node
+kube_control_plane
+```
+
+## Group vars and overriding variables precedence
+
+The group variables to control main deployment options are located in the directory ``inventory/sample/group_vars``.
+Optional variables are located in the `inventory/sample/group_vars/all.yml`.
+Mandatory variables that are common for at least one role (or a node group) can be found in the
+`inventory/sample/group_vars/k8s_cluster.yml`.
+There are also role vars for docker, kubernetes preinstall and control plane roles.
+According to the [ansible docs](https://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable),
+those cannot be overridden from the group vars. In order to override, one should use
+the `-e` runtime flags (most simple way) or other layers described in the docs.
+
+Kubespray uses only a few layers to override things (or expect them to
+be overridden for roles):
+
+Layer | Comment
+------|--------
+**role defaults** | provides best UX to override things for Kubespray deployments
+inventory vars | Unused
+**inventory group_vars** | Expects users to use ``all.yml``,``k8s_cluster.yml`` etc. to override things
+inventory host_vars | Unused
+playbook group_vars | Unused
+playbook host_vars | Unused
+**host facts** | Kubespray overrides for internal roles' logic, like state flags
+play vars | Unused
+play vars_prompt | Unused
+play vars_files | Unused
+registered vars | Unused
+set_facts | Kubespray overrides those, for some places
+**role and include vars** | Provides bad UX to override things! Use extra vars to enforce
+block vars (only for tasks in block) | Kubespray overrides for internal roles' logic
+task vars (only for the task) | Unused for roles, but only for helper scripts
+**extra vars** (always win precedence) | override with ``ansible-playbook -e @foo.yml``
+
+## Ansible tags
+
+The following tags are defined in playbooks:
+
+| Tag name | Used for
+|--------------------------------|---------
+| annotate | Create kube-router annotation
+| apps | K8s apps definitions
+| asserts | Check tasks for download role
+| aws-ebs-csi-driver | Configuring csi driver: aws-ebs
+| azure-csi-driver | Configuring csi driver: azure
+| bastion | Setup ssh config for bastion
+| bootstrap-os | Anything related to host OS configuration
+| calico | Network plugin Calico
+| calico_rr | Configuring Calico route reflector
+| canal | Network plugin Canal
+| cephfs-provisioner | Configuring CephFS
+| cert-manager | Configuring certificate manager for K8s
+| cilium | Network plugin Cilium
+| cinder-csi-driver | Configuring csi driver: cinder
+| client | Kubernetes clients role
+| cloud-provider | Cloud-provider related tasks
+| cluster-roles | Configuring cluster wide application (psp ...)
+| cni | CNI plugins for Network Plugins
+| containerd | Configuring containerd engine runtime for hosts
+| container_engine_accelerator | Enable nvidia accelerator for runtimes
+| container-engine | Configuring container engines
+| container-runtimes | Configuring container runtimes
+| coredns | Configuring coredns deployment
+| crio | Configuring crio container engine for hosts
+| crun | Configuring crun runtime
+| csi-driver | Configuring csi driver
+| dashboard | Installing and configuring the Kubernetes Dashboard
+| dns | Remove dns entries when resetting
+| docker | Configuring docker engine runtime for hosts
+| download | Fetching container images to a delegate host
+| etcd | Configuring etcd cluster
+| etcd-secrets | Configuring etcd certs/keys
+| etchosts | Configuring /etc/hosts entries for hosts
+| external-cloud-controller | Configure cloud controllers
+| external-openstack | Cloud controller : openstack
+| external-provisioner | Configure external provisioners
+| external-vsphere | Cloud controller : vsphere
+| facts | Gathering facts and misc check results
+| files | Remove files when resetting
+| flannel | Network plugin flannel
+| gce | Cloud-provider GCP
+| gcp-pd-csi-driver | Configuring csi driver: gcp-pd
+| gvisor | Configuring gvisor runtime
+| helm | Installing and configuring Helm
+| ingress-controller | Configure ingress controllers
+| ingress_alb | AWS ALB Ingress Controller
+| init | Windows kubernetes init nodes
+| iptables | Flush and clear iptable when resetting
+| k8s-pre-upgrade | Upgrading K8s cluster
+| k8s-secrets | Configuring K8s certs/keys
+| k8s-gen-tokens | Configuring K8s tokens
+| kata-containers | Configuring kata-containers runtime
+| krew | Install and manage krew
+| kubeadm | Roles linked to kubeadm tasks
+| kube-apiserver | Configuring static pod kube-apiserver
+| kube-controller-manager | Configuring static pod kube-controller-manager
+| kube-vip | Installing and configuring kube-vip
+| kubectl | Installing kubectl and bash completion
+| kubelet | Configuring kubelet service
+| kube-ovn | Network plugin kube-ovn
+| kube-router | Network plugin kube-router
+| kube-proxy | Configuring static pod kube-proxy
+| localhost | Special steps for the localhost (ansible runner)
+| local-path-provisioner | Configure External provisioner: local-path
+| local-volume-provisioner | Configure External provisioner: local-volume
+| macvlan | Network plugin macvlan
+| master | Configuring K8s master node role
+| metallb | Installing and configuring metallb
+| metrics_server | Configuring metrics_server
+| netchecker | Installing netchecker K8s app
+| network | Configuring networking plugins for K8s
+| mounts | Umount kubelet dirs when reseting
+| multus | Network plugin multus
+| nginx | Configuring LB for kube-apiserver instances
+| node | Configuring K8s minion (compute) node role
+| nodelocaldns | Configuring nodelocaldns daemonset
+| node-label | Tasks linked to labeling of nodes
+| node-webhook | Tasks linked to webhook (grating access to resources)
+| nvidia_gpu | Enable nvidia accelerator for runtimes
+| oci | Cloud provider: oci
+| persistent_volumes | Configure csi volumes
+| persistent_volumes_aws_ebs_csi | Configuring csi driver: aws-ebs
+| persistent_volumes_cinder_csi | Configuring csi driver: cinder
+| persistent_volumes_gcp_pd_csi | Configuring csi driver: gcp-pd
+| persistent_volumes_openstack | Configuring csi driver: openstack
+| policy-controller | Configuring Calico policy controller
+| post-remove | Tasks running post-remove operation
+| post-upgrade | Tasks running post-upgrade operation
+| pre-remove | Tasks running pre-remove operation
+| pre-upgrade | Tasks running pre-upgrade operation
+| preinstall | Preliminary configuration steps
+| registry | Configuring local docker registry
+| reset | Tasks running doing the node reset
+| resolvconf | Configuring /etc/resolv.conf for hosts/apps
+| rbd-provisioner | Configure External provisioner: rdb
+| services | Remove services (etcd, kubelet etc...) when resetting
+| snapshot | Enabling csi snapshot
+| snapshot-controller | Configuring csi snapshot controller
+| upgrade | Upgrading, f.e. container images/binaries
+| upload | Distributing images/binaries across hosts
+| vsphere-csi-driver | Configuring csi driver: vsphere
+| weave | Network plugin Weave
+| win_nodes | Running windows specific tasks
+| youki | Configuring youki runtime
+
+Note: Use the ``bash scripts/gen_tags.sh`` command to generate a list of all
+tags found in the codebase. New tags will be listed with the empty "Used for"
+field.
+
+## Example commands
+
+Example command to filter and apply only DNS configuration tasks and skip
+everything else related to host OS configuration and downloading images of containers:
+
+```ShellSession
+ansible-playbook -i inventory/sample/hosts.ini cluster.yml --tags preinstall,facts --skip-tags=download,bootstrap-os
+```
+
+And this play only removes the K8s cluster DNS resolver IP from hosts' /etc/resolv.conf files:
+
+```ShellSession
+ansible-playbook -i inventory/sample/hosts.ini -e dns_mode='none' cluster.yml --tags resolvconf
+```
+
+And this prepares all container images locally (at the ansible runner node) without installing
+or upgrading related stuff or trying to upload container to K8s cluster nodes:
+
+```ShellSession
+ansible-playbook -i inventory/sample/hosts.ini cluster.yml \
+ -e download_run_once=true -e download_localhost=true \
+ --tags download --skip-tags upload,upgrade
+```
+
+Note: use `--tags` and `--skip-tags` wise and only if you're 100% sure what you're doing.
+
+## Bastion host
+
+If you prefer to not make your nodes publicly accessible (nodes with private IPs only),
+you can use a so called *bastion* host to connect to your nodes. To specify and use a bastion,
+simply add a line to your inventory, where you have to replace x.x.x.x with the public IP of the
+bastion host.
+
+```ShellSession
+[bastion]
+bastion ansible_host=x.x.x.x
+```
+
+For more information about Ansible and bastion hosts, read
+[Running Ansible Through an SSH Bastion Host](https://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/)
+
+## Mitogen
+
+Mitogen support is deprecated, please see [mitogen related docs](/docs/mitogen.md) for useage and reasons for deprecation.
+
+## Beyond ansible 2.9
+
+Ansible project has decided, in order to ease their maintenance burden, to split between
+two projects which are now joined under the Ansible umbrella.
+
+Ansible-base (2.10.x branch) will contain just the ansible language implementation while
+ansible modules that were previously bundled into a single repository will be part of the
+ansible 3.x package. Pleasee see [this blog post](https://blog.while-true-do.io/ansible-release-3-0-0/)
+that explains in detail the need and the evolution plan.
+
+**Note:** this change means that ansible virtual envs cannot be upgraded with `pip install -U`.
+You first need to uninstall your old ansible (pre 2.10) version and install the new one.
+
+```ShellSession
+pip uninstall ansible ansible-base ansible-core
+cd kubespray/
+pip install -U .
+```
--- /dev/null
+# Architecture compatibility
+
+The following table shows the impact of the CPU architecture on compatible features:
+
+- amd64: Cluster using only x86/amd64 CPUs
+- arm64: Cluster using only arm64 CPUs
+- amd64 + arm64: Cluster with a mix of x86/amd64 and arm64 CPUs
+
+| kube_network_plugin | amd64 | arm64 | amd64 + arm64 |
+| ------------------- | ----- | ----- | ------------- |
+| Calico | Y | Y | Y |
+| Weave | Y | Y | Y |
+| Flannel | Y | N | N |
+| Canal | Y | N | N |
+| Cilium | Y | Y | N |
+| Contib | Y | N | N |
+| kube-router | Y | N | N |
--- /dev/null
+# AWS EBS CSI Driver
+
+AWS EBS CSI driver allows you to provision EBS volumes for pods in EC2 instances. The old in-tree AWS cloud provider is deprecated and will be removed in future versions of Kubernetes. So transitioning to the CSI driver is advised.
+
+To enable AWS EBS CSI driver, uncomment the `aws_ebs_csi_enabled` option in `group_vars/all/aws.yml` and set it to `true`.
+
+To set the number of replicas for the AWS CSI controller, you can change `aws_ebs_csi_controller_replicas` option in `group_vars/all/aws.yml`.
+
+Make sure to add a role, for your EC2 instances hosting Kubernetes, that allows it to do the actions necessary to request a volume and attach it: [AWS CSI Policy](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/example-iam-policy.json)
+
+If you want to deploy the AWS EBS storage class used with the CSI Driver, you should set `persistent_volumes_enabled` in `group_vars/k8s_cluster/k8s_cluster.yml` to `true`.
+
+You can now run the kubespray playbook (cluster.yml) to deploy Kubernetes over AWS EC2 with EBS CSI Driver enabled.
+
+## Usage example
+
+To check if AWS EBS CSI Driver is deployed properly, check that the ebs-csi pods are running:
+
+```ShellSession
+$ kubectl -n kube-system get pods | grep ebs
+ebs-csi-controller-85d86bccc5-8gtq5 4/4 Running 4 40s
+ebs-csi-node-n4b99 3/3 Running 3 40s
+```
+
+Check the associated storage class (if you enabled persistent_volumes):
+
+```ShellSession
+$ kubectl get storageclass
+NAME PROVISIONER AGE
+ebs-sc ebs.csi.aws.com 45s
+```
+
+You can run a PVC and an example Pod using this file `ebs-pod.yml`:
+
+```yml
+--
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: ebs-claim
+spec:
+ accessModes:
+ - ReadWriteOnce
+ storageClassName: ebs-sc
+ resources:
+ requests:
+ storage: 1Gi
+---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: app
+spec:
+ containers:
+ - name: app
+ image: centos
+ command: ["/bin/sh"]
+ args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
+ volumeMounts:
+ - name: persistent-storage
+ mountPath: /data
+ volumes:
+ - name: persistent-storage
+ persistentVolumeClaim:
+ claimName: ebs-claim
+```
+
+Apply this conf to your cluster: ```kubectl apply -f ebs-pod.yml```
+
+You should see the PVC provisioned and bound:
+
+```ShellSession
+$ kubectl get pvc
+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
+ebs-claim Bound pvc-0034cb9e-1ddd-4b3f-bb9e-0b5edbf5194c 1Gi RWO ebs-sc 50s
+```
+
+And the volume mounted to the example Pod (wait until the Pod is Running):
+
+```ShellSession
+$ kubectl exec -it app -- df -h | grep data
+/dev/nvme1n1 1014M 34M 981M 4% /data
+```
+
+## More info
+
+For further information about the AWS EBS CSI Driver, you can refer to this page: [AWS EBS Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/).
--- /dev/null
+# AWS
+
+To deploy kubespray on [AWS](https://aws.amazon.com/) uncomment the `cloud_provider` option in `group_vars/all.yml` and set it to `'aws'`. Refer to the [Kubespray Configuration](#kubespray-configuration) for customizing the provider.
+
+Prior to creating your instances, you **must** ensure that you have created IAM roles and policies for both "kubernetes-master" and "kubernetes-node". You can find the IAM policies [here](https://github.com/kubernetes-sigs/kubespray/tree/master/contrib/aws_iam/). See the [IAM Documentation](https://aws.amazon.com/documentation/iam/) if guidance is needed on how to set these up. When you bring your instances online, associate them with the respective IAM role. Nodes that are only to be used for Etcd do not need a role.
+
+You would also need to tag the resources in your VPC accordingly for the aws provider to utilize them. Tag the subnets, route tables and all instances that kubernetes will be run on with key `kubernetes.io/cluster/$cluster_name` (`$cluster_name` must be a unique identifier for the cluster). Tag the subnets that must be targeted by external ELBs with the key `kubernetes.io/role/elb` and internal ELBs with the key `kubernetes.io/role/internal-elb`.
+
+Make sure your VPC has both DNS Hostnames support and Private DNS enabled.
+
+The next step is to make sure the hostnames in your `inventory` file are identical to your internal hostnames in AWS. This may look something like `ip-111-222-333-444.us-west-2.compute.internal`. You can then specify how Ansible connects to these instances with `ansible_ssh_host` and `ansible_ssh_user`.
+
+You can now create your cluster!
+
+## Dynamic Inventory
+
+There is also a dynamic inventory script for AWS that can be used if desired. However, be aware that it makes some certain assumptions about how you'll create your inventory. It also does not handle all use cases and groups that we may use as part of more advanced deployments. Additions welcome.
+
+This will produce an inventory that is passed into Ansible that looks like the following:
+
+```json
+{
+ "_meta": {
+ "hostvars": {
+ "ip-172-31-3-xxx.us-east-2.compute.internal": {
+ "ansible_ssh_host": "172.31.3.xxx"
+ },
+ "ip-172-31-8-xxx.us-east-2.compute.internal": {
+ "ansible_ssh_host": "172.31.8.xxx"
+ }
+ }
+ },
+ "etcd": [
+ "ip-172-31-3-xxx.us-east-2.compute.internal"
+ ],
+ "k8s_cluster": {
+ "children": [
+ "kube_control_plane",
+ "kube_node"
+ ]
+ },
+ "kube_control_plane": [
+ "ip-172-31-3-xxx.us-east-2.compute.internal"
+ ],
+ "kube_node": [
+ "ip-172-31-8-xxx.us-east-2.compute.internal"
+ ]
+}
+```
+
+Guide:
+
+- Create instances in AWS as needed.
+- Either during or after creation, add tags to the instances with a key of `kubespray-role` and a value of `kube_control_plane`, `etcd`, or `kube_node`. You can also share roles like `kube_control_plane, etcd`
+- Copy the `kubespray-aws-inventory.py` script from `kubespray/contrib/aws_inventory` to the `kubespray/inventory` directory.
+- Set the following AWS credentials and info as environment variables in your terminal:
+
+```ShellSession
+export AWS_ACCESS_KEY_ID="xxxxx"
+export AWS_SECRET_ACCESS_KEY="yyyyy"
+export REGION="us-east-2"
+```
+
+- We will now create our cluster. There will be either one or two small changes. The first is that we will specify `-i inventory/kubespray-aws-inventory.py` as our inventory script. The other is conditional. If your AWS instances are public facing, you can set the `VPC_VISIBILITY` variable to `public` and that will result in public IP and DNS names being passed into the inventory. This causes your cluster.yml command to look like `VPC_VISIBILITY="public" ansible-playbook ... cluster.yml`
+
+## Kubespray configuration
+
+Declare the cloud config variables for the `aws` provider as follows. Setting these variables are optional and depend on your use case.
+
+Variable|Type|Comment
+---|---|---
+aws_zone|string|Force set the AWS zone. Recommended to leave blank.
+aws_vpc|string|The AWS VPC flag enables the possibility to run the master components on a different aws account, on a different cloud provider or on-premise. If the flag is set also the KubernetesClusterTag must be provided
+aws_subnet_id|string|SubnetID enables using a specific subnet to use for ELB's
+aws_route_table_id|string|RouteTableID enables using a specific RouteTable
+aws_role_arn|string|RoleARN is the IAM role to assume when interaction with AWS APIs
+aws_kubernetes_cluster_tag|string|KubernetesClusterTag is the legacy cluster id we'll use to identify our cluster resources
+aws_kubernetes_cluster_id|string|KubernetesClusterID is the cluster id we'll use to identify our cluster resources
+aws_disable_security_group_ingress|bool|The aws provider creates an inbound rule per load balancer on the node security group. However, this can run into the AWS security group rule limit of 50 if many LoadBalancers are created. This flag disables the automatic ingress creation. It requires that the user has setup a rule that allows inbound traffic on kubelet ports from the local VPC subnet (so load balancers can access it). E.g. 10.82.0.0/16 30000-32000.
+aws_elb_security_group|string|Only in Kubelet version >= 1.7 : AWS has a hard limit of 500 security groups. For large clusters creating a security group for each ELB can cause the max number of security groups to be reached. If this is set instead of creating a new Security group for each ELB this security group will be used instead.
+aws_disable_strict_zone_check|bool|During the instantiation of an new AWS cloud provider, the detected region is validated against a known set of regions. In a non-standard, AWS like environment (e.g. Eucalyptus), this check may be undesirable. Setting this to true will disable the check and provide a warning that the check was skipped. Please note that this is an experimental feature and work-in-progress for the moment.
--- /dev/null
+# Azure Disk CSI Driver
+
+The Azure Disk CSI driver allows you to provision volumes for pods with a Kubernetes deployment over Azure Cloud. The CSI driver replaces to volume provisioning done by the in-tree azure cloud provider which is deprecated.
+
+This documentation is an updated version of the in-tree Azure cloud provider documentation (azure.md).
+
+To deploy Azure Disk CSI driver, uncomment the `azure_csi_enabled` option in `group_vars/all/azure.yml` and set it to `true`.
+
+## Azure Disk CSI Storage Class
+
+If you want to deploy the Azure Disk storage class to provision volumes dynamically, you should set `persistent_volumes_enabled` in `group_vars/k8s_cluster/k8s_cluster.yml` to `true`.
+
+## Parameters
+
+Before creating the instances you must first set the `azure_csi_` variables in the `group_vars/all.yml` file.
+
+All of the values can be retrieved using the azure cli tool which can be downloaded here: <https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest>
+
+After installation you have to run `az login` to get access to your account.
+
+### azure\_csi\_tenant\_id + azure\_csi\_subscription\_id
+
+Run `az account show` to retrieve your subscription id and tenant id:
+`azure_csi_tenant_id` -> tenantId field
+`azure_csi_subscription_id` -> id field
+
+### azure\_csi\_location
+
+The region your instances are located in, it can be something like `francecentral` or `norwayeast`. A full list of region names can be retrieved via `az account list-locations`
+
+### azure\_csi\_resource\_group
+
+The name of the resource group your instances are in, a list of your resource groups can be retrieved via `az group list`
+
+Or you can do `az vm list | grep resourceGroup` and get the resource group corresponding to the VMs of your cluster.
+
+The resource group name is not case sensitive.
+
+### azure\_csi\_vnet\_name
+
+The name of the virtual network your instances are in, can be retrieved via `az network vnet list`
+
+### azure\_csi\_vnet\_resource\_group
+
+The name of the resource group your vnet is in, can be retrieved via `az network vnet list | grep resourceGroup` and get the resource group corresponding to the vnet of your cluster.
+
+### azure\_csi\_subnet\_name
+
+The name of the subnet your instances are in, can be retrieved via `az network vnet subnet list --resource-group RESOURCE_GROUP --vnet-name VNET_NAME`
+
+### azure\_csi\_security\_group\_name
+
+The name of the network security group your instances are in, can be retrieved via `az network nsg list`
+
+### azure\_csi\_aad\_client\_id + azure\_csi\_aad\_client\_secret
+
+These will have to be generated first:
+
+- Create an Azure AD Application with:
+`az ad app create --display-name kubespray --identifier-uris http://kubespray --homepage http://kubespray.com --password CLIENT_SECRET`
+
+Display name, identifier-uri, homepage and the password can be chosen
+
+Note the AppId in the output.
+
+- Create Service principal for the application with:
+`az ad sp create --id AppId`
+
+This is the AppId from the last command
+
+- Create the role assignment with:
+`az role assignment create --role "Owner" --assignee http://kubespray --subscription SUBSCRIPTION_ID`
+
+azure\_csi\_aad\_client\_id must be set to the AppId, azure\_csi\_aad\_client\_secret is your chosen secret.
+
+### azure\_csi\_use\_instance\_metadata
+
+Use instance metadata service where possible. Boolean value.
+
+## Test the Azure Disk CSI driver
+
+To test the dynamic provisioning using Azure CSI driver, make sure to have the storage class deployed (through persistent volumes), and apply the following manifest:
+
+```yml
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: pvc-azuredisk
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 1Gi
+ storageClassName: disk.csi.azure.com
+---
+kind: Pod
+apiVersion: v1
+metadata:
+ name: nginx-azuredisk
+spec:
+ nodeSelector:
+ kubernetes.io/os: linux
+ containers:
+ - image: nginx
+ name: nginx-azuredisk
+ command:
+ - "/bin/sh"
+ - "-c"
+ - while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
+ volumeMounts:
+ - name: azuredisk
+ mountPath: "/mnt/azuredisk"
+ volumes:
+ - name: azuredisk
+ persistentVolumeClaim:
+ claimName: pvc-azuredisk
+```
--- /dev/null
+# Azure
+
+To deploy Kubernetes on [Azure](https://azure.microsoft.com) uncomment the `cloud_provider` option in `group_vars/all/all.yml` and set it to `'azure'`.
+
+All your instances are required to run in a resource group and a routing table has to be attached to the subnet your instances are in.
+
+Not all features are supported yet though, for a list of the current status have a look [here](https://github.com/Azure/AKS)
+
+## Parameters
+
+Before creating the instances you must first set the `azure_` variables in the `group_vars/all/all.yml` file.
+
+All of the values can be retrieved using the Azure CLI tool which can be downloaded here: <https://docs.microsoft.com/en-gb/cli/azure/install-azure-cli>
+After installation you have to run `az login` to get access to your account.
+
+### azure_cloud
+
+Azure Stack has different API endpoints, depending on the Azure Stack deployment. These need to be provided to the Azure SDK.
+Possible values are: `AzureChinaCloud`, `AzureGermanCloud`, `AzurePublicCloud` and `AzureUSGovernmentCloud`.
+The full list of existing settings for the AzureChinaCloud, AzureGermanCloud, AzurePublicCloud and AzureUSGovernmentCloud
+is available in the source code [here](https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/docs/cloud-provider-config.md)
+
+### azure\_tenant\_id + azure\_subscription\_id
+
+run `az account show` to retrieve your subscription id and tenant id:
+`azure_tenant_id` -> Tenant ID field
+`azure_subscription_id` -> ID field
+
+### azure\_location
+
+The region your instances are located, can be something like `westeurope` or `westcentralus`. A full list of region names can be retrieved via `az account list-locations`
+
+### azure\_resource\_group
+
+The name of the resource group your instances are in, can be retrieved via `az group list`
+
+### azure\_vmtype
+
+The type of the vm. Supported values are `standard` or `vmss`. If vm is type of `Virtual Machines` then value is `standard`. If vm is part of `Virtual Machine Scale Sets` then value is `vmss`
+
+### azure\_vnet\_name
+
+The name of the virtual network your instances are in, can be retrieved via `az network vnet list`
+
+### azure\_vnet\_resource\_group
+
+The name of the resource group that contains the vnet.
+
+### azure\_subnet\_name
+
+The name of the subnet your instances are in, can be retrieved via `az network vnet subnet list --resource-group RESOURCE_GROUP --vnet-name VNET_NAME`
+
+### azure\_security\_group\_name
+
+The name of the network security group your instances are in, can be retrieved via `az network nsg list`
+
+### azure\_security\_group\_resource\_group
+
+The name of the resource group that contains the network security group. Defaults to `azure_vnet_resource_group`
+
+### azure\_route\_table\_name
+
+The name of the route table used with your instances.
+
+### azure\_route\_table\_resource\_group
+
+The name of the resource group that contains the route table. Defaults to `azure_vnet_resource_group`
+
+### azure\_aad\_client\_id + azure\_aad\_client\_secret
+
+These will have to be generated first:
+
+- Create an Azure AD Application with:
+`az ad app create --display-name kubernetes --identifier-uris http://kubernetes --homepage http://example.com --password CLIENT_SECRET`
+display name, identifier-uri, homepage and the password can be chosen
+Note the AppId in the output.
+- Create Service principal for the application with:
+`az ad sp create --id AppId`
+This is the AppId from the last command
+- Create the role assignment with:
+`az role assignment create --role "Owner" --assignee http://kubernetes --subscription SUBSCRIPTION_ID`
+
+azure\_aad\_client\_id must be set to the AppId, azure\_aad\_client\_secret is your chosen secret.
+
+### azure\_loadbalancer\_sku
+
+Sku of Load Balancer and Public IP. Candidate values are: basic and standard.
+
+### azure\_exclude\_master\_from\_standard\_lb
+
+azure\_exclude\_master\_from\_standard\_lb excludes master nodes from `standard` load balancer.
+
+### azure\_disable\_outbound\_snat
+
+azure\_disable\_outbound\_snat disables the outbound SNAT for public load balancer rules. It should only be set when azure\_exclude\_master\_from\_standard\_lb is `standard`.
+
+### azure\_primary\_availability\_set\_name
+
+(Optional) The name of the availability set that should be used as the load balancer backend .If this is set, the Azure
+cloudprovider will only add nodes from that availability set to the load balancer backend pool. If this is not set, and
+multiple agent pools (availability sets) are used, then the cloudprovider will try to add all nodes to a single backend
+pool which is forbidden. In other words, if you use multiple agent pools (availability sets), you MUST set this field.
+
+### azure\_use\_instance\_metadata
+
+Use instance metadata service where possible
+
+## Provisioning Azure with Resource Group Templates
+
+You'll find Resource Group Templates and scripts to provision the required infrastructure to Azure in [*contrib/azurerm*](../contrib/azurerm/README.md)
--- /dev/null
+# bootstrap-os
+
+Bootstrap an Ansible host to be able to run Ansible modules.
+
+This role will:
+
+* configure the package manager (if applicable) to be able to fetch packages
+* install Python
+* install the necessary packages to use Ansible's package manager modules
+* set the hostname of the host to `{{ inventory_hostname }}` when requested
+
+## Requirements
+
+A host running an operating system that is supported by Kubespray.
+See [Supported Linux Distributions](https://github.com/kubernetes-sigs/kubespray#supported-linux-distributions) for a current list.
+
+SSH access to the host.
+
+## Role Variables
+
+Variables are listed with their default values, if applicable.
+
+### General variables
+
+* `http_proxy`/`https_proxy`
+ The role will configure the package manager (if applicable) to download packages via a proxy.
+
+* `override_system_hostname: true`
+ The role will set the hostname of the machine to the name it has according to Ansible's inventory (the variable `{{ inventory_hostname }}`).
+
+### Per distribution variables
+
+#### Flatcar Container Linux
+
+* `coreos_locksmithd_disable: false`
+ Whether `locksmithd` (responsible for rolling restarts) should be disabled or be left alone.
+
+#### CentOS/RHEL/AlmaLinux/Rocky Linux
+
+* `centos_fastestmirror_enabled: false`
+ Whether the [fastestmirror](https://wiki.centos.org/PackageManagement/Yum/FastestMirror) yum plugin should be enabled.
+
+## Dependencies
+
+The `kubespray-defaults` role is expected to be run before this role.
+
+## Example Playbook
+
+Remember to disable fact gathering since Python might not be present on hosts.
+
+ - hosts: all
+ gather_facts: false # not all hosts might be able to run modules yet
+ roles:
+ - kubespray-defaults
+ - bootstrap-os
+
+## License
+
+Apache 2.0
--- /dev/null
+# Calico
+
+Check if the calico-node container is running
+
+```ShellSession
+docker ps | grep calico
+```
+
+The **calicoctl.sh** is wrap script with configured access credentials for command calicoctl allows to check the status of the network workloads.
+
+* Check the status of Calico nodes
+
+```ShellSession
+calicoctl.sh node status
+```
+
+* Show the configured network subnet for containers
+
+```ShellSession
+calicoctl.sh get ippool -o wide
+```
+
+* Show the workloads (ip addresses of containers and their location)
+
+```ShellSession
+calicoctl.sh get workloadEndpoint -o wide
+```
+
+and
+
+```ShellSession
+calicoctl.sh get hostEndpoint -o wide
+```
+
+## Configuration
+
+### Optional : Define datastore type
+
+The default datastore, Kubernetes API datastore is recommended for on-premises deployments, and supports only Kubernetes workloads; etcd is the best datastore for hybrid deployments.
+
+Allowed values are `kdd` (default) and `etcd`.
+
+Note: using kdd and more than 50 nodes, consider using the `typha` daemon to provide scaling.
+
+To re-define you need to edit the inventory and add a group variable `calico_datastore`
+
+```yml
+calico_datastore: kdd
+```
+
+### Optional : Define network backend
+
+In some cases you may want to define Calico network backend. Allowed values are `bird`, `vxlan` or `none`. `vxlan` is the default value.
+
+To re-define you need to edit the inventory and add a group variable `calico_network_backend`
+
+```yml
+calico_network_backend: none
+```
+
+### Optional : Define the default pool CIDRs
+
+By default, `kube_pods_subnet` is used as the IP range CIDR for the default IP Pool, and `kube_pods_subnet_ipv6` for IPv6.
+In some cases you may want to add several pools and not have them considered by Kubernetes as external (which means that they must be within or equal to the range defined in `kube_pods_subnet` and `kube_pods_subnet_ipv6` ), it starts with the default IP Pools of which IP range CIDRs can by defined in group_vars (k8s_cluster/k8s-net-calico.yml):
+
+```ShellSession
+calico_pool_cidr: 10.233.64.0/20
+calico_pool_cidr_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+```
+
+### Optional : BGP Peering with border routers
+
+In some cases you may want to route the pods subnet and so NAT is not needed on the nodes.
+For instance if you have a cluster spread on different locations and you want your pods to talk each other no matter where they are located.
+The following variables need to be set:
+`peer_with_router` to enable the peering with the datacenter's border router (default value: false).
+you'll need to edit the inventory and add a hostvar `local_as` by node.
+
+```ShellSession
+node1 ansible_ssh_host=95.54.0.12 local_as=xxxxxx
+```
+
+### Optional : Defining BGP peers
+
+Peers can be defined using the `peers` variable (see docs/calico_peer_example examples).
+In order to define global peers, the `peers` variable can be defined in group_vars with the "scope" attribute of each global peer set to "global".
+In order to define peers on a per node basis, the `peers` variable must be defined in hostvars.
+NB: Ansible's `hash_behaviour` is by default set to "replace", thus defining both global and per node peers would end up with having only per node peers. If having both global and per node peers defined was meant to happen, global peers would have to be defined in hostvars for each host (as well as per node peers)
+
+Since calico 3.4, Calico supports advertising Kubernetes service cluster IPs over BGP, just as it advertises pod IPs.
+This can be enabled by setting the following variable as follow in group_vars (k8s_cluster/k8s-net-calico.yml)
+
+```yml
+calico_advertise_cluster_ips: true
+```
+
+Since calico 3.10, Calico supports advertising Kubernetes service ExternalIPs over BGP in addition to cluster IPs advertising.
+This can be enabled by setting the following variable in group_vars (k8s_cluster/k8s-net-calico.yml)
+
+```yml
+calico_advertise_service_external_ips:
+- x.x.x.x/24
+- y.y.y.y/32
+```
+
+### Optional : Define global AS number
+
+Optional parameter `global_as_num` defines Calico global AS number (`/calico/bgp/v1/global/as_num` etcd key).
+It defaults to "64512".
+
+### Optional : BGP Peering with route reflectors
+
+At large scale you may want to disable full node-to-node mesh in order to
+optimize your BGP topology and improve `calico-node` containers' start times.
+
+To do so you can deploy BGP route reflectors and peer `calico-node` with them as
+recommended here:
+
+* <https://hub.docker.com/r/calico/routereflector/>
+* <https://docs.projectcalico.org/v3.1/reference/private-cloud/l3-interconnect-fabric>
+
+You need to edit your inventory and add:
+
+* `calico_rr` group with nodes in it. `calico_rr` can be combined with
+ `kube_node` and/or `kube_control_plane`. `calico_rr` group also must be a child
+ group of `k8s_cluster` group.
+* `cluster_id` by route reflector node/group (see details
+[here](https://hub.docker.com/r/calico/routereflector/))
+
+Here's an example of Kubespray inventory with standalone route reflectors:
+
+```ini
+[all]
+rr0 ansible_ssh_host=10.210.1.10 ip=10.210.1.10
+rr1 ansible_ssh_host=10.210.1.11 ip=10.210.1.11
+node2 ansible_ssh_host=10.210.1.12 ip=10.210.1.12
+node3 ansible_ssh_host=10.210.1.13 ip=10.210.1.13
+node4 ansible_ssh_host=10.210.1.14 ip=10.210.1.14
+node5 ansible_ssh_host=10.210.1.15 ip=10.210.1.15
+
+[kube_control_plane]
+node2
+node3
+
+[etcd]
+node2
+node3
+node4
+
+[kube_node]
+node2
+node3
+node4
+node5
+
+[k8s_cluster:children]
+kube_node
+kube_control_plane
+calico_rr
+
+[calico_rr]
+rr0
+rr1
+
+[rack0]
+rr0
+rr1
+node2
+node3
+node4
+node5
+
+[rack0:vars]
+cluster_id="1.0.0.1"
+calcio_rr_id=rr1
+calico_group_id=rr1
+```
+
+The inventory above will deploy the following topology assuming that calico's
+`global_as_num` is set to `65400`:
+
+
+
+### Optional : Define default endpoint to host action
+
+By default Calico blocks traffic from endpoints to the host itself by using an iptables DROP action. When using it in kubernetes the action has to be changed to RETURN (default in kubespray) or ACCEPT (see <https://github.com/projectcalico/felix/issues/660> and <https://github.com/projectcalico/calicoctl/issues/1389).> Otherwise all network packets from pods (with hostNetwork=False) to services endpoints (with hostNetwork=True) within the same node are dropped.
+
+To re-define default action please set the following variable in your inventory:
+
+```yml
+calico_endpoint_to_host_action: "ACCEPT"
+```
+
+### Optional : Define address on which Felix will respond to health requests
+
+Since Calico 3.2.0, HealthCheck default behavior changed from listening on all interfaces to just listening on localhost.
+
+To re-define health host please set the following variable in your inventory:
+
+```yml
+calico_healthhost: "0.0.0.0"
+```
+
+### Optional : Configure Calico Node probe timeouts
+
+Under certain conditions a deployer may need to tune the Calico liveness and readiness probes timeout settings. These can be configured like this:
+
+```yml
+calico_node_livenessprobe_timeout: 10
+calico_node_readinessprobe_timeout: 10
+```
+
+## Config encapsulation for cross server traffic
+
+Calico supports two types of encapsulation: [VXLAN and IP in IP](https://docs.projectcalico.org/v3.11/networking/vxlan-ipip). VXLAN is the more mature implementation and enabled by default, please check your environment if you need *IP in IP* encapsulation.
+
+*IP in IP* and *VXLAN* is mutualy exclusive modes.
+
+Kubespray defaults have changed after version 2.18 from auto-enabling `ipip` mode to auto-enabling `vxlan`. This was done to facilitate wider deployment scenarios including those where vxlan acceleration is provided by the underlying network devices.
+
+If you are running your cluster with the default calico settings and are upgrading to a release post 2.18.x (i.e. 2.19 and later or `master` branch) then you have two options:
+
+* perform a manual migration to vxlan before upgrading kubespray (see migrating from IP in IP to VXLAN below)
+* pin the pre-2.19 settings in your ansible inventory (see IP in IP mode settings below)
+
+### IP in IP mode
+
+To configure Ip in Ip mode you need to use the bird network backend.
+
+```yml
+calico_ipip_mode: 'Always' # Possible values is `Always`, `CrossSubnet`, `Never`
+calico_vxlan_mode: 'Never'
+calico_network_backend: 'bird'
+```
+
+### BGP mode
+
+To enable BGP no-encapsulation mode:
+
+```yml
+calico_ipip_mode: 'Never'
+calico_vxlan_mode: 'Never'
+calico_network_backend: 'bird'
+```
+
+### Migrating from IP in IP to VXLAN
+
+If you would like to migrate from the old IP in IP with `bird` network backends default to the new VXLAN based encapsulation you need to perform this change before running an upgrade of your cluster; the `cluster.yml` and `upgrade-cluster.yml` playbooks will refuse to continue if they detect incompatible settings.
+
+Execute the following sters on one of the control plane nodes, ensure the cluster in healthy before proceeding.
+
+```shell
+calicoctl.sh patch felixconfig default -p '{"spec":{"vxlanEnabled":true}}'
+calicoctl.sh patch ippool default-pool -p '{"spec":{"ipipMode":"Never", "vxlanMode":"Always"}}'
+```
+
+**Note:** if you created multiple ippools you will need to patch all of them individually to change their encapsulation. The kubespray playbooks only handle the default ippool creaded by kubespray.
+
+Wait for the `vxlan.calico` interfaces to be created on all cluster nodes and traffic to be routed through it then you can disable `ipip`.
+
+```shell
+calicoctl.sh patch felixconfig default -p '{"spec":{"ipipEnabled":false}}'
+```
+
+## Configuring interface MTU
+
+This is an advanced topic and should usually not be modified unless you know exactly what you are doing. Calico is smart enough to deal with the defaults and calculate the proper MTU. If you do need to set up a custom MTU you can change `calico_veth_mtu` as follows:
+
+* If Wireguard is enabled, subtract 60 from your network MTU (i.e. 1500-60=1440)
+* If using VXLAN or BPF mode is enabled, subtract 50 from your network MTU (i.e. 1500-50=1450)
+* If using IPIP, subtract 20 from your network MTU (i.e. 1500-20=1480)
+* if not using any encapsulation, set to your network MTU (i.e. 1500 or 9000)
+
+```yaml
+calico_veth_mtu: 1440
+```
+
+## Cloud providers configuration
+
+Please refer to the official documentation, for example [GCE configuration](http://docs.projectcalico.org/v1.5/getting-started/docker/installation/gce) requires a security rule for calico ip-ip tunnels. Note, calico is always configured with ``calico_ipip_mode: Always`` if the cloud provider was defined.
+
+### Optional : Ignore kernel's RPF check setting
+
+By default the felix agent(calico-node) will abort if the Kernel RPF setting is not 'strict'. If you want Calico to ignore the Kernel setting:
+
+```yml
+calico_node_ignorelooserpf: true
+```
+
+Note that in OpenStack you must allow `ipip` traffic in your security groups,
+otherwise you will experience timeouts.
+To do this you must add a rule which allows it, for example:
+
+### Optional : Felix configuration via extraenvs of calico node
+
+Possible environment variable parameters for [configuring Felix](https://docs.projectcalico.org/reference/felix/configuration)
+
+```yml
+calico_node_extra_envs:
+ FELIX_DEVICEROUTESOURCEADDRESS: 172.17.0.1
+```
+
+```ShellSession
+neutron security-group-rule-create --protocol 4 --direction egress k8s-a0tp4t
+neutron security-group-rule-create --protocol 4 --direction igress k8s-a0tp4t
+```
+
+### Optional : Use Calico CNI host-local IPAM plugin
+
+Calico currently supports two types of CNI IPAM plugins, `host-local` and `calico-ipam` (default).
+
+To allow Calico to determine the subnet to use from the Kubernetes API based on the `Node.podCIDR` field, enable the following setting.
+
+```yml
+calico_ipam_host_local: true
+```
+
+Refer to Project Calico section [Using host-local IPAM](https://docs.projectcalico.org/reference/cni-plugin/configuration#using-host-local-ipam) for further information.
+
+### Optional : Disable CNI logging to disk
+
+Calico CNI plugin logs to /var/log/calico/cni/cni.log and to stderr.
+stderr of CNI plugins can be found in the logs of container runtime.
+
+You can disable Calico CNI logging to disk by setting `calico_cni_log_file_path: false`.
+
+## eBPF Support
+
+Calico supports eBPF for its data plane see [an introduction to the Calico eBPF Dataplane](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/) for further information.
+
+Note that it is advisable to always use the latest version of Calico when using the eBPF dataplane.
+
+### Enabling eBPF support
+
+To enable the eBPF dataplane support ensure you add the following to your inventory. Note that the `kube-proxy` is incompatible with running Calico in eBPF mode and the kube-proxy should be removed from the system.
+
+```yaml
+calico_bpf_enabled: true
+```
+
+**NOTE:** there is known incompatibility in using the `kernel-kvm` kernel package on Ubuntu OSes because it is missing support for `CONFIG_NET_SCHED` which is a requirement for Calico eBPF support. When using Calico eBPF with Ubuntu ensure you run the `-generic` kernel.
+
+### Cleaning up after kube-proxy
+
+Calico node cannot clean up after kube-proxy has run in ipvs mode. If you are converting an existing cluster to eBPF you will need to ensure the `kube-proxy` DaemonSet is deleted and that ipvs rules are cleaned.
+
+To check that kube-proxy was running in ipvs mode:
+
+```ShellSession
+# ipvsadm -l
+```
+
+To clean up any ipvs leftovers:
+
+```ShellSession
+# ipvsadm -C
+```
+
+### Calico access to the kube-api
+
+Calico node, typha and kube-controllers need to be able to talk to the kubernetes API. Please reference the [Enabling eBPF Calico Docs](https://docs.projectcalico.org/maintenance/ebpf/enabling-bpf) for guidelines on how to do this.
+
+Kubespray sets up the `kubernetes-services-endpoint` configmap based on the contents of the `loadbalancer_apiserver` inventory variable documented in [HA Mode](/docs/ha-mode.md).
+
+If no external loadbalancer is used, Calico eBPF can also use the localhost loadbalancer option. In this case Calico Automatic Host Endpoints need to be enabled to allow services like `coredns` and `metrics-server` to communicate with the kubernetes host endpoint. See [this blog post](https://www.projectcalico.org/securing-kubernetes-nodes-with-calico-automatic-host-endpoints/) on enabling automatic host endpoints.
+
+```yaml
+loadbalancer_apiserver_localhost: true
+use_localhost_as_kubeapi_loadbalancer: true
+```
+
+### Tunneled versus Direct Server Return
+
+By default Calico usese Tunneled service mode but it can use direct server return (DSR) in order to optimize the return path for a service.
+
+To configure DSR:
+
+```yaml
+calico_bpf_service_mode: "DSR"
+```
+
+### eBPF Logging and Troubleshooting
+
+In order to enable Calico eBPF mode logging:
+
+```yaml
+calico_bpf_log_level: "Debug"
+```
+
+To view the logs you need to use the `tc` command to read the kernel trace buffer:
+
+```ShellSession
+tc exec bpf debug
+```
+
+Please see [Calico eBPF troubleshooting guide](https://docs.projectcalico.org/maintenance/troubleshoot/troubleshoot-ebpf#ebpf-program-debug-logs).
+
+## Wireguard Encryption
+
+Calico supports using Wireguard for encryption. Please see the docs on [encryptiong cluster pod traffic](https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic).
+
+To enable wireguard support:
+
+```yaml
+calico_wireguard_enabled: true
+```
+
+The following OSes will require enabling the EPEL repo in order to bring in wireguard tools:
+
+* CentOS 7 & 8
+* AlmaLinux 8
+* Rocky Linux 8
+* Amazon Linux 2
+
+```yaml
+epel_enabled: true
+```
--- /dev/null
+# ---
+# peers:
+# - router_id: "10.99.0.34"
+# as: "65xxx"
+# sourceaddress: "None"
+# - router_id: "10.99.0.35"
+# as: "65xxx"
+# sourceaddress: "None"
+
+# loadbalancer_apiserver:
+# address: "10.99.0.44"
+# port: "8383"
--- /dev/null
+# ---
+# peers:
+# - router_id: "10.99.0.2"
+# as: "65xxx"
+# sourceaddress: "None"
+# - router_id: "10.99.0.3"
+# as: "65xxx"
+# sourceaddress: "None"
+
+# loadbalancer_apiserver:
+# address: "10.99.0.21"
+# port: "8383"
--- /dev/null
+# CentOS and derivatives
+
+## CentOS 7
+
+The maximum python version offically supported in CentOS is 3.6. Ansible as of version 5 (ansible core 2.12.x) increased their python requirement to python 3.8 and above.
+Kubespray supports multiple ansible versions but only the default (5.x) gets wide testing coverage. If your deployment host is CentOS 7 it is recommended to use one of the earlier versions still supported.
+
+## CentOS 8
+
+CentOS 8 / Oracle Linux 8 / AlmaLinux 8 / Rocky Linux 8 ship only with iptables-nft (ie without iptables-legacy similar to RHEL8)
+The only tested configuration for now is using Calico CNI
+You need to add `calico_iptables_backend: "NFT"` to your configuration.
+
+If you have containers that are using iptables in the host network namespace (`hostNetwork=true`),
+you need to ensure they are using iptables-nft.
+An example how k8s do the autodetection can be found [in this PR](https://github.com/kubernetes/kubernetes/pull/82966)
--- /dev/null
+# Installation Guide
+
+- [Installation Guide](#installation-guide)
+ - [Kubernetes TLS Root CA Certificate/Key Secret](#kubernetes-tls-root-ca-certificatekey-secret)
+ - [Securing Ingress Resources](#securing-ingress-resources)
+ - [Create New TLS Root CA Certificate and Key](#create-new-tls-root-ca-certificate-and-key)
+ - [Install Cloudflare PKI/TLS `cfssl` Toolkit.](#install-cloudflare-pkitls-cfssl-toolkit)
+ - [Create Root Certificate Authority (CA) Configuration File](#create-root-certificate-authority-ca-configuration-file)
+ - [Create Certficate Signing Request (CSR) Configuration File](#create-certficate-signing-request-csr-configuration-file)
+ - [Create TLS Root CA Certificate and Key](#create-tls-root-ca-certificate-and-key)
+
+Cert-Manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self signed. It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
+
+## Kubernetes TLS Root CA Certificate/Key Secret
+
+If you're planning to secure your ingress resources using TLS client certificates, you'll need to create and deploy the Kubernetes `ca-key-pair` secret consisting of the Root CA certificate and key to your K8s cluster.
+
+For further information, read the official [Cert-Manager CA Configuration](https://cert-manager.io/docs/configuration/ca/) doc.
+
+`cert-manager` can now be enabled by editing your K8s cluster addons inventory e.g. `inventory\sample\group_vars\k8s_cluster\addons.yml` and setting `cert_manager_enabled` to true.
+
+```ini
+# Cert manager deployment
+cert_manager_enabled: true
+```
+
+If you don't have a TLS Root CA certificate and key available, you can create these by following the steps outlined in section [Create New TLS Root CA Certificate and Key](#create-new-tls-root-ca-certificate-and-key) using the Cloudflare PKI/TLS `cfssl` toolkit. TLS Root CA certificates and keys can also be created using `ssh-keygen` and OpenSSL, if `cfssl` is not available.
+
+## Securing Ingress Resources
+
+A common use-case for cert-manager is requesting TLS signed certificates to secure your ingress resources. This can be done by simply adding annotations to your Ingress resources and cert-manager will facilitate creating the Certificate resource for you. A small sub-component of cert-manager, ingress-shim, is responsible for this.
+
+To enable the Nginx Ingress controller as part of your Kubespray deployment, simply edit your K8s cluster addons inventory e.g. `inventory\sample\group_vars\k8s_cluster\addons.yml` and set `ingress_nginx_enabled` to true.
+
+```ini
+# Nginx ingress controller deployment
+ingress_nginx_enabled: true
+```
+
+For example, if you're using the Nginx ingress controller, you can secure the Prometheus ingress by adding the annotation `cert-manager.io/cluster-issuer: ca-issuer` and the `spec.tls` section to the `Ingress` resource definition.
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: prometheus-k8s
+ namespace: monitoring
+ labels:
+ prometheus: k8s
+ annotations:
+ kubernetes.io/ingress.class: "nginx"
+ cert-manager.io/cluster-issuer: ca-issuer
+spec:
+ tls:
+ - hosts:
+ - prometheus.example.com
+ secretName: prometheus-dashboard-certs
+ rules:
+ - host: prometheus.example.com
+ http:
+ paths:
+ - path: /
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: prometheus-k8s
+ port:
+ name: web
+```
+
+Once deployed to your K8s cluster, every 3 months cert-manager will automatically rotate the Prometheus `prometheus.example.com` TLS client certificate and key, and store these as the Kubernetes `prometheus-dashboard-certs` secret.
+
+Please consult the official upstream documentation:
+
+- [cert-manager Ingress Usage](https://cert-manager.io/v1.5-docs/usage/ingress/)
+- [cert-manager Ingress Tutorial](https://cert-manager.io/v1.5-docs/tutorials/acme/ingress/#step-3-assign-a-dns-name)
+
+### ACME
+
+The ACME Issuer type represents a single account registered with the Automated Certificate Management Environment (ACME) Certificate Authority server. When you create a new ACME Issuer, cert-manager will generate a private key which is used to identify you with the ACME server.
+
+Certificates issued by public ACME servers are typically trusted by client’s computers by default. This means that, for example, visiting a website that is backed by an ACME certificate issued for that URL, will be trusted by default by most client’s web browsers. ACME certificates are typically free.
+
+- [ACME Configuration](https://cert-manager.io/v1.5-docs/configuration/acme/)
+- [ACME HTTP Validation](https://cert-manager.io/v1.5-docs/tutorials/acme/http-validation/)
+ - [HTTP01 Challenges](https://cert-manager.io/v1.5-docs/configuration/acme/http01/)
+- [ACME DNS Validation](https://cert-manager.io/v1.5-docs/tutorials/acme/dns-validation/)
+ - [DNS01 Challenges](https://cert-manager.io/v1.5-docs/configuration/acme/dns01/)
+- [ACME FAQ](https://cert-manager.io/v1.5-docs/faq/acme/)
+
+#### ACME With An Internal Certificate Authority
+
+The ACME Issuer with an internal certificate authority requires cert-manager to trust the certificate authority. This trust must be done at the cert-manager deployment level.
+To add a trusted certificate authority to cert-manager, add it's certificate to `group_vars/k8s-cluster/addons.yml`:
+
+```yaml
+cert_manager_trusted_internal_ca: |
+ -----BEGIN CERTIFICATE-----
+ [REPLACE with your CA certificate]
+ -----END CERTIFICATE-----
+```
+
+Once the CA is trusted, you can define your issuer normally.
+
+### Create New TLS Root CA Certificate and Key
+
+#### Install Cloudflare PKI/TLS `cfssl` Toolkit
+
+e.g. For Ubuntu/Debian distributions, the toolkit is part of the `golang-cfssl` package.
+
+```shell
+sudo apt-get install -y golang-cfssl
+```
+
+#### Create Root Certificate Authority (CA) Configuration File
+
+The default TLS certificate expiry time period is `8760h` which is 5 years from the date the certificate is created.
+
+```shell
+$ cat > ca-config.json <<EOF
+{
+ "signing": {
+ "default": {
+ "expiry": "8760h"
+ },
+ "profiles": {
+ "kubernetes": {
+ "usages": ["signing", "key encipherment", "server auth", "client auth"],
+ "expiry": "8760h"
+ }
+ }
+ }
+}
+EOF
+```
+
+#### Create Certficate Signing Request (CSR) Configuration File
+
+The TLS certificate `names` details can be updated to your own specific requirements.
+
+```shell
+$ cat > ca-csr.json <<EOF
+{
+ "CN": "Kubernetes",
+ "key": {
+ "algo": "rsa",
+ "size": 2048
+ },
+ "names": [
+ {
+ "C": "US",
+ "L": "Portland",
+ "O": "Kubernetes",
+ "OU": "CA",
+ "ST": "Oregon"
+ }
+ ]
+}
+EOF
+```
+
+#### Create TLS Root CA Certificate and Key
+
+```shell
+$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
+ca.pem
+ca-key.pem
+```
+
+Check the TLS Root CA certificate has the correct `Not Before` and `Not After` dates, and ensure it is indeed a valid Certificate Authority with the X509v3 extension `CA:TRUE`.
+
+```shell
+$ openssl x509 -text -noout -in ca.pem
+
+Certificate:
+ Data:
+ Version: 3 (0x2)
+ Serial Number:
+ 6a:d4:d8:48:7f:98:4f:54:68:9a:e1:73:02:fa:d0:41:79:25:08:49
+ Signature Algorithm: sha256WithRSAEncryption
+ Issuer: C = US, ST = Oregon, L = Portland, O = Kubernetes, OU = CA, CN = Kubernetes
+ Validity
+ Not Before: Jul 10 15:21:00 2020 GMT
+ Not After : Jul 9 15:21:00 2025 GMT
+ Subject: C = US, ST = Oregon, L = Portland, O = Kubernetes, OU = CA, CN = Kubernetes
+ Subject Public Key Info:
+ ...
+ X509v3 extensions:
+ X509v3 Key Usage: critical
+ Certificate Sign, CRL Sign
+ X509v3 Basic Constraints: critical
+ CA:TRUE
+ X509v3 Subject Key Identifier:
+ D4:38:B5:E2:26:49:5E:0D:E3:DC:D9:70:73:3B:C4:19:6A:43:4A:F2
+ ...
+```
--- /dev/null
+# CI Setup
+
+## Pipeline
+
+1. unit-tests: fast jobs for fast feedback (linting, etc...)
+2. deploy-part1: small number of jobs to test if the PR works with default settings
+3. deploy-part2: slow jobs testing different platforms, OS, settings, CNI, etc...
+4. deploy-part3: very slow jobs (upgrades, etc...)
+
+## Runners
+
+Kubespray has 3 types of GitLab runners:
+
+- packet runners: used for E2E jobs (usually long)
+- light runners: used for short lived jobs
+- auto scaling runners: used for on-demand resources, see [GitLab docs](https://docs.gitlab.com/runner/configuration/autoscale.html) for more info
+
+## Vagrant
+
+Vagrant jobs are using the [quay.io/kubespray/vagrant](/test-infra/vagrant-docker/Dockerfile) docker image with `/var/run/libvirt/libvirt-sock` exposed from the host, allowing the container to boot VMs on the host.
+
+## CI Variables
+
+In CI we have a set of overrides we use to ensure greater success of our CI jobs and avoid throttling by various APIs we depend on. See:
+
+- [Docker mirrors](/tests/common/_docker_hub_registry_mirror.yml)
+- [Test settings](/tests/common/_kubespray_test_settings.yml)
--- /dev/null
+# CI test coverage
+
+To generate this Matrix run `./tests/scripts/md-table/main.py`
+
+## containerd
+
+| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
+amazon | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
+debian10 | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
+debian11 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian9 | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
+fedora35 | :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: |
+fedora36 | :x: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
+opensuse | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: |
+ubuntu18 | :white_check_mark: | :x: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :white_check_mark: |
+ubuntu20 | :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: |
+ubuntu22 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+
+## crio
+
+| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+amazon | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian10 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian11 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian9 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora35 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora36 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+opensuse | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu18 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu20 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu22 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+
+## docker
+
+| OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
+|---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+amazon | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+centos7 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian10 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian11 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+debian9 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora35 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+fedora36 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
+opensuse | :x: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu16 | :x: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
+ubuntu18 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu20 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
+ubuntu22 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
--- /dev/null
+# Cilium
+
+## Kube-proxy replacement with Cilium
+
+Cilium can run without kube-proxy by setting `cilium_kube_proxy_replacement`
+to `strict`.
+
+Without kube-proxy, cilium needs to know the address of the kube-apiserver
+and this must be set globally for all cilium components (agents and operators).
+Hence, in this configuration in Kubespray, Cilium will always contact
+the external loadbalancer (even from a node in the control plane)
+and if there is no external load balancer It will ignore any local load
+balancer deployed by Kubespray and **only contacts the first master**.
+
+## Cilium Operator
+
+Unlike some operators, Cilium Operator does not exist for installation purposes.
+> The Cilium Operator is responsible for managing duties in the cluster which should logically be handled once for the entire cluster, rather than once for each node in the cluster.
+
+### Adding custom flags to the Cilium Operator
+
+You can set additional cilium-operator container arguments using `cilium_operator_custom_args`.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+Accepts an array or a string.
+
+```yml
+cilium_operator_custom_args: ["--foo=bar", "--baz=qux"]
+```
+
+or
+
+```yml
+cilium_operator_custom_args: "--foo=bar"
+```
+
+You do not need to add a custom flag to enable debugging. Instead, feel free to use the `CILIUM_DEBUG` variable.
+
+### Adding extra volumes and mounting them
+
+You can use `cilium_operator_extra_volumes` to add extra volumes to the Cilium Operator, and use `cilium_operator_extra_volume_mounts` to mount those volumes.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+```yml
+cilium_operator_extra_volumes:
+ - configMap:
+ name: foo
+ name: foo-mount-path
+
+cilium_operator_extra_volume_mounts:
+ - mountPath: /tmp/foo/bar
+ name: foo-mount-path
+ readOnly: true
+```
+
+## Choose Cilium version
+
+```yml
+cilium_version: v1.11.3
+```
+
+## Add variable to config
+
+Use following variables:
+
+Example:
+
+```yml
+cilium_config_extra_vars:
+ enable-endpoint-routes: true
+```
+
+## Change Identity Allocation Mode
+
+Cilium assigns an identity for each endpoint. This identity is used to enforce basic connectivity between endpoints.
+
+Cilium currently supports two different identity allocation modes:
+
+- "crd" stores identities in kubernetes as CRDs (custom resource definition).
+ - These can be queried with `kubectl get ciliumid`
+- "kvstore" stores identities in an etcd kvstore.
+
+## Enable Transparent Encryption
+
+Cilium supports the transparent encryption of Cilium-managed host traffic and
+traffic between Cilium-managed endpoints either using IPsec or Wireguard.
+
+Wireguard option is only available in Cilium 1.10.0 and newer.
+
+### IPsec Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/)
+
+To enable IPsec encryption, you just need to set three variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "ipsec"
+```
+
+The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable.
+Kubespray does not automate this process.
+Cilium documentation currently recommends creating a key using the following command:
+
+```shell
+echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"
+```
+
+Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable.
+
+### Wireguard Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/)
+
+To enable Wireguard encryption, you just need to set two variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "wireguard"
+```
+
+Kubespray currently supports Linux distributions with Wireguard Kernel mode on Linux 5.6 and newer.
+
+## Install Cilium Hubble
+
+k8s-net-cilium.yml:
+
+```yml
+cilium_enable_hubble: true ## enable support hubble in cilium
+cilium_hubble_install: true ## install hubble-relay, hubble-ui
+cilium_hubble_tls_generate: true ## install hubble-certgen and generate certificates
+```
+
+To validate that Hubble UI is properly configured, set up a port forwarding for hubble-ui service:
+
+```shell script
+kubectl port-forward -n kube-system svc/hubble-ui 12000:80
+```
+
+and then open [http://localhost:12000/](http://localhost:12000/).
+
+## Hubble metrics
+
+```yml
+cilium_enable_hubble_metrics: true
+cilium_hubble_metrics:
+ - dns
+ - drop
+ - tcp
+ - flow
+ - icmp
+ - http
+```
+
+[More](https://docs.cilium.io/en/v1.9/operations/metrics/#hubble-exported-metrics)
+
+## Upgrade considerations
+
+### Rolling-restart timeouts
+
+Cilium relies on the kernel's BPF support, which is extremely fast at runtime but incurs a compilation penalty on initialization and update.
+
+As a result, the Cilium DaemonSet pods can take a significant time to start, which scales with the number of nodes and endpoints in your cluster.
+
+As part of cluster.yml, this DaemonSet is restarted, and Kubespray's [default timeouts for this operation](../roles/network_plugin/cilium/defaults/main.yml)
+are not appropriate for large clusters.
+
+This means that you will likely want to update these timeouts to a value more in-line with your cluster's number of nodes and their respective CPU performance.
+This is configured by the following values:
+
+```yaml
+# Configure how long to wait for the Cilium DaemonSet to be ready again
+cilium_rolling_restart_wait_retries_count: 30
+cilium_rolling_restart_wait_retries_delay_seconds: 10
+```
+
+The total time allowed (count * delay) should be at least `($number_of_nodes_in_cluster * $cilium_pod_start_time)` for successful rolling updates. There are no
+drawbacks to making it higher and giving yourself a time buffer to accommodate transient slowdowns.
+
+Note: To find the `$cilium_pod_start_time` for your cluster, you can simply restart a Cilium pod on a node of your choice and look at how long it takes for it
+to become ready.
+
+Note 2: The default CPU requests/limits for Cilium pods is set to a very conservative 100m:500m which will likely yield very slow startup for Cilium pods. You
+probably want to significantly increase the CPU limit specifically if short bursts of CPU from Cilium are acceptable to you.
--- /dev/null
+# Cinder CSI Driver
+
+Cinder CSI driver allows you to provision volumes over an OpenStack deployment. The Kubernetes historic in-tree cloud provider is deprecated and will be removed in future versions.
+
+To enable Cinder CSI driver, uncomment the `cinder_csi_enabled` option in `group_vars/all/openstack.yml` and set it to `true`.
+
+To set the number of replicas for the Cinder CSI controller, you can change `cinder_csi_controller_replicas` option in `group_vars/all/openstack.yml`.
+
+You need to source the OpenStack credentials you use to deploy your machines that will host Kubernetes: `source path/to/your/openstack-rc` or `. path/to/your/openstack-rc`.
+
+Make sure the hostnames in your `inventory` file are identical to your instance names in OpenStack. Otherwise [cinder](https://docs.openstack.org/cinder/latest/) won't work as expected.
+
+If you want to deploy the cinder provisioner used with Cinder CSI Driver, you should set `persistent_volumes_enabled` in `group_vars/k8s_cluster/k8s_cluster.yml` to `true`.
+
+You can now run the kubespray playbook (cluster.yml) to deploy Kubernetes over OpenStack with Cinder CSI Driver enabled.
+
+## Usage example
+
+To check if Cinder CSI Driver works properly, see first that the cinder-csi pods are running:
+
+```ShellSession
+$ kubectl -n kube-system get pods | grep cinder
+csi-cinder-controllerplugin-7f8bf99785-cpb5v 5/5 Running 0 100m
+csi-cinder-nodeplugin-rm5x2 2/2 Running 0 100m
+```
+
+Check the associated storage class (if you enabled persistent_volumes):
+
+```ShellSession
+$ kubectl get storageclass
+NAME PROVISIONER AGE
+cinder-csi cinder.csi.openstack.org 100m
+```
+
+You can run a PVC and an Nginx Pod using this file `nginx.yaml`:
+
+```yml
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: csi-pvc-cinderplugin
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 1Gi
+ storageClassName: cinder-csi
+
+---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: nginx
+spec:
+ containers:
+ - image: nginx
+ imagePullPolicy: IfNotPresent
+ name: nginx
+ ports:
+ - containerPort: 80
+ protocol: TCP
+ volumeMounts:
+ - mountPath: /var/lib/www/html
+ name: csi-data-cinderplugin
+ volumes:
+ - name: csi-data-cinderplugin
+ persistentVolumeClaim:
+ claimName: csi-pvc-cinderplugin
+ readOnly: false
+```
+
+Apply this conf to your cluster: ```kubectl apply -f nginx.yml```
+
+You should see the PVC provisioned and bound:
+
+```ShellSession
+$ kubectl get pvc
+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
+csi-pvc-cinderplugin Bound pvc-f21ad0a1-5b7b-405e-a462-48da5cb76beb 1Gi RWO cinder-csi 8s
+```
+
+And the volume mounted to the Nginx Pod (wait until the Pod is Running):
+
+```ShellSession
+kubectl exec -it nginx -- df -h | grep /var/lib/www/html
+/dev/vdb 976M 2.6M 958M 1% /var/lib/www/html
+```
+
+## Compatibility with in-tree cloud provider
+
+It is not necessary to enable OpenStack as a cloud provider for Cinder CSI Driver to work.
+Though, you can run both the in-tree openstack cloud provider and the Cinder CSI Driver at the same time. The storage class provisioners associated to each one of them are differently named.
+
+## Cinder v2 support
+
+For the moment, only Cinder v3 is supported by the CSI Driver.
+
+## More info
+
+For further information about the Cinder CSI Driver, you can refer to this page: [Cloud Provider OpenStack](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md).
--- /dev/null
+# Cloud providers
+
+## Provisioning
+
+You can deploy instances in your cloud environment in several different ways. Examples include Terraform, Ansible (ec2 and gce modules), and manual creation.
+
+## Deploy kubernetes
+
+With ansible-playbook command
+
+```ShellSession
+ansible-playbook -u smana -e ansible_ssh_user=admin -e cloud_provider=[aws|gce] -b --become-user=root -i inventory/single.cfg cluster.yml
+```
--- /dev/null
+CNI
+==============
+
+This network plugin only unpacks CNI plugins version `cni_version` into `/opt/cni/bin` and instructs kubelet to use cni, that is adds following cli params:
+
+`KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"`
+
+It's intended usage is for custom CNI configuration, e.g. manual routing tables + bridge + loopback CNI plugin outside kubespray scope. Furthermore, it's used for non-kubespray supported CNI plugins which you can install afterward.
+
+You are required to fill `/etc/cni/net.d` with valid CNI configuration after using kubespray.
--- /dev/null
+# Comparison
+
+## Kubespray vs Kops
+
+Kubespray runs on bare metal and most clouds, using Ansible as its substrate for
+provisioning and orchestration. [Kops](https://github.com/kubernetes/kops) performs the provisioning and orchestration
+itself, and as such is less flexible in deployment platforms. For people with
+familiarity with Ansible, existing Ansible deployments or the desire to run a
+Kubernetes cluster across multiple platforms, Kubespray is a good choice. Kops,
+however, is more tightly integrated with the unique features of the clouds it
+supports so it could be a better choice if you know that you will only be using
+one platform for the foreseeable future.
+
+## Kubespray vs Kubeadm
+
+[Kubeadm](https://github.com/kubernetes/kubeadm) provides domain Knowledge of Kubernetes clusters' life cycle
+management, including self-hosted layouts, dynamic discovery services and so
+on. Had it belonged to the new [operators world](https://coreos.com/blog/introducing-operators.html),
+it may have been named a "Kubernetes cluster operator". Kubespray however,
+does generic configuration management tasks from the "OS operators" ansible
+world, plus some initial K8s clustering (with networking plugins included) and
+control plane bootstrapping.
+
+Kubespray has started using `kubeadm` internally for cluster creation since v2.3
+in order to consume life cycle management domain knowledge from it
+and offload generic OS configuration things from it, which hopefully benefits both sides.
--- /dev/null
+# containerd
+
+[containerd] An industry-standard container runtime with an emphasis on simplicity, robustness and portability
+Kubespray supports basic functionality for using containerd as the default container runtime in a cluster.
+
+_To use the containerd container runtime set the following variables:_
+
+## k8s_cluster.yml
+
+When kube_node contains etcd, you define your etcd cluster to be as well schedulable for Kubernetes workloads. Thus containerd and dockerd can not run at same time, must be set to bellow for running etcd cluster with only containerd.
+
+```yaml
+container_manager: containerd
+```
+
+## etcd.yml
+
+```yaml
+etcd_deployment_type: host
+```
+
+## Containerd config
+
+Example: define registry mirror for docker hub
+
+```yaml
+containerd_registries:
+ "docker.io":
+ - "https://mirror.gcr.io"
+ - "https://registry-1.docker.io"
+```
+
+`containerd_registries` is ignored for pulling images when `image_command_tool=nerdctl`
+(the default for `container_manager=containerd`). Use `crictl` instead, it supports
+`containerd_registries` but lacks proper multi-arch support (see
+[#8375](https://github.com/kubernetes-sigs/kubespray/issues/8375)):
+
+```yaml
+image_command_tool: crictl
+```
+
+[containerd]: https://containerd.io/
--- /dev/null
+# CRI-O
+
+[CRI-O] is a lightweight container runtime for Kubernetes.
+Kubespray supports basic functionality for using CRI-O as the default container runtime in a cluster.
+
+* Kubernetes supports CRI-O on v1.11.1 or later.
+* etcd: configure either kubeadm managed etcd or host deployment
+
+_To use the CRI-O container runtime set the following variables:_
+
+## all/all.yml
+
+```yaml
+download_container: false
+skip_downloads: false
+etcd_deployment_type: host # optionally kubeadm
+```
+
+## k8s_cluster/k8s_cluster.yml
+
+```yaml
+container_manager: crio
+```
+
+## all/crio.yml
+
+Enable docker hub registry mirrors
+
+```yaml
+crio_registries:
+ - prefix: docker.io
+ insecure: false
+ blocked: false
+ location: registry-1.docker.io
+ unqualified: false
+ mirrors:
+ - location: 192.168.100.100:5000
+ insecure: true
+ - location: mirror.gcr.io
+ insecure: false
+```
+
+## Note about pids_limit
+
+For heavily mult-threaded workloads like databases, the default of 1024 for pids-limit is too low.
+This parameter controls not just the number of processes but also the amount of threads
+(since a thread is technically a process with shared memory). See [cri-o#1921]
+
+In order to increase the default `pids_limit` for cri-o based deployments you need to set the `crio_pids_limit`
+for your `k8s_cluster` ansible group or per node depending on the use case.
+
+```yaml
+crio_pids_limit: 4096
+```
+
+[CRI-O]: https://cri-o.io/
+[cri-o#1921]: https://github.com/cri-o/cri-o/issues/1921
+
+## Note about user namespaces
+
+CRI-O has support for user namespaces. This feature is optional and can be enabled by setting the following two variables.
+
+```yaml
+crio_runtimes:
+ - name: runc
+ path: /usr/bin/runc
+ type: oci
+ root: /run/runc
+ allowed_annotations:
+ - "io.kubernetes.cri-o.userns-mode"
+
+crio_remap_enable: true
+```
+
+The `allowed_annotations` configures `crio.conf` accordingly.
+
+The `crio_remap_enable` configures the `/etc/subuid` and `/etc/subgid` files to add an entry for the **containers** user.
+By default, 16M uids and gids are reserved for user namespaces (256 pods * 65536 uids/gids) at the end of the uid/gid space.
--- /dev/null
+# Debian Jessie
+
+Debian Jessie installation Notes:
+
+- Add
+
+ ```GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"```
+
+ to /etc/default/grub. Then update with
+
+ ```ShellSession
+ sudo update-grub
+ sudo update-grub2
+ sudo reboot
+ ```
+
+- Add the [backports](https://backports.debian.org/Instructions/) which contain Systemd 2.30 and update Systemd.
+
+ ```apt-get -t jessie-backports install systemd```
+
+ (Necessary because the default Systemd version (2.15) does not support the "Delegate" directive in service files)
+
+- Add the Ansible repository and install Ansible to get a proper version
+
+ ```ShellSession
+ sudo add-apt-repository ppa:ansible/ansible
+ sudo apt-get update
+ sudo apt-get install ansible
+
+ ```
+
+- Install Jinja2 and Python-Netaddr
+
+ ```sudo apt-get install python-jinja2=2.8-1~bpo8+1 python-netaddr```
+
+Now you can continue with [Preparing your deployment](getting-started.md#starting-custom-deployment)
--- /dev/null
+# K8s DNS stack by Kubespray
+
+For K8s cluster nodes, Kubespray configures a [Kubernetes DNS](https://kubernetes.io/docs/admin/dns/)
+[cluster add-on](https://releases.k8s.io/master/cluster/addons/README.md)
+to serve as an authoritative DNS server for a given ``dns_domain`` and its
+``svc, default.svc`` default subdomains (a total of ``ndots: 5`` max levels).
+
+Other nodes in the inventory, like external storage nodes or a separate etcd cluster
+node group, considered non-cluster and left up to the user to configure DNS resolve.
+
+## DNS variables
+
+There are several global variables which can be used to modify DNS settings:
+
+### ndots
+
+ndots value to be used in ``/etc/resolv.conf``
+
+It is important to note that multiple search domains combined with high ``ndots``
+values lead to poor performance of DNS stack, so please choose it wisely.
+
+### searchdomains
+
+Custom search domains to be added in addition to the cluster search domains (``default.svc.{{ dns_domain }}, svc.{{ dns_domain }}``).
+
+Most Linux systems limit the total number of search domains to 6 and the total length of all search domains
+to 256 characters. Depending on the length of ``dns_domain``, you're limited to less than the total limit.
+
+Please note that ``resolvconf_mode: docker_dns`` will automatically add your systems search domains as
+additional search domains. Please take this into the accounts for the limits.
+
+### nameservers
+
+This variable is only used by ``resolvconf_mode: host_resolvconf``. These nameservers are added to the hosts
+``/etc/resolv.conf`` *after* ``upstream_dns_servers`` and thus serve as backup nameservers. If this variable
+is not set, a default resolver is chosen (depending on cloud provider or 8.8.8.8 when no cloud provider is specified).
+
+### upstream_dns_servers
+
+DNS servers to be added *after* the cluster DNS. Used by all ``resolvconf_mode`` modes. These serve as backup
+DNS servers in early cluster deployment when no cluster DNS is available yet.
+
+### coredns_external_zones
+
+Array of optional external zones to coredns forward queries to. It's injected into
+`coredns`' config file before default kubernetes zone. Use it as an optimization for well-known zones and/or internal-only
+domains, i.e. VPN for internal networks (default is unset)
+
+Example:
+
+```yaml
+coredns_external_zones:
+- zones:
+ - example.com
+ - example.io:1053
+ nameservers:
+ - 1.1.1.1
+ - 2.2.2.2
+ cache: 5
+- zones:
+ - https://mycompany.local:4453
+ nameservers:
+ - 192.168.0.53
+ cache: 0
+```
+
+or as INI
+
+```ini
+coredns_external_zones='[{"cache": 30,"zones":["example.com","example.io:453"],"nameservers":["1.1.1.1","2.2.2.2"]}]'
+```
+
+### dns_etchosts (coredns)
+
+Optional hosts file content to coredns use as /etc/hosts file. This will also be used by nodelocaldns, if enabled.
+
+Example:
+
+```yaml
+dns_etchosts: |
+ 192.168.0.100 api.example.com
+ 192.168.0.200 ingress.example.com
+```
+
+### enable_coredns_reverse_dns_lookups
+
+Whether reverse DNS lookups are enabled in the coredns config. Defaults to `true`.
+
+### CoreDNS default zone cache plugin
+
+If you wish to configure the caching behaviour of CoreDNS on the default zone, you can do so using the `coredns_default_zone_cache_block` string block.
+
+An example value (more information on the [plugin's documentation](https://coredns.io/plugins/cache/)) to:
+
+* raise the max cache TTL to 3600 seconds
+* raise the max amount of success responses to cache to 3000
+* disable caching of denial responses altogether
+* enable pre-fetching of lookups with at least 10 lookups per minute before they expire
+
+Would be as follows:
+
+```yaml
+coredns_default_zone_cache_block: |
+ cache 3600 {
+ success 3000
+ denial 0
+ prefetch 10 1m
+ }
+```
+
+## DNS modes supported by Kubespray
+
+You can modify how Kubespray sets up DNS for your cluster with the variables ``dns_mode`` and ``resolvconf_mode``.
+
+### dns_mode
+
+``dns_mode`` configures how Kubespray will setup cluster DNS. There are four modes available:
+
+#### dns_mode: coredns (default)
+
+This installs CoreDNS as the default cluster DNS for all queries.
+
+#### dns_mode: coredns_dual
+
+This installs CoreDNS as the default cluster DNS for all queries, plus a secondary CoreDNS stack.
+
+#### dns_mode: manual
+
+This does not install coredns, but allows you to specify
+`manual_dns_server`, which will be configured on nodes for handling Pod DNS.
+Use this method if you plan to install your own DNS server in the cluster after
+initial deployment.
+
+#### dns_mode: none
+
+This does not install any of DNS solution at all. This basically disables cluster DNS completely and
+leaves you with a non functional cluster.
+
+## resolvconf_mode
+
+``resolvconf_mode`` configures how Kubespray will setup DNS for ``hostNetwork: true`` PODs and non-k8s containers.
+There are three modes available:
+
+### resolvconf_mode: host_resolvconf (default)
+
+This activates the classic Kubespray behavior that modifies the hosts ``/etc/resolv.conf`` file and dhclient
+configuration to point to the cluster dns server (either coredns or coredns_dual, depending on dns_mode).
+
+As cluster DNS is not available on early deployment stage, this mode is split into 2 stages. In the first
+stage (``dns_early: true``), ``/etc/resolv.conf`` is configured to use the DNS servers found in ``upstream_dns_servers``
+and ``nameservers``. Later, ``/etc/resolv.conf`` is reconfigured to use the cluster DNS server first, leaving
+the other nameservers as backups.
+
+Also note, existing records will be purged from the `/etc/resolv.conf`,
+including resolvconf's base/head/cloud-init config files and those that come from dhclient.
+
+### resolvconf_mode: docker_dns
+
+This sets up the docker daemon with additional --dns/--dns-search/--dns-opt flags.
+
+The following nameservers are added to the docker daemon (in the same order as listed here):
+
+* cluster nameserver (depends on dns_mode)
+* content of optional upstream_dns_servers variable
+* host system nameservers (read from hosts /etc/resolv.conf)
+
+The following search domains are added to the docker daemon (in the same order as listed here):
+
+* cluster domains (``default.svc.{{ dns_domain }}``, ``svc.{{ dns_domain }}``)
+* content of optional searchdomains variable
+* host system search domains (read from hosts /etc/resolv.conf)
+
+The following dns options are added to the docker daemon
+
+* ndots:{{ ndots }}
+* timeout:2
+* attempts:2
+
+These dns options can be overridden by setting a different list:
+
+```yaml
+docker_dns_options:
+- ndots:{{ ndots }}
+- timeout:2
+- attempts:2
+- rotate
+```
+
+For normal PODs, k8s will ignore these options and setup its own DNS settings for the PODs, taking
+the --cluster_dns (either coredns or coredns_dual, depending on dns_mode) kubelet option into account.
+For ``hostNetwork: true`` PODs however, k8s will let docker setup DNS settings. Docker containers which
+are not started/managed by k8s will also use these docker options.
+
+The host system name servers are added to ensure name resolution is also working while cluster DNS is not
+running yet. This is especially important in early stages of cluster deployment. In this early stage,
+DNS queries to the cluster DNS will timeout after a few seconds, resulting in the system nameserver being
+used as a backup nameserver. After cluster DNS is running, all queries will be answered by the cluster DNS
+servers, which in turn will forward queries to the system nameserver if required.
+
+### resolvconf_mode: none
+
+Does nothing regarding ``/etc/resolv.conf``. This leaves you with a cluster that works as expected in most cases.
+The only exception is that ``hostNetwork: true`` PODs and non-k8s managed containers will not be able to resolve
+cluster service names.
+
+## Nodelocal DNS cache
+
+Setting ``enable_nodelocaldns`` to ``true`` will make pods reach out to the dns (core-dns) caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking. The local caching agent will query core-dns (depending on what main DNS plugin is configured in your cluster) for cache misses of cluster hostnames(cluster.local suffix by default).
+
+More information on the rationale behind this implementation can be found [here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0030-nodelocal-dns-cache.md).
+
+**As per the 2.10 release, Nodelocal DNS cache is enabled by default.**
+
+### External zones
+
+It's possible to extent the `nodelocaldns`' configuration by adding an array of external zones. For example:
+
+```yaml
+nodelocaldns_external_zones:
+- zones:
+ - example.com
+ - example.io:1053
+ nameservers:
+ - 1.1.1.1
+ - 2.2.2.2
+ cache: 5
+- zones:
+ - https://mycompany.local:4453
+ nameservers:
+ - 192.168.0.53
+```
+
+### dns_etchosts (nodelocaldns)
+
+See [dns_etchosts](#dns_etchosts-coredns) above.
+
+### Nodelocal DNS HA
+
+Under some circumstances the single POD nodelocaldns implementation may not be able to be replaced soon enough and a cluster upgrade or a nodelocaldns upgrade can cause DNS requests to time out for short intervals. If for any reason your applications cannot tollerate this behavior you can enable a redundant nodelocal DNS pod on each node:
+
+```yaml
+enable_nodelocaldns_secondary: true
+```
+
+**Note:** when the nodelocaldns secondary is enabled, the primary is instructed to no longer tear down the iptables rules it sets up to direct traffic to itself. In case both daemonsets have failing pods on the same node, this can cause a DNS blackout with traffic no longer being forwarded to the coredns central service as a fallback. Please ensure you account for this also if you decide to disable the nodelocaldns cache.
+
+There is a time delta (in seconds) allowed for the secondary nodelocaldns to survive in case both primary and secondary daemonsets are updated at the same time. It is advised to tune this variable after you have performed some tests in your own environment.
+
+```yaml
+nodelocaldns_secondary_skew_seconds: 5
+```
+
+## Limitations
+
+* Kubespray has yet ways to configure Kubedns addon to forward requests SkyDns can
+ not answer with authority to arbitrary recursive resolvers. This task is left
+ for future. See [official SkyDns docs](https://github.com/skynetservices/skydns)
+ for details.
+
+* There is
+ [no way to specify a custom value](https://github.com/kubernetes/kubernetes/issues/33554)
+ for the SkyDNS ``ndots`` param.
+
+* the ``searchdomains`` have a limitation of a 6 names and 256 chars
+ length. Due to default ``svc, default.svc`` subdomains, the actual
+ limits are a 4 names and 239 chars respectively.
+
+* the ``nameservers`` have a limitation of a 3 servers, although there
+ is a way to mitigate that with the ``upstream_dns_servers``,
+ see below. Anyway, the ``nameservers`` can take no more than a two
+ custom DNS servers because of one slot is reserved for a Kubernetes
+ cluster needs.
--- /dev/null
+# Docker support
+
+The docker runtime is supported by kubespray and while the `dockershim` is deprecated to be removed in kubernetes 1.24+ there are alternative ways to use docker such as through the [cri-dockerd](https://github.com/Mirantis/cri-dockerd) project supported by Mirantis.
+
+Using the docker container manager:
+
+```yaml
+container_manager: docker
+```
+
+*Note:* `cri-dockerd` has replaced `dockershim` across supported kubernetes version in kubespray 2.20.
+
+Enabling the `overlay2` graph driver:
+
+```yaml
+docker_storage_options: -s overlay2
+```
+
+Enabling `docker_container_storage_setup`, it will configure devicemapper driver on Centos7 or RedHat7.
+Deployers must be define a disk path for `docker_container_storage_setup_devs`, otherwise docker-storage-setup will be executed incorrectly.
+
+```yaml
+docker_container_storage_setup: true
+docker_container_storage_setup_devs: /dev/vdb
+```
+
+Changing the Docker cgroup driver (native.cgroupdriver); valid options are `systemd` or `cgroupfs`, default is `systemd`:
+
+```yaml
+docker_cgroup_driver: systemd
+```
+
+If you have more than 3 nameservers kubespray will only use the first 3 else it will fail. Set the `docker_dns_servers_strict` to `false` to prevent deployment failure.
+
+```yaml
+docker_dns_servers_strict: false
+```
+
+Set the path used to store Docker data:
+
+```yaml
+docker_daemon_graph: "/var/lib/docker"
+```
+
+Changing the docker daemon iptables support:
+
+```yaml
+docker_iptables_enabled: "false"
+```
+
+Docker log options:
+
+```yaml
+# Rotate container stderr/stdout logs at 50m and keep last 5
+docker_log_opts: "--log-opt max-size=50m --log-opt max-file=5"
+```
+
+Changre the docker `bin_dir`, this should not be changed unless you use a custom docker package:
+
+```yaml
+docker_bin_dir: "/usr/bin"
+```
+
+To keep docker packages after installation; speeds up repeated ansible provisioning runs when '1'.
+kubespray deletes the docker package on each run, so caching the package makes sense:
+
+```yaml
+docker_rpm_keepcache: 1
+```
+
+Allowing insecure-registry access to self hosted registries. Can be ipaddress and domain_name.
+
+```yaml
+## example define 172.19.16.11 or mirror.registry.io
+docker_insecure_registries:
+ - mirror.registry.io
+ - 172.19.16.11
+```
+
+Adding other registry, i.e. China registry mirror:
+
+```yaml
+docker_registry_mirrors:
+ - https://registry.docker-cn.com
+ - https://mirror.aliyuncs.com
+```
+
+Overriding default system MountFlags value. This option takes a mount propagation flag: `shared`, `slave` or `private`, which control whether mounts in the file system namespace set up for docker will receive or propagate mounts and unmounts. Leave empty for system default:
+
+```yaml
+docker_mount_flags:
+```
+
+Adding extra options to pass to the docker daemon:
+
+```yaml
+## This string should be exactly as you wish it to appear.
+docker_options: ""
+```
--- /dev/null
+# Downloading binaries and containers
+
+Kubespray supports several download/upload modes. The default is:
+
+* Each node downloads binaries and container images on its own, which is ``download_run_once: False``.
+* For K8s apps, pull policy is ``k8s_image_pull_policy: IfNotPresent``.
+* For system managed containers, like kubelet or etcd, pull policy is ``download_always_pull: False``, which is pull if only the wanted repo and tag/sha256 digest differs from that the host has.
+
+There is also a "pull once, push many" mode as well:
+
+* Setting ``download_run_once: True`` will make kubespray download container images and binaries only once and then push them to the cluster nodes. The default download delegate node is the first `kube_control_plane`.
+* Set ``download_localhost: True`` to make localhost the download delegate. This can be useful if cluster nodes cannot access external addresses. To use this requires that the container runtime is installed and running on the Ansible master and that the current user is either in the docker group or can do passwordless sudo, to be able to use the container runtime. Note: even if `download_localhost` is false, files will still be copied to the Ansible server (local host) from the delegated download node, and then distributed from the Ansible server to all cluster nodes.
+
+NOTE: When `download_run_once` is true and `download_localhost` is false, all downloads will be done on the delegate node, including downloads for container images that are not required on that node. As a consequence, the storage required on that node will probably be more than if download_run_once was false, because all images will be loaded into the storage of the container runtime on that node, instead of just the images required for that node.
+
+On caching:
+
+* When `download_run_once` is `True`, all downloaded files will be cached locally in `download_cache_dir`, which defaults to `/tmp/kubespray_cache`. On subsequent provisioning runs, this local cache will be used to provision the nodes, minimizing bandwidth usage and improving provisioning time. Expect about 800MB of disk space to be used on the ansible node for the cache. Disk space required for the image cache on the kubernetes nodes is a much as is needed for the largest image, which is currently slightly less than 150MB.
+* By default, if `download_run_once` is false, kubespray will not retrieve the downloaded images and files from the download delegate node to the local cache, or use that cache to pre-provision those nodes. If you have a full cache with container images and files and you don’t need to download anything, but want to use a cache - set `download_force_cache` to `True`.
+* By default, cached images that are used to pre-provision the remote nodes will be deleted from the remote nodes after use, to save disk space. Setting `download_keep_remote_cache` will prevent the files from being deleted. This can be useful while developing kubespray, as it can decrease provisioning times. As a consequence, the required storage for images on the remote nodes will increase from 150MB to about 550MB, which is currently the combined size of all required container images.
+
+Container images and binary files are described by the vars like ``foo_version``,
+``foo_download_url``, ``foo_checksum`` for binaries and ``foo_image_repo``,
+``foo_image_tag`` or optional ``foo_digest_checksum`` for containers.
+
+Container images may be defined by its repo and tag, for example:
+`andyshinn/dnsmasq:2.72`. Or by repo and tag and sha256 digest:
+`andyshinn/dnsmasq@sha256:7c883354f6ea9876d176fe1d30132515478b2859d6fc0cbf9223ffdc09168193`.
+
+Note, the SHA256 digest and the image tag must be both specified and correspond
+to each other. The given example above is represented by the following vars:
+
+```yaml
+dnsmasq_digest_checksum: 7c883354f6ea9876d176fe1d30132515478b2859d6fc0cbf9223ffdc09168193
+dnsmasq_image_repo: andyshinn/dnsmasq
+dnsmasq_image_tag: '2.72'
+```
+
+The full list of available vars may be found in the download's ansible role defaults. Those also allow to specify custom urls and local repositories for binaries and container
+images as well. See also the DNS stack docs for the related intranet configuration,
+so the hosts can resolve those urls and repos.
--- /dev/null
+# Encrypting Secret Data at Rest
+
+Before enabling Encrypting Secret Data at Rest, please read the following documentation carefully.
+
+<https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/>
+
+As you can see from the documentation above, 5 encryption providers are supported as of today (22.02.2022).
+
+As default value for the provider we have chosen `secretbox`.
+
+Alternatively you can use the values `identity`, `aesgcm`, `aescbc` or `kms`.
+
+| Provider | Why we have decided against the value as default |
+|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| identity | no encryption |
+| aesgcm | Must be rotated every 200k writes |
+| aescbc | Not recommended due to CBC's vulnerability to padding oracle attacks. |
+| kms | Is the official recommended way, but assumes that a key management service independent of Kubernetes exists, we cannot assume this in all environments, so not a suitable default value. |
+
+## Details about Secretbox
+
+Secretbox uses [Poly1305](https://cr.yp.to/mac.html) as message-authentication code and [XSalsa20](https://www.xsalsa20.com/) as secret-key authenticated encryption and secret-key encryption.
--- /dev/null
+# Equinix Metal
+
+Kubespray provides support for bare metal deployments using the [Equinix Metal](http://metal.equinix.com).
+Deploying upon bare metal allows Kubernetes to run at locations where an existing public or private cloud might not exist such
+as cell tower, edge collocated installations. The deployment mechanism used by Kubespray for Equinix Metal is similar to that used for
+AWS and OpenStack clouds (notably using Terraform to deploy the infrastructure). Terraform uses the Equinix Metal provider plugin
+to provision and configure hosts which are then used by the Kubespray Ansible playbooks. The Ansible inventory is generated
+dynamically from the Terraform state file.
+
+## Local Host Configuration
+
+To perform this installation, you will need a localhost to run Terraform/Ansible (laptop, VM, etc) and an account with Equinix Metal.
+In this example, we're using an m1.large CentOS 7 OpenStack VM as the localhost to kickoff the Kubernetes installation.
+You'll need Ansible, Git, and PIP.
+
+```bash
+sudo yum install epel-release
+sudo yum install ansible
+sudo yum install git
+sudo yum install python-pip
+```
+
+## Playbook SSH Key
+
+An SSH key is needed by Kubespray/Ansible to run the playbooks.
+This key is installed into the bare metal hosts during the Terraform deployment.
+You can generate a key new key or use an existing one.
+
+```bash
+ssh-keygen -f ~/.ssh/id_rsa
+```
+
+## Install Terraform
+
+Terraform is required to deploy the bare metal infrastructure. The steps below are for installing on CentOS 7.
+[More terraform installation options are available.](https://learn.hashicorp.com/terraform/getting-started/install.html)
+
+Grab the latest version of Terraform and install it.
+
+```bash
+echo "https://releases.hashicorp.com/terraform/$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')/terraform_$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')_linux_amd64.zip"
+sudo yum install unzip
+sudo unzip terraform_0.14.10_linux_amd64.zip -d /usr/local/bin/
+```
+
+## Download Kubespray
+
+Pull over Kubespray and setup any required libraries.
+
+```bash
+git clone https://github.com/kubernetes-sigs/kubespray
+cd kubespray
+```
+
+## Install Ansible
+
+Install Ansible according to [Ansible installation guide](/docs/ansible.md#installing-ansible)
+
+## Cluster Definition
+
+In this example, a new cluster called "alpha" will be created.
+
+```bash
+cp -LRp contrib/terraform/packet/sample-inventory inventory/alpha
+cd inventory/alpha/
+ln -s ../../contrib/terraform/packet/hosts
+```
+
+Details about the cluster, such as the name, as well as the authentication tokens and project ID
+for Equinix Metal need to be defined. To find these values see [Equinix Metal API Accounts](https://metal.equinix.com/developers/docs/accounts/).
+
+```bash
+vi cluster.tfvars
+```
+
+* cluster_name = alpha
+* packet_project_id = ABCDEFGHIJKLMNOPQRSTUVWXYZ123456
+* public_key_path = 12345678-90AB-CDEF-GHIJ-KLMNOPQRSTUV
+
+## Deploy Bare Metal Hosts
+
+Initializing Terraform will pull down any necessary plugins/providers.
+
+```bash
+terraform init ../../contrib/terraform/packet/
+```
+
+Run Terraform to deploy the hardware.
+
+```bash
+terraform apply -var-file=cluster.tfvars ../../contrib/terraform/packet
+```
+
+## Run Kubespray Playbooks
+
+With the bare metal infrastructure deployed, Kubespray can now install Kubernetes and setup the cluster.
+
+```bash
+ansible-playbook --become -i inventory/alpha/hosts cluster.yml
+```
--- /dev/null
+# etcd
+
+## Deployment Types
+
+It is possible to deploy etcd with three methods. To change the default deployment method (host), use the `etcd_deployment_type` variable. Possible values are `host`, `kubeadm`, and `docker`.
+
+### Host
+
+Host deployment is the default method. Using this method will result in etcd installed as a systemd service.
+
+### Docker
+
+Installs docker in etcd group members and runs etcd on docker containers. Only usable when `container_manager` is set to `docker`.
+
+### Kubeadm
+
+This deployment method is experimental and is only available for new deployments. This deploys etcd as a static pod in master hosts.
+
+## Metrics
+
+To expose metrics on a separate HTTP port, define it in the inventory with:
+
+```yaml
+etcd_metrics_port: 2381
+```
+
+To create a service `etcd-metrics` and associated endpoints in the `kube-system` namespace,
+define it's labels in the inventory with:
+
+```yaml
+etcd_metrics_service_labels:
+ k8s-app: etcd
+ app.kubernetes.io/managed-by: Kubespray
+ app: kube-prometheus-stack-kube-etcd
+ release: prometheus-stack
+```
+
+The last two labels in the above example allows to scrape the metrics from the
+[kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
+chart with the following Helm `values.yaml` :
+
+```yaml
+kubeEtcd:
+ service:
+ enabled: false
+```
--- /dev/null
+# Fedora CoreOS
+
+Tested with stable version 34.20210611.3.0
+
+Because package installation with `rpm-ostree` requires a reboot, playbook may fail while bootstrap.
+Restart playbook again.
+
+## Containers
+
+Tested with
+
+- containerd
+- crio
+
+## Network
+
+### calico
+
+To use calico create sysctl file with ignition:
+
+```yaml
+files:
+ - path: /etc/sysctl.d/reverse-path-filter.conf
+ contents:
+ inline: |
+ net.ipv4.conf.all.rp_filter=1
+```
+
+## libvirt setup
+
+### Prepare
+
+Prepare ignition and serve via http (a.e. python -m http.server )
+
+```json
+{
+ "ignition": {
+ "version": "3.0.0"
+ },
+
+ "passwd": {
+ "users": [
+ {
+ "name": "ansibleUser",
+ "sshAuthorizedKeys": [
+ "ssh-rsa ..publickey.."
+ ],
+ "groups": [ "wheel" ]
+ }
+ ]
+ }
+}
+```
+
+### create guest
+
+```shell script
+machine_name=myfcos1
+ignition_url=http://mywebserver/fcos.ign
+
+fcos_version=34.20210611.3.0
+kernel=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-kernel-x86_64
+initrd=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-initramfs.x86_64.img
+rootfs=https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/${fcos_version}/x86_64/fedora-coreos-${fcos_version}-live-rootfs.x86_64.img
+kernel_args="console=ttyS0 coreos.live.rootfs_url=${rootfs} coreos.inst.install_dev=/dev/sda coreos.inst.stream=stable coreos.inst.ignition_url=${ignition_url}"
+sudo virt-install --name ${machine_name} --ram 4048 --graphics=none --vcpus 2 --disk size=20 \
+ --network bridge=virbr0 \
+ --install kernel=${kernel},initrd=${initrd},kernel_args_overwrite=yes,kernel_args="${kernel_args}"
+```
--- /dev/null
+# Flannel
+
+Flannel is a network fabric for containers, designed for Kubernetes
+
+**Warning:** You may encounter this [bug](https://github.com/coreos/flannel/pull/1282) with `VXLAN` backend, while waiting on a newer Flannel version the current workaround (`ethtool --offload flannel.1 rx off tx off`) is showcase in kubespray [networking test](tests/testcases/040_check-network-adv.yml:31).
+
+## Verifying flannel install
+
+* Flannel configuration file should have been created there
+
+```ShellSession
+cat /run/flannel/subnet.env
+FLANNEL_NETWORK=10.233.0.0/18
+FLANNEL_SUBNET=10.233.16.1/24
+FLANNEL_MTU=1450
+FLANNEL_IPMASQ=false
+```
+
+* Check if the network interface has been created
+
+```ShellSession
+ip a show dev flannel.1
+4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
+ link/ether e2:f3:a7:0f:bf:cb brd ff:ff:ff:ff:ff:ff
+ inet 10.233.16.0/18 scope global flannel.1
+ valid_lft forever preferred_lft forever
+ inet6 fe80::e0f3:a7ff:fe0f:bfcb/64 scope link
+ valid_lft forever preferred_lft forever
+```
+
+* Try to run a container and check its ip address
+
+```ShellSession
+kubectl run test --image=busybox --command -- tail -f /dev/null
+replicationcontroller "test" created
+
+kubectl describe po test-34ozs | grep ^IP
+IP: 10.233.16.2
+```
+
+```ShellSession
+kubectl exec test-34ozs -- ip a show dev eth0
+8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
+ link/ether 02:42:0a:e9:2b:03 brd ff:ff:ff:ff:ff:ff
+ inet 10.233.16.2/24 scope global eth0
+ valid_lft forever preferred_lft forever
+ inet6 fe80::42:aff:fee9:2b03/64 scope link tentative flags 08
+ valid_lft forever preferred_lft forever
+```
--- /dev/null
+Flatcar Container Linux bootstrap
+===============
+
+Example with Ansible:
+
+Before running the cluster playbook you must satisfy the following requirements:
+
+General Flatcar Pre-Installation Notes:
+
+- Ensure that the bin_dir is set to `/opt/bin`
+- ansible_python_interpreter should be `/opt/bin/python`. This will be laid down by the bootstrap task.
+- The resolvconf_mode setting of `docker_dns` **does not** work for Flatcar. This is because we do not edit the systemd service file for docker on Flatcar nodes. Instead, just use the default `host_resolvconf` mode. It should work out of the box.
+
+Then you can proceed to [cluster deployment](#run-deployment)
--- /dev/null
+# GCP Load Balancers for type=LoadBalacer of Kubernetes Services
+
+Google Cloud Platform can be used for creation of Kubernetes Service Load Balancer.
+
+This feature is able to deliver by adding parameters to kube-controller-manager and kubelet. You need specify:
+
+ --cloud-provider=gce
+ --cloud-config=/etc/kubernetes/cloud-config
+
+To get working it in kubespray, you need to add tag to GCE instances and specify it in kubespray group vars and also set cloud_provider to gce. So for example, in file group_vars/all/gcp.yml:
+
+ cloud_provider: gce
+ gce_node_tags: k8s-lb
+
+When you will setup it and create SVC in Kubernetes with type=LoadBalancer, cloud provider will create public IP and will set firewall.
+Note: Cloud provider run under VM service account, so this account needs to have correct permissions to be able to create all GCP resources.
--- /dev/null
+# GCP Persistent Disk CSI Driver
+
+The GCP Persistent Disk CSI driver allows you to provision volumes for pods with a Kubernetes deployment over Google Cloud Platform. The CSI driver replaces to volume provioning done by the in-tree azure cloud provider which is deprecated.
+
+To deploy GCP Persistent Disk CSI driver, uncomment the `gcp_pd_csi_enabled` option in `group_vars/all/gcp.yml` and set it to `true`.
+
+## GCP Persistent Disk Storage Class
+
+If you want to deploy the GCP Persistent Disk storage class to provision volumes dynamically, you should set `persistent_volumes_enabled` in `group_vars/k8s_cluster/k8s_cluster.yml` to `true`.
+
+## GCP credentials
+
+In order for the CSI driver to provision disks, you need to create for it a service account on GCP with the appropriate permissions.
+
+Follow these steps to configure it:
+
+```ShellSession
+# This will open a web page for you to authenticate
+gcloud auth login
+export PROJECT=nameofmyproject
+gcloud config set project $PROJECT
+
+git clone https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver $GOPATH/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver
+
+export GCE_PD_SA_NAME=my-gce-pd-csi-sa
+export GCE_PD_SA_DIR=/my/safe/credentials/directory
+
+./deploy/setup-project.sh
+```
+
+The above will create a file named `cloud-sa.json` in the specified `GCE_PD_SA_DIR`. This file contains the service account with the appropriate credentials for the CSI driver to perform actions on GCP to request disks for pods.
+
+You need to provide this file's path through the variable `gcp_pd_csi_sa_cred_file` in `inventory/mycluster/group_vars/all/gcp.yml`
+
+You can now deploy Kubernetes with Kubespray over GCP.
+
+## GCP PD CSI Driver test
+
+To test the dynamic provisioning using GCP PD CSI driver, make sure to have the storage class deployed (through persistent volumes), and apply the following manifest:
+
+```yml
+---
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+ name: podpvc
+spec:
+ accessModes:
+ - ReadWriteOnce
+ storageClassName: csi-gce-pd
+ resources:
+ requests:
+ storage: 1Gi
+
+---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: web-server
+spec:
+ containers:
+ - name: web-server
+ image: nginx
+ volumeMounts:
+ - mountPath: /var/lib/www/html
+ name: mypvc
+ volumes:
+ - name: mypvc
+ persistentVolumeClaim:
+ claimName: podpvc
+ readOnly: false
+```
+
+## GCP PD documentation
+
+You can find the official GCP Persistent Disk CSI driver installation documentation here: [GCP PD CSI Driver](https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/master/docs/kubernetes/user-guides/driver-install.md
+)
--- /dev/null
+# Getting started
+
+## Building your own inventory
+
+Ansible inventory can be stored in 3 formats: YAML, JSON, or INI-like. There is
+an example inventory located
+[here](https://github.com/kubernetes-sigs/kubespray/blob/master/inventory/sample/inventory.ini).
+
+You can use an
+[inventory generator](https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/inventory_builder/inventory.py)
+to create or modify an Ansible inventory. Currently, it is limited in
+functionality and is only used for configuring a basic Kubespray cluster inventory, but it does
+support creating inventory file for large clusters as well. It now supports
+separated ETCD and Kubernetes control plane roles from node role if the size exceeds a
+certain threshold. Run `python3 contrib/inventory_builder/inventory.py help` for more information.
+
+Example inventory generator usage:
+
+```ShellSession
+cp -r inventory/sample inventory/mycluster
+declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
+CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+```
+
+Then use `inventory/mycluster/hosts.yml` as inventory file.
+
+## Starting custom deployment
+
+Once you have an inventory, you may want to customize deployment data vars
+and start the deployment:
+
+**IMPORTANT**: Edit my\_inventory/groups\_vars/\*.yaml to override data vars:
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.yml cluster.yml -b -v \
+ --private-key=~/.ssh/private_key
+```
+
+See more details in the [ansible guide](/docs/ansible.md).
+
+### Adding nodes
+
+You may want to add worker, control plane or etcd nodes to your existing cluster. This can be done by re-running the `cluster.yml` playbook, or you can target the bare minimum needed to get kubelet installed on the worker and talking to your control planes. This is especially helpful when doing something like autoscaling your clusters.
+
+- Add the new worker node to your inventory in the appropriate group (or utilize a [dynamic inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html)).
+- Run the ansible-playbook command, substituting `cluster.yml` for `scale.yml`:
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.yml scale.yml -b -v \
+ --private-key=~/.ssh/private_key
+```
+
+### Remove nodes
+
+You may want to remove **control plane**, **worker**, or **etcd** nodes from your
+existing cluster. This can be done by re-running the `remove-node.yml`
+playbook. First, all specified nodes will be drained, then stop some
+kubernetes services and delete some certificates,
+and finally execute the kubectl command to delete these nodes.
+This can be combined with the add node function. This is generally helpful
+when doing something like autoscaling your clusters. Of course, if a node
+is not working, you can remove the node and install it again.
+
+Use `--extra-vars "node=<nodename>,<nodename2>"` to select the node(s) you want to delete.
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.yml remove-node.yml -b -v \
+--private-key=~/.ssh/private_key \
+--extra-vars "node=nodename,nodename2"
+```
+
+If a node is completely unreachable by ssh, add `--extra-vars reset_nodes=false`
+to skip the node reset step. If one node is unavailable, but others you wish
+to remove are able to connect via SSH, you could set `reset_nodes=false` as a host
+var in inventory.
+
+## Connecting to Kubernetes
+
+By default, Kubespray configures kube_control_plane hosts with insecure access to
+kube-apiserver via port 8080. A kubeconfig file is not necessary in this case,
+because kubectl will use <http://localhost:8080> to connect. The kubeconfig files
+generated will point to localhost (on kube_control_planes) and kube_node hosts will
+connect either to a localhost nginx proxy or to a loadbalancer if configured.
+More details on this process are in the [HA guide](/docs/ha-mode.md).
+
+Kubespray permits connecting to the cluster remotely on any IP of any
+kube_control_plane host on port 6443 by default. However, this requires
+authentication. One can get a kubeconfig from kube_control_plane hosts
+(see [below](#accessing-kubernetes-api)).
+
+For more information on kubeconfig and accessing a Kubernetes cluster, refer to
+the Kubernetes [documentation](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/).
+
+## Accessing Kubernetes Dashboard
+
+Supported version is kubernetes-dashboard v2.0.x :
+
+- Login option : token/kubeconfig by default
+- Deployed by default in "kube-system" namespace, can be overridden with `dashboard_namespace: kubernetes-dashboard` in inventory,
+- Only serves over https
+
+Access is described in [dashboard docs](https://github.com/kubernetes/dashboard/tree/master/docs/user/accessing-dashboard). With kubespray's default deployment in kube-system namespace, instead of kubernetes-dashboard :
+
+- Proxy URL is <http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login>
+- kubectl commands must be run with "-n kube-system"
+
+Accessing through Ingress is highly recommended. For proxy access, please note that proxy must listen to [localhost](https://github.com/kubernetes/dashboard/issues/692#issuecomment-220492484) (`proxy --address="x.x.x.x"` will not work)
+
+For token authentication, guide to create Service Account is provided in [dashboard sample user](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md) doc. Still take care of default namespace.
+
+Access can also by achieved via ssh tunnel on a control plane :
+
+```bash
+# localhost:8081 will be sent to control-plane-1's own localhost:8081
+ssh -L8001:localhost:8001 user@control-plane-1
+sudo -i
+kubectl proxy
+```
+
+## Accessing Kubernetes API
+
+The main client of Kubernetes is `kubectl`. It is installed on each kube_control_plane
+host and can optionally be configured on your ansible host by setting
+`kubectl_localhost: true` and `kubeconfig_localhost: true` in the configuration:
+
+- If `kubectl_localhost` enabled, `kubectl` will download onto `/usr/local/bin/` and setup with bash completion. A helper script `inventory/mycluster/artifacts/kubectl.sh` also created for setup with below `admin.conf`.
+- If `kubeconfig_localhost` enabled `admin.conf` will appear in the `inventory/mycluster/artifacts/` directory after deployment.
+- The location where these files are downloaded to can be configured via the `artifacts_dir` variable.
+
+NOTE: The controller host name in the admin.conf file might be a private IP. If so, change it to use the controller's public IP or the cluster's load balancer.
+
+You can see a list of nodes by running the following commands:
+
+```ShellSession
+cd inventory/mycluster/artifacts
+./kubectl.sh get nodes
+```
+
+If desired, copy admin.conf to ~/.kube/config.
+
+## Setting up your first cluster
+
+[Setting up your first cluster](/docs/setting-up-your-first-cluster.md) is an
+ applied step-by-step guide for setting up your first cluster with Kubespray.
--- /dev/null
+# gVisor
+
+[gVisor](https://gvisor.dev/docs/) is an application kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system.
+
+gVisor includes an Open Container Initiative (OCI) runtime called runsc that makes it easy to work with existing container tooling. The runsc runtime integrates with Docker and Kubernetes, making it simple to run sandboxed containers.
+
+## Usage
+
+To enable gVisor you should be using a container manager that is compatible with selecting the [RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) such as `containerd`.
+
+Containerd support:
+
+```yaml
+container_manager: containerd
+gvisor_enabled: true
+```
--- /dev/null
+# HA endpoints for K8s
+
+The following components require a highly available endpoints:
+
+* etcd cluster,
+* kube-apiserver service instances.
+
+The latter relies on a 3rd side reverse proxy, like Nginx or HAProxy, to
+achieve the same goal.
+
+## Etcd
+
+The etcd clients (kube-api-masters) are configured with the list of all etcd peers. If the etcd-cluster has multiple instances, it's configured in HA already.
+
+## Kube-apiserver
+
+K8s components require a loadbalancer to access the apiservers via a reverse
+proxy. Kubespray includes support for an nginx-based proxy that resides on each
+non-master Kubernetes node. This is referred to as localhost loadbalancing. It
+is less efficient than a dedicated load balancer because it creates extra
+health checks on the Kubernetes apiserver, but is more practical for scenarios
+where an external LB or virtual IP management is inconvenient. This option is
+configured by the variable `loadbalancer_apiserver_localhost` (defaults to
+`True`. Or `False`, if there is an external `loadbalancer_apiserver` defined).
+You may also define the port the local internal loadbalancer uses by changing,
+`loadbalancer_apiserver_port`. This defaults to the value of
+`kube_apiserver_port`. It is also important to note that Kubespray will only
+configure kubelet and kube-proxy on non-master nodes to use the local internal
+loadbalancer.
+
+If you choose to NOT use the local internal loadbalancer, you will need to
+use the [kube-vip](kube-vip.md) ansible role or configure your own loadbalancer to achieve HA. By default, it only configures a non-HA endpoint, which points to the
+`access_ip` or IP address of the first server node in the `kube_control_plane` group.
+It can also configure clients to use endpoints for a given loadbalancer type.
+The following diagram shows how traffic to the apiserver is directed.
+
+
+
+A user may opt to use an external loadbalancer (LB) instead. An external LB
+provides access for external clients, while the internal LB accepts client
+connections only to the localhost.
+Given a frontend `VIP` address and `IP1, IP2` addresses of backends, here is
+an example configuration for a HAProxy service acting as an external LB:
+
+```raw
+listen kubernetes-apiserver-https
+ bind <VIP>:8383
+ mode tcp
+ option log-health-checks
+ timeout client 3h
+ timeout server 3h
+ server master1 <IP1>:6443 check check-ssl verify none inter 10000
+ server master2 <IP2>:6443 check check-ssl verify none inter 10000
+ balance roundrobin
+```
+
+ Note: That's an example config managed elsewhere outside of Kubespray.
+
+And the corresponding example global vars for such a "cluster-aware"
+external LB with the cluster API access modes configured in Kubespray:
+
+```yml
+apiserver_loadbalancer_domain_name: "my-apiserver-lb.example.com"
+loadbalancer_apiserver:
+ address: <VIP>
+ port: 8383
+```
+
+ Note: The default kubernetes apiserver configuration binds to all interfaces,
+ so you will need to use a different port for the vip from that the API is
+ listening on, or set the `kube_apiserver_bind_address` so that the API only
+ listens on a specific interface (to avoid conflict with haproxy binding the
+ port on the VIP address)
+
+This domain name, or default "lb-apiserver.kubernetes.local", will be inserted
+into the `/etc/hosts` file of all servers in the `k8s_cluster` group and wired
+into the generated self-signed TLS/SSL certificates as well. Note that
+the HAProxy service should as well be HA and requires a VIP management, which
+is out of scope of this doc.
+
+There is a special case for an internal and an externally configured (not with
+Kubespray) LB used simultaneously. Keep in mind that the cluster is not aware
+of such an external LB and you need no to specify any configuration variables
+for it.
+
+ Note: TLS/SSL termination for externally accessed API endpoints' will **not**
+ be covered by Kubespray for that case. Make sure your external LB provides it.
+ Alternatively you may specify an externally load balanced VIPs in the
+ `supplementary_addresses_in_ssl_keys` list. Then, kubespray will add them into
+ the generated cluster certificates as well.
+
+Aside of that specific case, the `loadbalancer_apiserver` considered mutually
+exclusive to `loadbalancer_apiserver_localhost`.
+
+Access API endpoints are evaluated automatically, as the following:
+
+| Endpoint type | kube_control_plane | non-master | external |
+|------------------------------|------------------------------------------|-------------------------|-----------------------|
+| Local LB (default) | `https://dbip:sp` | `https://lc:nsp` | `https://m[0].aip:sp` |
+| Local LB (default) + cbip | `https://cbip:sp` and `https://lc:nsp` | `https://lc:nsp` | `https://m[0].aip:sp` |
+| Local LB + Unmanaged here LB | `https://dbip:sp` | `https://lc:nsp` | `https://ext` |
+| External LB, no internal | `https://dbip:sp` | `<https://lb:lp>` | `https://lb:lp` |
+| No ext/int LB | `https://dbip:sp` | `<https://m[0].aip:sp>` | `https://m[0].aip:sp` |
+
+Where:
+
+* `m[0]` - the first node in the `kube_control_plane` group;
+* `lb` - LB FQDN, `apiserver_loadbalancer_domain_name`;
+* `ext` - Externally load balanced VIP:port and FQDN, not managed by Kubespray;
+* `lc` - localhost;
+* `cbip` - a custom bind IP, `kube_apiserver_bind_address`;
+* `dbip` - localhost for the default bind IP '0.0.0.0';
+* `nsp` - nginx secure port, `loadbalancer_apiserver_port`, defers to `sp`;
+* `sp` - secure port, `kube_apiserver_port`;
+* `lp` - LB port, `loadbalancer_apiserver.port`, defers to the secure port;
+* `ip` - the node IP, defers to the ansible IP;
+* `aip` - `access_ip`, defers to the ip.
+
+A second and a third column represent internal cluster access modes. The last
+column illustrates an example URI to access the cluster APIs externally.
+Kubespray has nothing to do with it, this is informational only.
+
+As you can see, the masters' internal API endpoints are always
+contacted via the local bind IP, which is `https://bip:sp`.
+
+## Optional configurations
+
+### ETCD with a LB
+
+In order to use an external loadbalancing (L4/TCP or L7 w/ SSL Passthrough VIP), the following variables need to be overridden in group_vars
+
+* `etcd_access_addresses`
+* `etcd_client_url`
+* `etcd_cert_alt_names`
+* `etcd_cert_alt_ips`
+
+#### Example of a VIP w/ FQDN
+
+```yaml
+etcd_access_addresses: https://etcd.example.com:2379
+etcd_client_url: https://etcd.example.com:2379
+etcd_cert_alt_names:
+ - "etcd.kube-system.svc.{{ dns_domain }}"
+ - "etcd.kube-system.svc"
+ - "etcd.kube-system"
+ - "etcd"
+ - "etcd.example.com" # This one needs to be added to the default etcd_cert_alt_names
+```
+
+#### Example of a VIP w/o FQDN (IP only)
+
+```yaml
+etcd_access_addresses: https://2.3.7.9:2379
+etcd_client_url: https://2.3.7.9:2379
+etcd_cert_alt_ips:
+ - "2.3.7.9"
+```
--- /dev/null
+# Cluster Hardening
+
+If you want to improve the security on your cluster and make it compliant with the [CIS Benchmarks](https://learn.cisecurity.org/benchmarks), here you can find a configuration to harden your **kubernetes** installation.
+
+To apply the hardening configuration, create a file (eg. `hardening.yaml`) and paste the content of the following code snippet into that.
+
+## Minimum Requirements
+
+The **kubernetes** version should be at least `v1.23.6` to have all the most recent security features (eg. the new `PodSecurity` admission plugin, etc).
+
+**N.B.** Some of these configurations have just been added to **kubespray**, so ensure that you have the latest version to make it works properly. Also, ensure that other configurations doesn't override these.
+
+`hardening.yaml`:
+
+```yaml
+# Hardening
+---
+
+## kube-apiserver
+authorization_modes: ['Node','RBAC']
+# AppArmor-based OS
+#kube_apiserver_feature_gates: ['AppArmor=true']
+kube_apiserver_request_timeout: 120s
+kube_apiserver_service_account_lookup: true
+
+# enable kubernetes audit
+kubernetes_audit: true
+audit_log_path: "/var/log/kube-apiserver-log.json"
+audit_log_maxage: 30
+audit_log_maxbackups: 10
+audit_log_maxsize: 100
+
+tls_min_version: VersionTLS12
+tls_cipher_suites:
+ - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
+ - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
+ - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
+
+# enable encryption at rest
+kube_encrypt_secret_data: true
+kube_encryption_resources: [secrets]
+kube_encryption_algorithm: "secretbox"
+
+kube_apiserver_enable_admission_plugins: ['EventRateLimit,AlwaysPullImages,ServiceAccount,NamespaceLifecycle,NodeRestriction,LimitRanger,ResourceQuota,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,PodNodeSelector,PodSecurity']
+kube_apiserver_admission_control_config_file: true
+# EventRateLimit plugin configuration
+kube_apiserver_admission_event_rate_limits:
+ limit_1:
+ type: Namespace
+ qps: 50
+ burst: 100
+ cache_size: 2000
+ limit_2:
+ type: User
+ qps: 50
+ burst: 100
+kube_profiling: false
+
+## kube-controller-manager
+kube_controller_manager_bind_address: 127.0.0.1
+kube_controller_terminated_pod_gc_threshold: 50
+# AppArmor-based OS
+#kube_controller_feature_gates: ["RotateKubeletServerCertificate=true","AppArmor=true"]
+kube_controller_feature_gates: ["RotateKubeletServerCertificate=true"]
+
+## kube-scheduler
+kube_scheduler_bind_address: 127.0.0.1
+kube_kubeadm_scheduler_extra_args:
+ profiling: false
+# AppArmor-based OS
+#kube_scheduler_feature_gates: ["AppArmor=true"]
+
+## etcd
+etcd_deployment_type: kubeadm
+
+## kubelet
+kubelet_authorization_mode_webhook: true
+kubelet_authentication_token_webhook: true
+kube_read_only_port: 0
+kubelet_rotate_server_certificates: true
+kubelet_protect_kernel_defaults: true
+kubelet_event_record_qps: 1
+kubelet_rotate_certificates: true
+kubelet_streaming_connection_idle_timeout: "5m"
+kubelet_make_iptables_util_chains: true
+kubelet_feature_gates: ["RotateKubeletServerCertificate=true","SeccompDefault=true"]
+kubelet_seccomp_default: true
+
+# additional configurations
+kube_owner: root
+kube_cert_group: root
+
+# create a default Pod Security Configuration and deny running of insecure pods
+# kube_system namespace is exempted by default
+kube_pod_security_use_default: true
+kube_pod_security_default_enforce: restricted
+```
+
+Let's take a deep look to the resultant **kubernetes** configuration:
+
+* The `anonymous-auth` (on `kube-apiserver`) is set to `true` by default. This is fine, because it is considered safe if you enable `RBAC` for the `authorization-mode`.
+* The `enable-admission-plugins` has not the `PodSecurityPolicy` admission plugin. This because it is going to be definitely removed from **kubernetes** `v1.25`. For this reason we decided to set the newest `PodSecurity` (for more details, please take a look here: <https://kubernetes.io/docs/concepts/security/pod-security-admission/>). Then, we set the `EventRateLimit` plugin, providing additional configuration files (that are automatically created under the hood and mounted inside the `kube-apiserver` container) to make it work.
+* The `encryption-provider-config` provide encryption at rest. This means that the `kube-apiserver` encrypt data that is going to be stored before they reach `etcd`. So the data is completely unreadable from `etcd` (in case an attacker is able to exploit this).
+* The `rotateCertificates` in `KubeletConfiguration` is set to `true` along with `serverTLSBootstrap`. This could be used in alternative to `tlsCertFile` and `tlsPrivateKeyFile` parameters. Additionally it automatically generates certificates by itself, but you need to manually approve them or at least using an operator to do this (for more details, please take a look here: <https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/>).
+* If you are installing **kubernetes** in an AppArmor-based OS (eg. Debian/Ubuntu) you can enable the `AppArmor` feature gate uncommenting the lines with the comment `# AppArmor-based OS` on top.
+
+Once you have the file properly filled, you can run the **Ansible** command to start the installation:
+
+```bash
+ansible-playbook -v cluster.yml \
+ -i inventory.ini \
+ -b --become-user=root \
+ --private-key ~/.ssh/id_ecdsa \
+ -e "@vars.yaml" \
+ -e "@hardening.yaml"
+```
+
+**N.B.** The `vars.yaml` contains our general cluster information (SANs, load balancer, dns, etc..) and `hardening.yaml` is the file described above.
+
+Once completed the cluster deployment, don't forget to approve the generated certificates (check them with `kubectl get csr`, approve with `kubectl certificate approve <csr_name>`). This action is necessary because the `secureTLSBootstrap` option and `RotateKubeletServerCertificate` feature gate for `kubelet` are enabled (CIS [4.2.11](https://www.tenable.com/audits/items/CIS_Kubernetes_v1.20_v1.0.0_Level_1_Worker.audit:05af3dfbca8e0c3fb3559c6c7de29191), [4.2.12](https://www.tenable.com/audits/items/CIS_Kubernetes_v1.20_v1.0.0_Level_1_Worker.audit:5351c76f8c5bff8f98c29a5200a35435)).
--- /dev/null
+# AWS ALB Ingress Controller
+
+**NOTE:** The current image version is `v1.1.6`. Please file any issues you find and note the version used.
+
+The AWS ALB Ingress Controller satisfies Kubernetes [ingress resources](https://kubernetes.io/docs/user-guide/ingress) by provisioning [Application Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html).
+
+This project was originated by [Ticketmaster](https://github.com/ticketmaster) and [CoreOS](https://github.com/coreos) as part of Ticketmaster's move to AWS and CoreOS Tectonic. Learn more about Ticketmaster's Kubernetes initiative from Justin Dean's video at [Tectonic Summit](https://www.youtube.com/watch?v=wqXVKneP0Hg).
+
+This project was donated to Kubernetes SIG-AWS to allow AWS, CoreOS, Ticketmaster and other SIG-AWS contributors to officially maintain the project. SIG-AWS reached this consensus on June 1, 2018.
+
+## Documentation
+
+Checkout our [Live Docs](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/)!
+
+## Getting started
+
+To get started with the controller, see our [walkthrough](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/walkthrough/echoserver/).
+
+## Setup
+
+- See [controller setup](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/setup/) on how to install ALB ingress controller
+- See [external-dns setup](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/external-dns/setup/) for how to setup the external-dns to manage route 53 records.
+
+## Building
+
+For details on building this project, see our [building guide](https://kubernetes-sigs.github.io/aws-alb-ingress-controller/BUILDING/).
+
+## Community, discussion, contribution, and support
+
+Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/).
+
+You can reach the maintainers of this project at:
+
+- [Slack channel](https://kubernetes.slack.com/messages/sig-aws)
+- [Mailing list](https://groups.google.com/forum/#!forum/kubernetes-sig-aws)
+
+### Code of conduct
+
+Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md).
+
+## License
+
+[](https://app.fossa.io/projects/git%2Bgithub.com%2Fcoreos%2Falb-ingress-controller?ref=badge_large)
--- /dev/null
+# Installation Guide
+
+## Contents
+
+- [Prerequisite Generic Deployment Command](#prerequisite-generic-deployment-command)
+ - [Provider Specific Steps](#provider-specific-steps)
+ - [Docker for Mac](#docker-for-mac)
+ - [minikube](#minikube)
+ - [AWS](#aws)
+ - [GCE - GKE](#gce-gke)
+ - [Azure](#azure)
+ - [Bare-metal](#bare-metal)
+ - [Verify installation](#verify-installation)
+ - [Detect installed version](#detect-installed-version)
+- [Using Helm](#using-helm)
+
+## Prerequisite Generic Deployment Command
+
+!!! attention
+ The default configuration watches Ingress object from *all the namespaces*.
+ To change this behavior use the flag `--watch-namespace` to limit the scope to a particular namespace.
+
+!!! warning
+ If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions.
+
+!!! attention
+ If you're using GKE you need to initialize your user as a cluster-admin with the following command:
+
+```console
+kubectl create clusterrolebinding cluster-admin-binding \
+--clusterrole cluster-admin \
+--user $(gcloud config get-value account)
+```
+
+The following **Mandatory Command** is required for all deployments except for AWS. See below for the AWS version.
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.2/deploy/static/provider/cloud/deploy.yaml
+```
+
+### Provider Specific Steps
+
+There are cloud provider specific yaml files.
+
+#### Docker for Mac
+
+Kubernetes is available in Docker for Mac (from [version 18.06.0-ce](https://docs.docker.com/docker-for-mac/release-notes/#stable-releases-of-2018))
+
+First you need to [enable kubernetes](https://docs.docker.com/docker-for-mac/#kubernetes).
+
+Then you have to create a service:
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
+```
+
+#### minikube
+
+For standard usage:
+
+```console
+minikube addons enable ingress
+```
+
+For development:
+
+1. Disable the ingress addon:
+
+```console
+minikube addons disable ingress
+```
+
+1. Execute `make dev-env`
+1. Confirm the `nginx-ingress-controller` deployment exists:
+
+```console
+$ kubectl get pods -n ingress-nginx
+NAME READY STATUS RESTARTS AGE
+default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s
+nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s
+```
+
+#### AWS
+
+In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of `Type=LoadBalancer`.
+Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB)
+Please check the [elastic load balancing AWS details page](https://aws.amazon.com/elasticloadbalancing/details/)
+
+##### Elastic Load Balancer - ELB
+
+This setup requires to choose in which layer (L4 or L7) we want to configure the Load Balancer:
+
+- [Layer 4](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer): Use an Network Load Balancer (NLB) with TCP as the listener protocol for ports 80 and 443.
+- [Layer 7](https://en.wikipedia.org/wiki/OSI_model#Layer_7:_Application_Layer): Use an Elastic Load Balancer (ELB) with HTTP as the listener protocol for port 80 and terminate TLS in the ELB
+
+For L4:
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy.yaml
+```
+
+For L7:
+
+Change the value of `service.beta.kubernetes.io/aws-load-balancer-ssl-cert` in the file `provider/aws/deploy-tls-termination.yaml` replacing the dummy id with a valid one. The dummy value is `"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"`
+
+Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the [ELB Idle Timeouts section](#elb-idle-timeouts) for additional information. If a change is required, users will need to update the value of `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` in `provider/aws/deploy-tls-termination.yaml`
+
+Then execute:
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy-tls-termination.yaml
+```
+
+This example creates an ELB with just two listeners, one in port 80 and another in port 443
+
+
+
+##### ELB Idle Timeouts
+
+In some scenarios users will need to modify the value of the ELB idle timeout.
+Users need to ensure the idle timeout is less than the [keepalive_timeout](http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) that is configured for NGINX.
+By default NGINX `keepalive_timeout` is set to `75s`.
+
+The default ELB idle timeout will work for most scenarios, unless the NGINX [keepalive_timeout](http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) has been modified,
+in which case `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` will need to be modified to ensure it is less than the `keepalive_timeout` the user has configured.
+
+_Please Note: An idle timeout of `3600s` is recommended when using WebSockets._
+
+More information with regards to idle timeouts for your Load Balancer can be found in the [official AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html).
+
+##### Network Load Balancer (NLB)
+
+This type of load balancer is supported since v1.10.0 as an ALPHA feature.
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/service-nlb.yaml
+```
+
+#### GCE-GKE
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
+```
+
+**Important Note:** proxy protocol is not supported in GCE/GKE
+
+#### Azure
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
+```
+
+#### Bare-metal
+
+Using [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport):
+
+```console
+kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
+```
+
+!!! tip
+ For extended notes regarding deployments on bare-metal, see [Bare-metal considerations](./baremetal.md).
+
+### Verify installation
+
+To check if the ingress controller pods have started, run the following command:
+
+```console
+kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch
+```
+
+Once the operator pods are running, you can cancel the above command by typing `Ctrl+C`.
+Now, you are ready to create your first ingress.
+
+### Detect installed version
+
+To detect which version of the ingress controller is running, exec into the pod and run `nginx-ingress-controller version` command.
+
+```console
+POD_NAMESPACE=ingress-nginx
+POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
+
+kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
+```
+
+## Using Helm
+
+NGINX Ingress controller can be installed via [Helm](https://helm.sh/) using the chart [ingress-nginx/ingress-nginx](https://kubernetes.github.io/ingress-nginx).
+Official documentation is [here](https://kubernetes.github.io/ingress-nginx/deploy/#using-helm)
+
+To install the chart with the release name `my-nginx`:
+
+```console
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm install my-nginx ingress-nginx/ingress-nginx
+```
+
+Detect installed version:
+
+```console
+POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
+kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version
+```
--- /dev/null
+# Kubespray (kubespray) in own ansible playbooks repo
+
+1. Fork [kubespray repo](https://github.com/kubernetes-sigs/kubespray) to your personal/organisation account on github.
+ Note:
+ * All forked public repos at github will be also public, so **never commit sensitive data to your public forks**.
+ * List of all forked repos could be retrieved from github page of original project.
+
+2. Add **forked repo** as submodule to desired folder in your existent ansible repo (for example 3d/kubespray):
+ ```git submodule add https://github.com/YOUR_GITHUB/kubespray.git kubespray```
+ Git will create `.gitmodules` file in your existent ansible repo:
+
+ ```ini
+ [submodule "3d/kubespray"]
+ path = 3d/kubespray
+ url = https://github.com/YOUR_GITHUB/kubespray.git
+ ```
+
+3. Configure git to show submodule status:
+```git config --global status.submoduleSummary true```
+
+4. Add *original* kubespray repo as upstream:
+```cd kubespray && git remote add upstream https://github.com/kubernetes-sigs/kubespray.git```
+
+5. Sync your master branch with upstream:
+
+ ```ShellSession
+ git checkout master
+ git fetch upstream
+ git merge upstream/master
+ git push origin master
+ ```
+
+6. Create a new branch which you will use in your working environment:
+```git checkout -b work```
+ ***Never*** use master branch of your repository for your commits.
+
+7. Modify path to library and roles in your ansible.cfg file (role naming should be unique, you may have to rename your existent roles if they have same names as kubespray project),
+ if you had roles in your existing ansible project before, you can add the path to those separated with `:`:
+
+8. ```ini
+ ...
+ library = ./library/:3d/kubespray/library/
+ roles_path = ./roles/:3d/kubespray/roles/
+ ...
+ ```
+
+9. Copy and modify configs from kubespray `group_vars` folder to corresponding `group_vars` folder in your existent project.
+You could rename *all.yml* config to something else, i.e. *kubespray.yml* and create corresponding group in your inventory file, which will include all hosts groups related to kubernetes setup.
+
+10. Modify your ansible inventory file by adding mapping of your existent groups (if any) to kubespray naming.
+ For example:
+
+ ```ini
+ ...
+ #Kargo groups:
+ [kube_node:children]
+ kubenode
+
+ [k8s_cluster:children]
+ kubernetes
+
+ [etcd:children]
+ kubemaster
+ kubemaster-ha
+
+ [kube_control_plane:children]
+ kubemaster
+ kubemaster-ha
+
+ [kubespray:children]
+ kubernetes
+ ```
+
+ * Last entry here needed to apply kubespray.yml config file, renamed from all.yml of kubespray project.
+
+11. Now you can include kubespray tasks in you existent playbooks by including cluster.yml file:
+
+ ```yml
+ - name: Import kubespray playbook
+ ansible.builtin.import_playbook: 3d/kubespray/cluster.yml
+ ```
+
+ Or your could copy separate tasks from cluster.yml into your ansible repository.
+
+12. Commit changes to your ansible repo. Keep in mind, that submodule folder is just a link to the git commit hash of your forked repo.
+When you update your "work" branch you need to commit changes to ansible repo as well.
+Other members of your team should use ```git submodule sync```, ```git submodule update --init``` to get actual code from submodule.
+
+## Contributing
+
+If you made useful changes or fixed a bug in existent kubespray repo, use this flow for PRs to original kubespray repo.
+
+1. Sign the [CNCF CLA](https://git.k8s.io/community/CLA.md).
+
+2. Change working directory to git submodule directory (3d/kubespray).
+
+3. Setup desired user.name and user.email for submodule.
+If kubespray is only one submodule in your repo you could use something like:
+```git submodule foreach --recursive 'git config user.name "First Last" && git config user.email "your-email-address@used.for.cncf"'```
+
+4. Sync with upstream master:
+
+ ```ShellSession
+ git fetch upstream
+ git merge upstream/master
+ git push origin master
+ ```
+
+5. Create new branch for the specific fixes that you want to contribute:
+```git checkout -b fixes-name-date-index```
+Branch name should be self explaining to you, adding date and/or index will help you to track/delete your old PRs.
+
+6. Find git hash of your commit in "work" repo and apply it to newly created "fix" repo:
+
+ ```ShellSession
+ git cherry-pick <COMMIT_HASH>
+ ```
+
+7. If you have several temporary-stage commits - squash them using [```git rebase -i```](https://eli.thegreenplace.net/2014/02/19/squashing-github-pull-requests-into-a-single-commit)
+Also you could use interactive rebase (```git rebase -i HEAD~10```) to delete commits which you don't want to contribute into original repo.
+
+8. When your changes is in place, you need to check upstream repo one more time because it could be changed during your work.
+Check that you're on correct branch:
+```git status```
+And pull changes from upstream (if any):
+```git pull --rebase upstream master```
+
+9. Now push your changes to your **fork** repo with ```git push```. If your branch doesn't exists on github, git will propose you to use something like ```git push --set-upstream origin fixes-name-date-index```.
+
+10. Open you forked repo in browser, on the main page you will see proposition to create pull request for your newly created branch. Check proposed diff of your PR. If something is wrong you could safely delete "fix" branch on github using ```git push origin --delete fixes-name-date-index```, ```git branch -D fixes-name-date-index``` and start whole process from the beginning.
+If everything is fine - add description about your changes (what they do and why they're needed) and confirm pull request creation.
--- /dev/null
+# Kata Containers
+
+[Kata Containers](https://katacontainers.io) is a secure container runtime with lightweight virtual machines that supports multiple hypervisor solutions.
+
+## Hypervisors
+
+_Qemu_ is the only hypervisor supported by Kubespray.
+
+## Installation
+
+To enable Kata Containers, set the following variables:
+
+**k8s-cluster.yml**:
+
+```yaml
+container_manager: containerd
+kata_containers_enabled: true
+```
+
+**etcd.yml**:
+
+```yaml
+etcd_deployment_type: host
+```
+
+## Usage
+
+By default, runc is used for pods.
+Kubespray generates the runtimeClass kata-qemu, and it is necessary to specify it as
+the runtimeClassName of a pod spec to use Kata Containers:
+
+```shell
+$ kubectl get runtimeclass
+NAME HANDLER AGE
+kata-qemu kata-qemu 3m34s
+$
+$ cat nginx.yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: mypod
+spec:
+ runtimeClassName: kata-qemu
+ containers:
+ - name: nginx
+ image: nginx:1.14.2
+$
+$ kubectl apply -f nginx.yaml
+```
+
+## Configuration
+
+### Recommended : Pod Overhead
+
+[Pod Overhead](https://kubernetes.io/docs/concepts/configuration/pod-overhead/) is a feature for accounting for the resources consumed by the Runtime Class used by the Pod.
+
+When this feature is enabled, Kubernetes will count the fixed amount of CPU and memory set in the configuration as used by the virtual machine and not by the containers running in the Pod.
+
+Pod Overhead is mandatory if you run Pods with Kata Containers that use [resources limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits).
+
+**Set cgroup driver**:
+
+To enable Pod Overhead feature you have to configure Kubelet with the appropriate cgroup driver, using the following configuration:
+
+`cgroupfs` works best:
+
+```yaml
+kubelet_cgroup_driver: cgroupfs
+```
+
+... but when using `cgroups v2` (see <https://www.redhat.com/en/blog/world-domination-cgroups-rhel-8-welcome-cgroups-v2>) you can use systemd as well:
+
+```yaml
+kubelet_cgroup_driver: systemd
+```
+
+**Qemu hypervisor configuration**:
+
+The configuration for the Qemu hypervisor uses the following values:
+
+```yaml
+kata_containers_qemu_overhead: true
+kata_containers_qemu_overhead_fixed_cpu: 10m
+kata_containers_qemu_overhead_fixed_memory: 290Mi
+```
+
+### Optional : Select Kata Containers version
+
+Optionally you can select the Kata Containers release version to be installed. The available releases are published in [GitHub](https://github.com/kata-containers/kata-containers/releases).
+
+```yaml
+kata_containers_version: 2.2.2
+```
+
+### Optional : Debug
+
+Debug is disabled by default for all the components of Kata Containers. You can change this behaviour with the following configuration:
+
+```yaml
+kata_containers_qemu_debug: 'false'
+```
--- /dev/null
+# Kube-OVN
+
+Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.
+
+For more information please check [Kube-OVN documentation](https://github.com/alauda/kube-ovn)
+
+**Warning:** Kernel version (`cat /proc/version`) needs to be different than `3.10.0-862` or kube-ovn won't start and will print this message:
+
+```bash
+kernel version 3.10.0-862 has a nat related bug that will affect ovs function, please update to a version greater than 3.10.0-898
+```
+
+## How to use it
+
+Enable kube-ovn in `group_vars/k8s_cluster/k8s_cluster.yml`
+
+```yml
+...
+kube_network_plugin: kube-ovn
+...
+```
+
+## Verifying kube-ovn install
+
+Kube-OVN run ovn and controller in `kube-ovn` namespace
+
+* Check the status of kube-ovn pods
+
+```ShellSession
+# From the CLI
+kubectl get pod -n kube-ovn
+
+# Output
+NAME READY STATUS RESTARTS AGE
+kube-ovn-cni-49lsm 1/1 Running 0 2d20h
+kube-ovn-cni-9db8f 1/1 Running 0 2d20h
+kube-ovn-cni-wftdk 1/1 Running 0 2d20h
+kube-ovn-controller-68d7bb48bd-7tnvg 1/1 Running 0 2d21h
+ovn-central-6675dbb7d9-d7z8m 1/1 Running 0 4d16h
+ovs-ovn-hqn8p 1/1 Running 0 4d16h
+ovs-ovn-hvpl8 1/1 Running 0 4d16h
+ovs-ovn-r5frh 1/1 Running 0 4d16h
+```
+
+* Check the default and node subnet
+
+```ShellSession
+# From the CLI
+kubectl get subnet
+
+# Output
+NAME PROTOCOL CIDR PRIVATE NAT
+join IPv4 100.64.0.0/16 false false
+ovn-default IPv4 10.16.0.0/16 false true
+```
--- /dev/null
+# Kube-router
+
+Kube-router is a L3 CNI provider, as such it will setup IPv4 routing between
+nodes to provide Pods' networks reachability.
+
+See [kube-router documentation](https://www.kube-router.io/).
+
+## Verifying kube-router install
+
+Kube-router runs its pods as a `DaemonSet` in the `kube-system` namespace:
+
+* Check the status of kube-router pods
+
+```ShellSession
+# From the CLI
+kubectl get pod --namespace=kube-system -l k8s-app=kube-router -owide
+
+# output
+NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
+kube-router-4f679 1/1 Running 0 2d 192.168.186.4 mykube-k8s-node-nf-2 <none>
+kube-router-5slf8 1/1 Running 0 2d 192.168.186.11 mykube-k8s-node-nf-3 <none>
+kube-router-lb6k2 1/1 Running 0 20h 192.168.186.14 mykube-k8s-node-nf-6 <none>
+kube-router-rzvrb 1/1 Running 0 20h 192.168.186.17 mykube-k8s-node-nf-4 <none>
+kube-router-v6n56 1/1 Running 0 2d 192.168.186.6 mykube-k8s-node-nf-1 <none>
+kube-router-wwhg8 1/1 Running 0 20h 192.168.186.16 mykube-k8s-node-nf-5 <none>
+kube-router-x2xs7 1/1 Running 0 2d 192.168.186.10 mykube-k8s-master-1 <none>
+```
+
+* Peek at kube-router container logs:
+
+```ShellSession
+# From the CLI
+kubectl logs --namespace=kube-system -l k8s-app=kube-router | grep Peer.Up
+
+# output
+time="2018-09-17T16:47:14Z" level=info msg="Peer Up" Key=192.168.186.6 State=BGP_FSM_OPENCONFIRM Topic=Peer
+time="2018-09-17T16:47:16Z" level=info msg="Peer Up" Key=192.168.186.11 State=BGP_FSM_OPENCONFIRM Topic=Peer
+time="2018-09-17T16:47:46Z" level=info msg="Peer Up" Key=192.168.186.10 State=BGP_FSM_OPENCONFIRM Topic=Peer
+time="2018-09-18T19:12:24Z" level=info msg="Peer Up" Key=192.168.186.14 State=BGP_FSM_OPENCONFIRM Topic=Peer
+time="2018-09-18T19:12:28Z" level=info msg="Peer Up" Key=192.168.186.17 State=BGP_FSM_OPENCONFIRM Topic=Peer
+time="2018-09-18T19:12:38Z" level=info msg="Peer Up" Key=192.168.186.16 State=BGP_FSM_OPENCONFIRM Topic=Peer
+[...]
+```
+
+## Gathering kube-router state
+
+Kube-router Pods come bundled with a "Pod Toolbox" which provides very
+useful internal state views for:
+
+* IPVS: via `ipvsadm`
+* BGP peering and routing info: via `gobgp`
+
+You need to `kubectl exec -it ...` into a kube-router container to use these, see
+<https://www.kube-router.io/docs/pod-toolbox/> for details.
+
+## Kube-router configuration
+
+You can change the default configuration by overriding `kube_router_...` variables
+(as found at `roles/network_plugin/kube-router/defaults/main.yml`),
+these are named to follow `kube-router` command-line options as per
+<https://www.kube-router.io/docs/user-guide/#try-kube-router-with-cluster-installers>.
+
+## Advanced BGP Capabilities
+
+<https://github.com/cloudnativelabs/kube-router#advanced-bgp-capabilities>
+
+If you have other networking devices or SDN systems that talk BGP, kube-router will fit in perfectly.
+From a simple full node-to-node mesh to per-node peering configurations, most routing needs can be attained.
+The configuration is Kubernetes native (annotations) just like the rest of kube-router.
+
+For more details please refer to the <https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md.>
+
+Next options will set up annotations for kube-router, using `kubectl annotate` command.
+
+```yml
+kube_router_annotations_master: []
+kube_router_annotations_node: []
+kube_router_annotations_all: []
+```
--- /dev/null
+# kube-vip
+
+kube-vip provides Kubernetes clusters with a virtual IP and load balancer for both the control plane (for building a highly-available cluster) and Kubernetes Services of type LoadBalancer without relying on any external hardware or software.
+
+## Install
+
+You have to explicitly enable the kube-vip extension:
+
+```yaml
+kube_vip_enabled: true
+```
+
+You also need to enable
+[kube-vip as HA, Load Balancer, or both](https://kube-vip.io/docs/installation/static/#kube-vip-as-ha-load-balancer-or-both):
+
+```yaml
+# HA for control-plane, requires a VIP
+kube_vip_controlplane_enabled: true
+kube_vip_address: 10.42.42.42
+loadbalancer_apiserver:
+ address: "{{ kube_vip_address }}"
+ port: 6443
+# kube_vip_interface: ens160
+
+# LoadBalancer for services
+kube_vip_services_enabled: false
+# kube_vip_services_interface: ens320
+```
+
+> Note: When using `kube-vip` as LoadBalancer for services,
+[additional manual steps](https://kube-vip.io/docs/usage/cloud-provider/)
+are needed.
+
+If using [ARP mode](https://kube-vip.io/docs/installation/static/#arp) :
+
+```yaml
+kube_vip_arp_enabled: true
+```
+
+If using [BGP mode](https://kube-vip.io/docs/installation/static/#bgp) :
+
+```yaml
+kube_vip_bgp_enabled: true
+kube_vip_local_as: 65000
+kube_vip_bgp_routerid: 192.168.0.2
+kube_vip_bgppeers:
+- 192.168.0.10:65000::false
+- 192.168.0.11:65000::false
+# kube_vip_bgp_peeraddress:
+# kube_vip_bgp_peerpass:
+# kube_vip_bgp_peeras:
+```
--- /dev/null
+# CephFS Volume Provisioner for Kubernetes 1.5+
+
+[](https://quay.io/repository/external_storage/cephfs-provisioner)
+
+Using Ceph volume client
+
+## Development
+
+Compile the provisioner
+
+``` console
+make
+```
+
+Make the container image and push to the registry
+
+``` console
+make push
+```
+
+## Test instruction
+
+- Start Kubernetes local cluster
+
+See [Kubernetes](https://kubernetes.io/)
+
+- Create a Ceph admin secret
+
+``` bash
+ceph auth get client.admin 2>&1 |grep "key = " |awk '{print $3'} |xargs echo -n > /tmp/secret
+kubectl create ns cephfs
+kubectl create secret generic ceph-secret-admin --from-file=/tmp/secret --namespace=cephfs
+```
+
+- Start CephFS provisioner
+
+The following example uses `cephfs-provisioner-1` as the identity for the instance and assumes kubeconfig is at `/root/.kube`. The identity should remain the same if the provisioner restarts. If there are multiple provisioners, each should have a different identity.
+
+``` bash
+docker run -ti -v /root/.kube:/kube -v /var/run/kubernetes:/var/run/kubernetes --privileged --net=host cephfs-provisioner /usr/local/bin/cephfs-provisioner -master=http://127.0.0.1:8080 -kubeconfig=/kube/config -id=cephfs-provisioner-1
+```
+
+Alternatively, deploy it in kubernetes, see [deployment](deploy/README.md).
+
+- Create a CephFS Storage Class
+
+Replace Ceph monitor's IP in [example class](example/class.yaml) with your own and create storage class:
+
+``` bash
+kubectl create -f example/class.yaml
+```
+
+- Create a claim
+
+``` bash
+kubectl create -f example/claim.yaml
+```
+
+- Create a Pod using the claim
+
+``` bash
+kubectl create -f example/test-pod.yaml
+```
+
+## Known limitations
+
+- Kernel CephFS doesn't work with SELinux, setting SELinux label in Pod's securityContext will not work.
+- Kernel CephFS doesn't support quota or capacity, capacity requested by PVC is not enforced or validated.
+- Currently each Ceph user created by the provisioner has `allow r` MDS cap to permit CephFS mount.
+
+## Acknowledgement
+
+Inspired by CephFS Manila provisioner and conversation with John Spray
--- /dev/null
+# Local Storage Provisioner
+
+The [local storage provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume)
+is NOT a dynamic storage provisioner as you would
+expect from a cloud provider. Instead, it simply creates PersistentVolumes for
+all mounts under the host_dir of the specified storage class.
+These storage classes are specified in the `local_volume_provisioner_storage_classes` nested dictionary.
+Example:
+
+```yaml
+local_volume_provisioner_storage_classes:
+ local-storage:
+ host_dir: /mnt/disks
+ mount_dir: /mnt/disks
+ fast-disks:
+ host_dir: /mnt/fast-disks
+ mount_dir: /mnt/fast-disks
+ block_cleaner_command:
+ - "/scripts/shred.sh"
+ - "2"
+ volume_mode: Filesystem
+ fs_type: ext4
+```
+
+For each key in `local_volume_provisioner_storage_classes` a storageClass with the
+same name is created. The subkeys of each storage class are converted to camelCase and added
+as attributes to the storageClass.
+The result of the above example is:
+
+```yaml
+data:
+ storageClassMap: |
+ local-storage:
+ hostDir: /mnt/disks
+ mountDir: /mnt/disks
+ fast-disks:
+ hostDir: /mnt/fast-disks
+ mountDir: /mnt/fast-disks
+ blockCleanerCommand:
+ - "/scripts/shred.sh"
+ - "2"
+ volumeMode: Filesystem
+ fsType: ext4
+```
+
+The default StorageClass is local-storage on /mnt/disks,
+the rest of this doc will use that path as an example.
+
+## Examples to create local storage volumes
+
+1. tmpfs method:
+
+``` bash
+for vol in vol1 vol2 vol3; do
+mkdir /mnt/disks/$vol
+mount -t tmpfs -o size=5G $vol /mnt/disks/$vol
+done
+```
+
+The tmpfs method is not recommended for production because the mount is not
+persistent and data will be deleted on reboot.
+
+1. Mount physical disks
+
+``` bash
+mkdir /mnt/disks/ssd1
+mount /dev/vdb1 /mnt/disks/ssd1
+```
+
+Physical disks are recommended for production environments because it offers
+complete isolation in terms of I/O and capacity.
+
+1. Mount unpartitioned physical devices
+
+``` bash
+for disk in /dev/sdc /dev/sdd /dev/sde; do
+ ln -s $disk /mnt/disks
+done
+```
+
+This saves time of precreating filesystems. Note that your storageclass must have
+volume_mode set to "Filesystem" and fs_type defined. If either is not set, the
+disk will be added as a raw block device.
+
+1. File-backed sparsefile method
+
+``` bash
+truncate /mnt/disks/disk5 --size 2G
+mkfs.ext4 /mnt/disks/disk5
+mkdir /mnt/disks/vol5
+mount /mnt/disks/disk5 /mnt/disks/vol5
+```
+
+If you have a development environment and only one disk, this is the best way
+to limit the quota of persistent volumes.
+
+1. Simple directories
+
+In a development environment using `mount --bind` works also, but there is no capacity
+management.
+
+1. Block volumeMode PVs
+
+Create a symbolic link under discovery directory to the block device on the node. To use
+raw block devices in pods, volume_type should be set to "Block".
+
+## Usage notes
+
+Beta PV.NodeAffinity field is used by default. If running against an older K8s
+version, the useAlphaAPI flag must be set in the configMap.
+
+The volume provisioner cannot calculate volume sizes correctly, so you should
+delete the daemonset pod on the relevant host after creating volumes. The pod
+will be recreated and read the size correctly.
+
+Make sure to make any mounts persist via /etc/fstab or with systemd mounts (for
+Flatcar Container Linux). Pods with persistent volume claims will not be
+able to start if the mounts become unavailable.
+
+## Further reading
+
+Refer to the upstream docs here: <https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume>
--- /dev/null
+# RBD Volume Provisioner for Kubernetes 1.5+
+
+`rbd-provisioner` is an out-of-tree dynamic provisioner for Kubernetes 1.5+.
+You can use it quickly & easily deploy ceph RBD storage that works almost
+anywhere.
+
+It works just like in-tree dynamic provisioner. For more information on how
+dynamic provisioning works, see [the docs](http://kubernetes.io/docs/user-guide/persistent-volumes/)
+or [this blog post](http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html).
+
+## Development
+
+Compile the provisioner
+
+```console
+make
+```
+
+Make the container image and push to the registry
+
+```console
+make push
+```
+
+## Test instruction
+
+* Start Kubernetes local cluster
+
+See [Kubernetes](https://kubernetes.io/).
+
+* Create a Ceph admin secret
+
+```bash
+ceph auth get client.admin 2>&1 |grep "key = " |awk '{print $3'} |xargs echo -n > /tmp/secret
+kubectl create secret generic ceph-admin-secret --from-file=/tmp/secret --namespace=kube-system
+```
+
+* Create a Ceph pool and a user secret
+
+```bash
+ceph osd pool create kube 8 8
+ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube'
+ceph auth get-key client.kube > /tmp/secret
+kubectl create secret generic ceph-secret --from-file=/tmp/secret --namespace=kube-system
+```
+
+* Start RBD provisioner
+
+The following example uses `rbd-provisioner-1` as the identity for the instance and assumes kubeconfig is at `/root/.kube`. The identity should remain the same if the provisioner restarts. If there are multiple provisioners, each should have a different identity.
+
+```bash
+docker run -ti -v /root/.kube:/kube -v /var/run/kubernetes:/var/run/kubernetes --privileged --net=host quay.io/external_storage/rbd-provisioner /usr/local/bin/rbd-provisioner -master=http://127.0.0.1:8080 -kubeconfig=/kube/config -id=rbd-provisioner-1
+```
+
+Alternatively, deploy it in kubernetes, see [deployment](deploy/README.md).
+
+* Create a RBD Storage Class
+
+Replace Ceph monitor's IP in [examples/class.yaml](examples/class.yaml) with your own and create storage class:
+
+```bash
+kubectl create -f examples/class.yaml
+```
+
+* Create a claim
+
+```bash
+kubectl create -f examples/claim.yaml
+```
+
+* Create a Pod using the claim
+
+```bash
+kubectl create -f examples/test-pod.yaml
+```
+
+## Acknowledgements
+
+* This provisioner is extracted from [Kubernetes core](https://github.com/kubernetes/kubernetes) with some modifications for this project.
--- /dev/null
+# Private Docker Registry in Kubernetes
+
+Kubernetes offers an optional private Docker registry addon, which you can turn
+on when you bring up a cluster or install later. This gives you a place to
+store truly private Docker images for your cluster.
+
+## How it works
+
+The private registry runs as a `Pod` in your cluster. It does not currently
+support SSL or authentication, which triggers Docker's "insecure registry"
+logic. To work around this, we run a proxy on each node in the cluster,
+exposing a port onto the node (via a hostPort), which Docker accepts as
+"secure", since it is accessed by `localhost`.
+
+## Turning it on
+
+Some cluster installs (e.g. GCE) support this as a cluster-birth flag. The
+`ENABLE_CLUSTER_REGISTRY` variable in `cluster/gce/config-default.sh` governs
+whether the registry is run or not. To set this flag, you can specify
+`KUBE_ENABLE_CLUSTER_REGISTRY=true` when running `kube-up.sh`. If your cluster
+does not include this flag, the following steps should work. Note that some of
+this is cloud-provider specific, so you may have to customize it a bit.
+
+### Make some storage
+
+The primary job of the registry is to store data. To do that we have to decide
+where to store it. For cloud environments that have networked storage, we can
+use Kubernetes's `PersistentVolume` abstraction. The following template is
+expanded by `salt` in the GCE cluster turnup, but can easily be adapted to
+other situations:
+
+<!-- BEGIN MUNGE: EXAMPLE registry-pv.yaml.in -->
+``` yaml
+kind: PersistentVolume
+apiVersion: v1
+metadata:
+ name: kube-system-kube-registry-pv
+spec:
+{% if pillar.get('cluster_registry_disk_type', '') == 'gce' %}
+ capacity:
+ storage: {{ pillar['cluster_registry_disk_size'] }}
+ accessModes:
+ - ReadWriteOnce
+ gcePersistentDisk:
+ pdName: "{{ pillar['cluster_registry_disk_name'] }}"
+ fsType: "ext4"
+{% endif %}
+```
+<!-- END MUNGE: EXAMPLE registry-pv.yaml.in -->
+
+If, for example, you wanted to use NFS you would just need to change the
+`gcePersistentDisk` block to `nfs`. See
+[here](https://kubernetes.io/docs/concepts/storage/volumes/) for more details on volumes.
+
+Note that in any case, the storage (in the case the GCE PersistentDisk) must be
+created independently - this is not something Kubernetes manages for you (yet).
+
+### I don't want or don't have persistent storage
+
+If you are running in a place that doesn't have networked storage, or if you
+just want to kick the tires on this without committing to it, you can easily
+adapt the `ReplicationController` specification below to use a simple
+`emptyDir` volume instead of a `persistentVolumeClaim`.
+
+## Claim the storage
+
+Now that the Kubernetes cluster knows that some storage exists, you can put a
+claim on that storage. As with the `PersistentVolume` above, you can start
+with the `salt` template:
+
+<!-- BEGIN MUNGE: EXAMPLE registry-pvc.yaml.in -->
+``` yaml
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+ name: kube-registry-pvc
+ namespace: kube-system
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: {{ pillar['cluster_registry_disk_size'] }}
+```
+<!-- END MUNGE: EXAMPLE registry-pvc.yaml.in -->
+
+This tells Kubernetes that you want to use storage, and the `PersistentVolume`
+you created before will be bound to this claim (unless you have other
+`PersistentVolumes` in which case those might get bound instead). This claim
+gives you the right to use this storage until you release the claim.
+
+## Run the registry
+
+Now we can run a Docker registry:
+
+<!-- BEGIN MUNGE: EXAMPLE registry-rc.yaml -->
+``` yaml
+apiVersion: v1
+kind: ReplicationController
+metadata:
+ name: kube-registry-v0
+ namespace: kube-system
+ labels:
+ k8s-app: registry
+ version: v0
+spec:
+ replicas: 1
+ selector:
+ k8s-app: registry
+ version: v0
+ template:
+ metadata:
+ labels:
+ k8s-app: registry
+ version: v0
+ spec:
+ containers:
+ - name: registry
+ image: registry:2
+ resources:
+ limits:
+ cpu: 100m
+ memory: 100Mi
+ env:
+ - name: REGISTRY_HTTP_ADDR
+ value: :5000
+ - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
+ value: /var/lib/registry
+ volumeMounts:
+ - name: image-store
+ mountPath: /var/lib/registry
+ ports:
+ - containerPort: 5000
+ name: registry
+ protocol: TCP
+ volumes:
+ - name: image-store
+ persistentVolumeClaim:
+ claimName: kube-registry-pvc
+```
+<!-- END MUNGE: EXAMPLE registry-rc.yaml -->
+
+*Note:* that if you have set multiple replicas, make sure your CSI driver has support for the `ReadWriteMany` accessMode.
+
+## Expose the registry in the cluster
+
+Now that we have a registry `Pod` running, we can expose it as a Service:
+
+<!-- BEGIN MUNGE: EXAMPLE registry-svc.yaml -->
+``` yaml
+apiVersion: v1
+kind: Service
+metadata:
+ name: kube-registry
+ namespace: kube-system
+ labels:
+ k8s-app: registry
+ kubernetes.io/name: "KubeRegistry"
+spec:
+ selector:
+ k8s-app: registry
+ ports:
+ - name: registry
+ port: 5000
+ protocol: TCP
+```
+<!-- END MUNGE: EXAMPLE registry-svc.yaml -->
+
+## Expose the registry on each node
+
+Now that we have a running `Service`, we need to expose it onto each Kubernetes
+`Node` so that Docker will see it as `localhost`. We can load a `Pod` on every
+node by creating following daemonset.
+
+<!-- BEGIN MUNGE: EXAMPLE ../../saltbase/salt/kube-registry-proxy/kube-registry-proxy.yaml -->
+``` yaml
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: kube-registry-proxy
+ namespace: kube-system
+ labels:
+ k8s-app: kube-registry-proxy
+ version: v0.4
+spec:
+ template:
+ metadata:
+ labels:
+ k8s-app: kube-registry-proxy
+ kubernetes.io/name: "kube-registry-proxy"
+ version: v0.4
+ spec:
+ containers:
+ - name: kube-registry-proxy
+ image: gcr.io/google_containers/kube-registry-proxy:0.4
+ resources:
+ limits:
+ cpu: 100m
+ memory: 50Mi
+ env:
+ - name: REGISTRY_HOST
+ value: kube-registry.kube-system.svc.cluster.local
+ - name: REGISTRY_PORT
+ value: "5000"
+ ports:
+ - name: registry
+ containerPort: 80
+ hostPort: 5000
+```
+<!-- END MUNGE: EXAMPLE ../../saltbase/salt/kube-registry-proxy/kube-registry-proxy.yaml -->
+
+When modifying replication-controller, service and daemon-set definitions, take
+care to ensure *unique* identifiers for the rc-svc couple and the daemon-set.
+Failing to do so will have register the localhost proxy daemon-sets to the
+upstream service. As a result they will then try to proxy themselves, which
+will, for obvious reasons, not work.
+
+This ensures that port 5000 on each node is directed to the registry `Service`.
+You should be able to verify that it is running by hitting port 5000 with a web
+browser and getting a 404 error:
+
+``` console
+$ curl localhost:5000
+404 page not found
+```
+
+## Using the registry
+
+To use an image hosted by this registry, simply say this in your `Pod`'s
+`spec.containers[].image` field:
+
+``` yaml
+ image: localhost:5000/user/container
+```
+
+Before you can use the registry, you have to be able to get images into it,
+though. If you are building an image on your Kubernetes `Node`, you can spell
+out `localhost:5000` when you build and push. More likely, though, you are
+building locally and want to push to your cluster.
+
+You can use `kubectl` to set up a port-forward from your local node to a
+running Pod:
+
+``` console
+$ POD=$(kubectl get pods --namespace kube-system -l k8s-app=registry \
+ -o template --template '{{range .items}}{{.metadata.name}} {{.status.phase}}{{"\n"}}{{end}}' \
+ | grep Running | head -1 | cut -f1 -d' ')
+
+$ kubectl port-forward --namespace kube-system $POD 5000:5000 &
+```
+
+Now you can build and push images on your local computer as
+`localhost:5000/yourname/container` and those images will be available inside
+your kubernetes cluster with the same name.
--- /dev/null
+# Overview
+
+Distributed system such as Kubernetes are designed to be resilient to the
+failures. More details about Kubernetes High-Availability (HA) may be found at
+[Building High-Availability Clusters](https://kubernetes.io/docs/admin/high-availability/)
+
+To have a simple view the most of parts of HA will be skipped to describe
+Kubelet<->Controller Manager communication only.
+
+By default the normal behavior looks like:
+
+1. Kubelet updates it status to apiserver periodically, as specified by
+ `--node-status-update-frequency`. The default value is **10s**.
+
+2. Kubernetes controller manager checks the statuses of Kubelet every
+ `–-node-monitor-period`. The default value is **5s**.
+
+3. In case the status is updated within `--node-monitor-grace-period` of time,
+ Kubernetes controller manager considers healthy status of Kubelet. The
+ default value is **40s**.
+
+> Kubernetes controller manager and Kubelet work asynchronously. It means that
+> the delay may include any network latency, API Server latency, etcd latency,
+> latency caused by load on one's control plane nodes and so on. So if
+> `--node-status-update-frequency` is set to 5s in reality it may appear in
+> etcd in 6-7 seconds or even longer when etcd cannot commit data to quorum
+> nodes.
+
+## Failure
+
+Kubelet will try to make `nodeStatusUpdateRetry` post attempts. Currently
+`nodeStatusUpdateRetry` is constantly set to 5 in
+[kubelet.go](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/kubelet.go#L102).
+
+Kubelet will try to update the status in
+[tryUpdateNodeStatus](https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/kubelet_node_status.go#L312)
+function. Kubelet uses `http.Client()` Golang method, but has no specified
+timeout. Thus there may be some glitches when API Server is overloaded while
+TCP connection is established.
+
+So, there will be `nodeStatusUpdateRetry` * `--node-status-update-frequency`
+attempts to set a status of node.
+
+At the same time Kubernetes controller manager will try to check
+`nodeStatusUpdateRetry` times every `--node-monitor-period` of time. After
+`--node-monitor-grace-period` it will consider node unhealthy. Pods will then be rescheduled based on the
+[Taint Based Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions)
+timers that you set on them individually, or the API Server's global timers:`--default-not-ready-toleration-seconds` &
+``--default-unreachable-toleration-seconds``.
+
+Kube proxy has a watcher over API. Once pods are evicted, Kube proxy will
+notice and will update iptables of the node. It will remove endpoints from
+services so pods from failed node won't be accessible anymore.
+
+## Recommendations for different cases
+
+## Fast Update and Fast Reaction
+
+If `--node-status-update-frequency` is set to **4s** (10s is default).
+`--node-monitor-period` to **2s** (5s is default).
+`--node-monitor-grace-period` to **20s** (40s is default).
+`--default-not-ready-toleration-seconds` and ``--default-unreachable-toleration-seconds`` are set to **30**
+(300 seconds is default). Note these two values should be integers representing the number of seconds ("s" or "m" for
+seconds\minutes are not specified).
+
+In such scenario, pods will be evicted in **50s** because the node will be
+considered as down after **20s**, and `--default-not-ready-toleration-seconds` or
+``--default-unreachable-toleration-seconds`` occur after **30s** more. However, this scenario creates an overhead on
+etcd as every node will try to update its status every 2 seconds.
+
+If the environment has 1000 nodes, there will be 15000 node updates per
+minute which may require large etcd containers or even dedicated nodes for etcd.
+
+> If we calculate the number of tries, the division will give 5, but in reality
+> it will be from 3 to 5 with `nodeStatusUpdateRetry` attempts of each try. The
+> total number of attempts will vary from 15 to 25 due to latency of all
+> components.
+
+## Medium Update and Average Reaction
+
+Let's set `--node-status-update-frequency` to **20s**
+`--node-monitor-grace-period` to **2m** and `--default-not-ready-toleration-seconds` and
+``--default-unreachable-toleration-seconds`` to **60**.
+In that case, Kubelet will try to update status every 20s. So, it will be 6 * 5
+= 30 attempts before Kubernetes controller manager will consider unhealthy
+status of node. After 1m it will evict all pods. The total time will be 3m
+before eviction process.
+
+Such scenario is good for medium environments as 1000 nodes will require 3000
+etcd updates per minute.
+
+> In reality, there will be from 4 to 6 node update tries. The total number of
+> of attempts will vary from 20 to 30.
+
+## Low Update and Slow reaction
+
+Let's set `--node-status-update-frequency` to **1m**.
+`--node-monitor-grace-period` will set to **5m** and `--default-not-ready-toleration-seconds` and
+``--default-unreachable-toleration-seconds`` to **60**. In this scenario, every kubelet will try to update the status
+every minute. There will be 5 * 5 = 25 attempts before unhealthy status. After 5m,
+Kubernetes controller manager will set unhealthy status. This means that pods
+will be evicted after 1m after being marked unhealthy. (6m in total).
+
+> In reality, there will be from 3 to 5 tries. The total number of attempt will
+> vary from 15 to 25.
+
+There can be different combinations such as Fast Update with Slow reaction to
+satisfy specific cases.
--- /dev/null
+# Kylin Linux
+
+Kylin Linux is supported with docker and containerd runtimes.
+
+**Note:** that Kylin Linux is not currently covered in kubespray CI and
+support for it is currently considered experimental.
+
+At present, only `Kylin Linux Advanced Server V10 (Sword)` has been adapted, which can support the deployment of aarch64 and x86_64 platforms.
+
+There are no special considerations for using Kylin Linux as the target OS
+for Kubespray deployments.
--- /dev/null
+Large deployments of K8s
+========================
+
+For a large scaled deployments, consider the following configuration changes:
+
+* Tune [ansible settings](https://docs.ansible.com/ansible/latest/intro_configuration.html)
+ for `forks` and `timeout` vars to fit large numbers of nodes being deployed.
+
+* Override containers' `foo_image_repo` vars to point to intranet registry.
+
+* Override the ``download_run_once: true`` and/or ``download_localhost: true``.
+ See download modes for details.
+
+* Adjust the `retry_stagger` global var as appropriate. It should provide sane
+ load on a delegate (the first K8s control plane node) then retrying failed
+ push or download operations.
+
+* Tune parameters for DNS related applications
+ Those are ``dns_replicas``, ``dns_cpu_limit``,
+ ``dns_cpu_requests``, ``dns_memory_limit``, ``dns_memory_requests``.
+ Please note that limits must always be greater than or equal to requests.
+
+* Tune CPU/memory limits and requests. Those are located in roles' defaults
+ and named like ``foo_memory_limit``, ``foo_memory_requests`` and
+ ``foo_cpu_limit``, ``foo_cpu_requests``. Note that 'Mi' memory units for K8s
+ will be submitted as 'M', if applied for ``docker run``, and cpu K8s units
+ will end up with the 'm' skipped for docker as well. This is required as
+ docker does not understand k8s units well.
+
+* Tune ``kubelet_status_update_frequency`` to increase reliability of kubelet.
+ ``kube_controller_node_monitor_grace_period``,
+ ``kube_controller_node_monitor_period``,
+ ``kube_apiserver_pod_eviction_not_ready_timeout_seconds`` &
+ ``kube_apiserver_pod_eviction_unreachable_timeout_seconds`` for better Kubernetes reliability.
+ Check out [Kubernetes Reliability](/docs/kubernetes-reliability.md)
+
+* Tune network prefix sizes. Those are ``kube_network_node_prefix``,
+ ``kube_service_addresses`` and ``kube_pods_subnet``.
+
+* Add calico_rr nodes if you are deploying with Calico or Canal. Nodes recover
+ from host/network interruption much quicker with calico_rr.
+
+* Check out the
+ [Inventory](/docs/getting-started.md#building-your-own-inventory)
+ section of the Getting started guide for tips on creating a large scale
+ Ansible inventory.
+
+* Override the ``etcd_events_cluster_setup: true`` store events in a separate
+ dedicated etcd instance.
+
+For example, when deploying 200 nodes, you may want to run ansible with
+``--forks=50``, ``--timeout=600`` and define the ``retry_stagger: 60``.
--- /dev/null
+# Macvlan
+
+## How to use it
+
+* Enable macvlan in `group_vars/k8s_cluster/k8s_cluster.yml`
+
+```yml
+...
+kube_network_plugin: macvlan
+...
+```
+
+* Adjust the `macvlan_interface` in `group_vars/k8s_cluster/k8s-net-macvlan.yml` or by host in the `host.yml` file:
+
+```yml
+all:
+ hosts:
+ node1:
+ ip: 10.2.2.1
+ access_ip: 10.2.2.1
+ ansible_host: 10.2.2.1
+ macvlan_interface: ens5
+```
+
+## Issue encountered
+
+* Service DNS
+
+reply from unexpected source:
+
+add `kube_proxy_masquerade_all: true` in `group_vars/all/all.yml`
+
+* Disable nodelocaldns
+
+The nodelocal dns IP is not reacheable.
+
+Disable it in `sample/group_vars/k8s_cluster/k8s_cluster.yml`
+
+```yml
+enable_nodelocaldns: false
+```
--- /dev/null
+# MetalLB
+
+MetalLB hooks into your Kubernetes cluster, and provides a network load-balancer implementation.
+It allows you to create Kubernetes services of type "LoadBalancer" in clusters that don't run on a cloud provider, and thus cannot simply hook into 3rd party products to provide load-balancers.
+The default operationg mode of MetalLB is in ["Layer2"](https://metallb.universe.tf/concepts/layer2/) but it can also operate in ["BGP"](https://metallb.universe.tf/concepts/bgp/) mode.
+
+## Prerequisites
+
+You have to configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface for MetalLB to work.
+
+```yaml
+kube_proxy_strict_arp: true
+```
+
+## Install
+
+You have to explicitly enable the MetalLB extension and set an IP address range from which to allocate LoadBalancer IPs.
+
+```yaml
+metallb_enabled: true
+metallb_speaker_enabled: true
+metallb_avoid_buggy_ips: true
+metallb_ip_range:
+ - 10.5.0.0/16
+```
+
+By default only the MetalLB BGP speaker is allowed to run on control plane nodes. If you have a single node cluster or a cluster where control plane are also worker nodes you may need to enable tolerations for the MetalLB controller:
+
+```yaml
+metallb_controller_tolerations:
+ - key: "node-role.kubernetes.io/master"
+ operator: "Equal"
+ value: ""
+ effect: "NoSchedule"
+ - key: "node-role.kubernetes.io/control-plane"
+ operator: "Equal"
+ value: ""
+ effect: "NoSchedule"
+```
+
+## BGP Mode
+
+When operating in BGP Mode MetalLB needs to have defined upstream peers:
+
+```yaml
+metallb_protocol: bgp
+metallb_ip_range:
+ - 10.5.0.0/16
+metallb_peers:
+ - peer_address: 192.0.2.1
+ peer_asn: 64512
+ my_asn: 4200000000
+ - peer_address: 192.0.2.2
+ peer_asn: 64513
+ my_asn: 4200000000
+```
+
+Some upstream BGP peers may require password authentication:
+
+```yaml
+metallb_protocol: bgp
+metallb_ip_range:
+ - 10.5.0.0/16
+metallb_peers:
+ - peer_address: 192.0.2.1
+ peer_asn: 64512
+ my_asn: 4200000000
+ password: "changeme"
+```
+
+When using calico >= 3.18 you can replace MetalLB speaker by calico Service LoadBalancer IP advertisement.
+See [calico service IPs advertisement documentation](https://docs.projectcalico.org/archive/v3.18/networking/advertise-service-ips#advertise-service-load-balancer-ip-addresses).
+In this scenarion you should disable the MetalLB speaker and configure the `calico_advertise_service_loadbalancer_ips` to match your `metallb_ip_range`
+
+```yaml
+metallb_speaker_enabled: false
+metallb_avoid_buggy_ips: true
+metallb_ip_range:
+ - 10.5.0.0/16
+calico_advertise_service_loadbalancer_ips: "{{ metallb_ip_range }}"
+```
+
+If you have additional loadbalancer IP pool in `metallb_additional_address_pools` , ensure to add them to the list.
+
+```yaml
+metallb_speaker_enabled: false
+metallb_ip_range:
+ - 10.5.0.0/16
+metallb_additional_address_pools:
+ kube_service_pool_1:
+ ip_range:
+ - 10.6.0.0/16
+ protocol: "bgp"
+ auto_assign: false
+ avoid_buggy_ips: true
+ kube_service_pool_2:
+ ip_range:
+ - 10.10.0.0/16
+ protocol: "bgp"
+ auto_assign: false
+ avoid_buggy_ips: true
+calico_advertise_service_loadbalancer_ips:
+ - 10.5.0.0/16
+ - 10.6.0.0/16
+ - 10.10.0.0/16
+```
--- /dev/null
+# Mitogen
+
+*Warning:* Mitogen support is now deprecated in kubespray due to upstream not releasing an updated version to support ansible 4.x (ansible-base 2.11.x) and above. The CI support has been stripped for mitogen and we are no longer validating any support or regressions for it. The supporting mitogen install playbook and integration documentation will be removed in a later version.
+
+[Mitogen for Ansible](https://mitogen.networkgenomics.com/ansible_detailed.html) allow a 1.25x - 7x speedup and a CPU usage reduction of at least 2x, depending on network conditions, modules executed, and time already spent by targets on useful work. Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible.
+
+## Install
+
+```ShellSession
+ansible-playbook contrib/mitogen/mitogen.yml
+```
+
+The above playbook sets the ansible `strategy` and `strategy_plugins` in `ansible.cfg` but you can also enable them if you use your own `ansible.cfg` by setting the environment varialbles:
+
+```ShellSession
+export ANSIBLE_STRATEGY=mitogen_linear
+export ANSIBLE_STRATEGY_PLUGINS=plugins/mitogen/ansible_mitogen/plugins/strategy
+```
+
+... or `ansible.cfg` setup:
+
+```ini
+[defaults]
+strategy_plugins = plugins/mitogen/ansible_mitogen/plugins/strategy
+strategy=mitogen_linear
+```
+
+## Limitation
+
+If you are experiencing problems, please see the [documentation](https://mitogen.networkgenomics.com/ansible_detailed.html#noteworthy-differences).
--- /dev/null
+# Multus
+
+Multus is a meta CNI plugin that provides multiple network interface support to
+pods. For each interface, Multus delegates CNI calls to secondary CNI plugins
+such as Calico, macvlan, etc.
+
+See [multus documentation](https://github.com/intel/multus-cni).
+
+## Multus installation
+
+Since Multus itself does not implement networking, it requires a master plugin, which is specified through the variable `kube_network_plugin`. To enable Multus an additional variable `kube_network_plugin_multus` must be set to `true`. For example,
+
+```yml
+kube_network_plugin: calico
+kube_network_plugin_multus: true
+```
+
+will install Multus and Calico and configure Multus to use Calico as the primary network plugin.
+
+## Using Multus
+
+Once Multus is installed, you can create CNI configurations (as a CRD objects) for additional networks, in this case a macvlan CNI configuration is defined. You may replace the config field with any valid CNI configuration where the CNI binary is available on the nodes.
+
+```ShellSession
+cat <<EOF | kubectl create -f -
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+ name: macvlan-conf
+spec:
+ config: '{
+ "cniVersion": "0.4.0",
+ "type": "macvlan",
+ "master": "eth0",
+ "mode": "bridge",
+ "ipam": {
+ "type": "host-local",
+ "subnet": "192.168.1.0/24",
+ "rangeStart": "192.168.1.200",
+ "rangeEnd": "192.168.1.216",
+ "routes": [
+ { "dst": "0.0.0.0/0" }
+ ],
+ "gateway": "192.168.1.1"
+ }
+ }'
+EOF
+```
+
+You may then create a pod with and additional interface that connects to this network using annotations. The annotation correlates to the name in the NetworkAttachmentDefinition above.
+
+```ShellSession
+cat <<EOF | kubectl create -f -
+apiVersion: v1
+kind: Pod
+metadata:
+ name: samplepod
+ annotations:
+ k8s.v1.cni.cncf.io/networks: macvlan-conf
+spec:
+ containers:
+ - name: samplepod
+ command: ["/bin/bash", "-c", "sleep 2000000000000"]
+ image: dougbtv/centos-network
+EOF
+```
+
+You may now inspect the pod and see that there is an additional interface configured:
+
+```ShellSession
+kubectl exec -it samplepod -- ip a
+```
+
+For more details on how to use Multus, please visit <https://github.com/intel/multus-cni>
--- /dev/null
+# Network Checker Application
+
+With the ``deploy_netchecker`` var enabled (defaults to false), Kubespray deploys a
+Network Checker Application from the 3rd side `mirantis/k8s-netchecker` docker
+images. It consists of the server and agents trying to reach the server by usual
+for Kubernetes applications network connectivity meanings. Therefore, this
+automatically verifies a pod to pod connectivity via the cluster IP and checks
+if DNS resolve is functioning as well.
+
+The checks are run by agents on a periodic basis and cover standard and host network
+pods as well. The history of performed checks may be found in the agents' application
+logs.
+
+To get the most recent and cluster-wide network connectivity report, run from
+any of the cluster nodes:
+
+```ShellSession
+curl http://localhost:31081/api/v1/connectivity_check
+```
+
+Note that Kubespray does not invoke the check but only deploys the application, if
+requested.
+
+There are related application specific variables:
+
+```yml
+netchecker_port: 31081
+agent_report_interval: 15
+netcheck_namespace: default
+```
+
+Note that the application verifies DNS resolve for FQDNs comprising only the
+combination of the ``netcheck_namespace.dns_domain`` vars, for example the
+``netchecker-service.default.svc.cluster.local``. If you want to deploy the application
+to the non default namespace, make sure as well to adjust the ``searchdomains`` var
+so the resulting search domain records to contain that namespace, like:
+
+```yml
+search: foospace.cluster.local default.cluster.local ...
+nameserver: ...
+```
--- /dev/null
+# Adding/replacing a node
+
+Modified from [comments in #3471](https://github.com/kubernetes-sigs/kubespray/issues/3471#issuecomment-530036084)
+
+## Limitation: Removal of first kube_control_plane and etcd-master
+
+Currently you can't remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node you have to:
+
+### 1) Change order of current control planes
+
+Modify the order of your control plane list by pushing your first entry to any other position. E.g. if you want to remove `node-1` of the following example:
+
+```yaml
+ children:
+ kube_control_plane:
+ hosts:
+ node-1:
+ node-2:
+ node-3:
+ kube_node:
+ hosts:
+ node-1:
+ node-2:
+ node-3:
+ etcd:
+ hosts:
+ node-1:
+ node-2:
+ node-3:
+```
+
+change your inventory to:
+
+```yaml
+ children:
+ kube_control_plane:
+ hosts:
+ node-2:
+ node-3:
+ node-1:
+ kube_node:
+ hosts:
+ node-2:
+ node-3:
+ node-1:
+ etcd:
+ hosts:
+ node-2:
+ node-3:
+ node-1:
+```
+
+## 2) Upgrade the cluster
+
+run `upgrade-cluster.yml` or `cluster.yml`. Now you are good to go on with the removal.
+
+## Adding/replacing a worker node
+
+This should be the easiest.
+
+### 1) Add new node to the inventory
+
+### 2) Run `scale.yml`
+
+You can use `--limit=NODE_NAME` to limit Kubespray to avoid disturbing other nodes in the cluster.
+
+Before using `--limit` run playbook `facts.yml` without the limit to refresh facts cache for all nodes.
+
+### 3) Remove an old node with remove-node.yml
+
+With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
+
+If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars: `-e node=NODE_NAME -e reset_nodes=false -e allow_ungraceful_removal=true`.
+Use this flag even when you remove other types of nodes like a control plane or etcd nodes.
+
+### 4) Remove the node from the inventory
+
+That's it.
+
+## Adding/replacing a control plane node
+
+### 1) Run `cluster.yml`
+
+Append the new host to the inventory and run `cluster.yml`. You can NOT use `scale.yml` for that.
+
+### 2) Restart kube-system/nginx-proxy
+
+In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.
+
+```sh
+# run in every host
+docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart
+```
+
+### 3) Remove old control plane nodes
+
+With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
+If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
+
+## Replacing a first control plane node
+
+### 1) Change control plane nodes order in inventory
+
+from
+
+```ini
+[kube_control_plane]
+ node-1
+ node-2
+ node-3
+```
+
+to
+
+```ini
+[kube_control_plane]
+ node-2
+ node-3
+ node-1
+```
+
+### 2) Remove old first control plane node from cluster
+
+With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=node-1` to the playbook to limit the execution to the node being removed.
+If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
+
+### 3) Edit cluster-info configmap in kube-public namespace
+
+`kubectl edit cm -n kube-public cluster-info`
+
+Change ip of old kube_control_plane node with ip of live kube_control_plane node (`server` field). Also, update `certificate-authority-data` field if you changed certs.
+
+### 4) Add new control plane node
+
+Update inventory (if needed)
+
+Run `cluster.yml` with `--limit=kube_control_plane`
+
+## Adding an etcd node
+
+You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replace or scale up operation. Either add two new nodes or remove an old one.
+
+### 1) Add the new node running cluster.yml
+
+Update the inventory and run `cluster.yml` passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`.
+If the node you want to add as an etcd node is already a worker or control plane node in your cluster, you have to remove him first using `remove-node.yml`.
+
+Run `upgrade-cluster.yml` also passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`. This is necessary to update all etcd configuration in the cluster.
+
+At this point, you will have an even number of nodes.
+Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node.
+Even so, running applications should continue to be available.
+
+If you add multiple etcd nodes with one run, you might want to append `-e etcd_retries=10` to increase the amount of retries between each etcd node join.
+Otherwise the etcd cluster might still be processing the first join and fail on subsequent nodes. `etcd_retries=10` might work to join 3 new nodes.
+
+### 2) Add the new node to apiserver config
+
+In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure the new etcd nodes are present in the apiserver command line parameter `--etcd-servers=...`.
+
+## Removing an etcd node
+
+### 1) Remove an old etcd node
+
+With the node still in the inventory, run `remove-node.yml` passing `-e node=NODE_NAME` as the name of the node that should be removed.
+If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
+
+### 2) Make sure only remaining nodes are in your inventory
+
+Remove `NODE_NAME` from your inventory file.
+
+### 3) Update kubernetes and network configuration files with the valid list of etcd members
+
+Run `cluster.yml` to regenerate the configuration files on all remaining nodes.
+
+### 4) Remove the old etcd node from apiserver config
+
+In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure only active etcd nodes are still present in the apiserver command line parameter `--etcd-servers=...`.
+
+### 5) Shutdown the old instance
+
+That's it.
--- /dev/null
+# NTP synchronization
+
+The Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems. Time synchronization is important to Kubernetes and Etcd.
+
+## Enable the NTP
+
+To start the ntpd(or chrony) service and enable it at system boot. There are related specific variables:
+
+```ShellSession
+ntp_enabled: true
+```
+
+The NTP service would be enabled and sync time automatically.
+
+## Custimize the NTP configure file
+
+In the Air-Gap environment, the node cannot access the NTP server by internet. So the node can use the customized ntp server by configuring ntp file.
+
+```ShellSession
+ntp_enabled: true
+ntp_manage_config: true
+ntp_servers:
+ - "0.your-ntp-server.org iburst"
+ - "1.your-ntp-server.org iburst"
+ - "2.your-ntp-server.org iburst"
+ - "3.your-ntp-server.org iburst"
+```
+
+## Advanced Configure
+
+Enable `tinker panic` is useful when running NTP in a VM environment to avoiding clock drift on VMs. It only takes effect when ntp_manage_config is true.
+
+```ShellSession
+ntp_tinker_panic: true
+```
+
+Force sync time immediately by NTP after the ntp installed, which is useful in newly installed system.
+
+```ShellSession
+ntp_force_sync_immediately: true
+```
--- /dev/null
+# Offline environment
+
+In case your servers don't have access to internet (for example when deploying on premises with security constraints), you need to setup:
+
+* a HTTP reverse proxy/cache/mirror to serve some static files (zips and binaries)
+* an internal Yum/Deb repository for OS packages
+* an internal container image registry that need to be populated with all container images used by Kubespray. Exhaustive list depends on your setup
+* [Optional] an internal PyPi server for kubespray python packages (only required if your OS doesn't provide all python packages/versions listed in `requirements.txt`)
+* [Optional] an internal Helm registry (only required if `helm_enabled=true`)
+
+## Configure Inventory
+
+Once all artifacts are accessible from your internal network, **adjust** the following variables in [your inventory](/inventory/sample/group_vars/all/offline.yml) to match your environment:
+
+```yaml
+# Registry overrides
+kube_image_repo: "{{ registry_host }}"
+gcr_image_repo: "{{ registry_host }}"
+docker_image_repo: "{{ registry_host }}"
+quay_image_repo: "{{ registry_host }}"
+
+kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
+kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
+kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
+# etcd is optional if you **DON'T** use etcd_deployment=host
+etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
+cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
+crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+# If using Calico
+calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
+# If using Calico with kdd
+calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_version }}.tar.gz"
+# Containerd
+containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
+runc_download_url: "{{ files_repo }}/runc.{{ image_arch }}"
+nerdctl_download_url: "{{ files_repo }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+# Insecure registries for containerd
+containerd_insecure_registries:
+ - "{{ registry_host }}"
+
+# CentOS/Redhat/AlmaLinux/Rocky Linux
+## Docker / Containerd
+docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
+docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+# Fedora
+## Docker
+docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
+docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+## Containerd
+containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
+containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+# Debian
+## Docker
+docker_debian_repo_base_url: "{{ debian_repo }}/docker-ce"
+docker_debian_repo_gpgkey: "{{ debian_repo }}/docker-ce/gpg"
+## Containerd
+containerd_debian_repo_base_url: "{{ ubuntu_repo }}/containerd"
+containerd_debian_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+containerd_debian_repo_repokey: 'YOURREPOKEY'
+
+# Ubuntu
+## Docker
+docker_ubuntu_repo_base_url: "{{ ubuntu_repo }}/docker-ce"
+docker_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/docker-ce/gpg"
+## Containerd
+containerd_ubuntu_repo_base_url: "{{ ubuntu_repo }}/containerd"
+containerd_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+containerd_ubuntu_repo_repokey: 'YOURREPOKEY'
+```
+
+For the OS specific settings, just define the one matching your OS.
+If you use the settings like the one above, you'll need to define in your inventory the following variables:
+
+* `registry_host`: Container image registry. If you _don't_ use the same repository path for the container images that the ones defined in [Download's role defaults](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/download/defaults/main.yml), you need to override the `*_image_repo` for these container images. If you want to make your life easier, use the same repository path, you won't have to override anything else.
+* `files_repo`: HTTP webserver or reverse proxy that is able to serve the files listed above. Path is not important, you can store them anywhere as long as it's accessible by kubespray. It's recommended to use `*_version` in the path so that you don't need to modify this setting everytime kubespray upgrades one of these components.
+* `yum_repo`/`debian_repo`/`ubuntu_repo`: OS package repository depending of your OS, should point to your internal repository. Adjust the path accordingly.
+
+## Install Kubespray Python Packages
+
+### Recommended way: Kubespray Container Image
+
+The easiest way is to use [kubespray container image](https://quay.io/kubespray/kubespray) as all the required packages are baked in the image.
+Just copy the container image in your private container image registry and you are all set!
+
+### Manual installation
+
+Look at the `requirements.txt` file and check if your OS provides all packages out-of-the-box (Using the OS package manager). For those missing, you need to either use a proxy that has Internet access (typically from a DMZ) or setup a PyPi server in your network that will host these packages.
+
+If you're using a HTTP(S) proxy to download your python packages:
+
+```bash
+sudo pip install --proxy=https://[username:password@]proxyserver:port -r requirements.txt
+```
+
+When using an internal PyPi server:
+
+```bash
+# If you host all required packages
+pip install -i https://pypiserver/pypi -r requirements.txt
+
+# If you only need the ones missing from the OS package manager
+pip install -i https://pypiserver/pypi package_you_miss
+```
+
+## Run Kubespray as usual
+
+Once all artifacts are in place and your inventory properly set up, you can run kubespray with the regular `cluster.yaml` command:
+
+```bash
+ansible-playbook -i inventory/my_airgap_cluster/hosts.yaml -b cluster.yml
+```
+
+If you use [Kubespray Container Image](#recommended-way:-kubespray-container-image), you can mount your inventory inside the container:
+
+```bash
+docker run --rm -it -v path_to_inventory/my_airgap_cluster:inventory/my_airgap_cluster myprivateregisry.com/kubespray/kubespray:v2.14.0 ansible-playbook -i inventory/my_airgap_cluster/hosts.yaml -b cluster.yml
+```
--- /dev/null
+
+# OpenStack
+
+## Known compatible public clouds
+
+Kubespray has been tested on a number of OpenStack Public Clouds including (in alphabetical order):
+
+- [Auro](https://auro.io/)
+- [Betacloud](https://www.betacloud.io/)
+- [CityCloud](https://www.citycloud.com/)
+- [DreamHost](https://www.dreamhost.com/cloud/computing/)
+- [ELASTX](https://elastx.se/)
+- [EnterCloudSuite](https://www.entercloudsuite.com/)
+- [FugaCloud](https://fuga.cloud/)
+- [Infomaniak](https://infomaniak.com)
+- [Open Telekom Cloud](https://cloud.telekom.de/) : requires to set the variable `wait_for_floatingip = "true"` in your cluster.tfvars
+- [OVHcloud](https://www.ovhcloud.com/)
+- [Rackspace](https://www.rackspace.com/)
+- [Ultimum](https://ultimum.io/)
+- [VexxHost](https://vexxhost.com/)
+- [Zetta](https://www.zetta.io/)
+
+## The in-tree cloud provider
+
+To deploy Kubespray on [OpenStack](https://www.openstack.org/) uncomment the `cloud_provider` option in `group_vars/all/all.yml` and set it to `openstack`.
+
+After that make sure to source in your OpenStack credentials like you would do when using `nova-client` or `neutron-client` by using `source path/to/your/openstack-rc` or `. path/to/your/openstack-rc`.
+
+For those who prefer to pass the OpenStack CA certificate as a string, one can
+base64 encode the cacert file and store it in the variable `openstack_cacert`.
+
+The next step is to make sure the hostnames in your `inventory` file are identical to your instance names in OpenStack.
+Otherwise [cinder](https://wiki.openstack.org/wiki/Cinder) won't work as expected.
+
+Unless you are using calico or kube-router you can now run the playbook.
+
+**Additional step needed when using calico or kube-router:**
+
+Being L3 CNI, calico and kube-router do not encapsulate all packages with the hosts' ip addresses. Instead the packets will be routed with the PODs ip addresses directly.
+
+OpenStack will filter and drop all packets from ips it does not know to prevent spoofing.
+
+In order to make L3 CNIs work on OpenStack you will need to tell OpenStack to allow pods packets by allowing the network they use.
+
+First you will need the ids of your OpenStack instances that will run kubernetes:
+
+ ```bash
+ openstack server list --project YOUR_PROJECT
+ +--------------------------------------+--------+----------------------------------+--------+-------------+
+ | ID | Name | Tenant ID | Status | Power State |
+ +--------------------------------------+--------+----------------------------------+--------+-------------+
+ | e1f48aad-df96-4bce-bf61-62ae12bf3f95 | k8s-1 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running |
+ | 725cd548-6ea3-426b-baaa-e7306d3c8052 | k8s-2 | fba478440cb2444a9e5cf03717eb5d6f | ACTIVE | Running |
+ ```
+
+Then you can use the instance ids to find the connected [neutron](https://wiki.openstack.org/wiki/Neutron) ports (though they are now configured through using OpenStack):
+
+ ```bash
+ openstack port list -c id -c device_id --project YOUR_PROJECT
+ +--------------------------------------+--------------------------------------+
+ | id | device_id |
+ +--------------------------------------+--------------------------------------+
+ | 5662a4e0-e646-47f0-bf88-d80fbd2d99ef | e1f48aad-df96-4bce-bf61-62ae12bf3f95 |
+ | e5ae2045-a1e1-4e99-9aac-4353889449a7 | 725cd548-6ea3-426b-baaa-e7306d3c8052 |
+ ```
+
+Given the port ids on the left, you can set the two `allowed-address`(es) in OpenStack. Note that you have to allow both `kube_service_addresses` (default `10.233.0.0/18`) and `kube_pods_subnet` (default `10.233.64.0/18`.)
+
+ ```bash
+ # allow kube_service_addresses and kube_pods_subnet network
+ openstack port set 5662a4e0-e646-47f0-bf88-d80fbd2d99ef --allowed-address ip-address=10.233.0.0/18 --allowed-address ip-address=10.233.64.0/18
+ openstack port set e5ae2045-a1e1-4e99-9aac-4353889449a7 --allowed-address ip-address=10.233.0.0/18 --allowed-address ip-address=10.233.64.0/18
+ ```
+
+If all the VMs in the tenant correspond to Kubespray deployment, you can "sweep run" above with:
+
+ ```bash
+ openstack port list --device-owner=compute:nova -c ID -f value | xargs -tI@ openstack port set @ --allowed-address ip-address=10.233.0.0/18 --allowed-address ip-address=10.233.64.0/18
+ ```
+
+Now you can finally run the playbook.
+
+## The external cloud provider
+
+The in-tree cloud provider is deprecated and will be removed in a future version of Kubernetes. The target release for removing all remaining in-tree cloud providers is set to 1.21.
+
+The new cloud provider is configured to have Octavia by default in Kubespray.
+
+- Enable the new external cloud provider in `group_vars/all/all.yml`:
+
+ ```yaml
+ cloud_provider: external
+ external_cloud_provider: openstack
+ ```
+
+- Enable Cinder CSI in `group_vars/all/openstack.yml`:
+
+ ```yaml
+ cinder_csi_enabled: true
+ ```
+
+- Enable topology support (optional), if your openstack provider has custom Zone names you can override the default "nova" zone by setting the variable `cinder_topology_zones`
+
+ ```yaml
+ cinder_topology: true
+ ```
+
+- Enabling `cinder_csi_ignore_volume_az: true`, ignores volumeAZ and schedules on any of the available node AZ.
+
+ ```yaml
+ cinder_csi_ignore_volume_az: true
+ ```
+
+- If you are using OpenStack loadbalancer(s) replace the `openstack_lbaas_subnet_id` with the new `external_openstack_lbaas_subnet_id`. **Note** The new cloud provider is using Octavia instead of Neutron LBaaS by default!
+- Enable 3 feature gates to allow migration of all volumes and storage classes (if you have any feature gates already set just add the 3 listed below):
+
+ ```yaml
+ kube_feature_gates:
+ - CSIMigration=true
+ - CSIMigrationOpenStack=true
+ - ExpandCSIVolumes=true
+ ```
+
+- If you are in a case of a multi-nic OpenStack VMs (see [kubernetes/cloud-provider-openstack#407](https://github.com/kubernetes/cloud-provider-openstack/issues/407) and [#6083](https://github.com/kubernetes-sigs/kubespray/issues/6083) for explanation), you should override the default OpenStack networking configuration:
+
+ ```yaml
+ external_openstack_network_ipv6_disabled: false
+ external_openstack_network_internal_networks: []
+ external_openstack_network_public_networks: []
+ ```
+
+- You can override the default OpenStack metadata configuration (see [#6338](https://github.com/kubernetes-sigs/kubespray/issues/6338) for explanation):
+
+ ```yaml
+ external_openstack_metadata_search_order: "configDrive,metadataService"
+ ```
+
+- Available variables for configuring lbaas:
+
+ ```yaml
+ external_openstack_lbaas_create_monitor: false
+ external_openstack_lbaas_monitor_delay: "1m"
+ external_openstack_lbaas_monitor_timeout: "30s"
+ external_openstack_lbaas_monitor_max_retries: "3"
+ external_openstack_lbaas_provider: octavia
+ external_openstack_lbaas_use_octavia: false
+ external_openstack_lbaas_network_id: "Neutron network ID to create LBaaS VIP"
+ external_openstack_lbaas_subnet_id: "Neutron subnet ID to create LBaaS VIP"
+ external_openstack_lbaas_floating_network_id: "Neutron network ID to get floating IP from"
+ external_openstack_lbaas_floating_subnet_id: "Neutron subnet ID to get floating IP from"
+ external_openstack_lbaas_method: "ROUND_ROBIN"
+ external_openstack_lbaas_manage_security_groups: false
+ external_openstack_lbaas_internal_lb: false
+
+ ```
+
+- Run `source path/to/your/openstack-rc` to read your OpenStack credentials like `OS_AUTH_URL`, `OS_USERNAME`, `OS_PASSWORD`, etc. Those variables are used for accessing OpenStack from the external cloud provider.
+- Run the `cluster.yml` playbook
--- /dev/null
+# openSUSE Leap 15.3 and Tumbleweed
+
+openSUSE Leap installation Notes:
+
+- Install Ansible
+
+ ```ShellSession
+ sudo zypper ref
+ sudo zypper -n install ansible
+
+ ```
+
+- Install Jinja2 and Python-Netaddr
+
+ ```sudo zypper -n install python-Jinja2 python-netaddr```
+
+Now you can continue with [Preparing your deployment](getting-started.md#starting-custom-deployment)
--- /dev/null
+# Setting up Environment Proxy
+
+If you set http and https proxy, all nodes and loadbalancer will be excluded from proxy with generating no_proxy variable in `roles/kubespray-defaults/tasks/no_proxy.yml`, if you have additional resources for exclude add them to `additional_no_proxy` variable. If you want fully override your `no_proxy` setting, then fill in just `no_proxy` and no nodes or loadbalancer addresses will be added to no_proxy.
+
+## Set proxy for http and https
+
+ `http_proxy:"http://example.proxy.tld:port"`
+ `https_proxy:"http://example.proxy.tld:port"`
+
+## Set default no_proxy (this will override default no_proxy generation)
+
+`no_proxy: "node1,node1_ip,node2,node2_ip...additional_host"`
+
+## Set additional addresses to default no_proxy (all cluster nodes and loadbalancer)
+
+`additional_no_proxy: "additional_host1,additional_host2"`
+
+## Exclude workers from no_proxy
+
+Since workers are included in the no_proxy variable, by default, docker engine will be restarted on all nodes (all
+pods will restart) when adding or removing workers. To override this behaviour by only including control plane nodes in the
+no_proxy variable, set:
+`no_proxy_exclude_workers: true`
--- /dev/null
+
+# Recovering the control plane
+
+To recover from broken nodes in the control plane use the "recover\-control\-plane.yml" playbook.
+
+* Backup what you can
+* Provision new nodes to replace the broken ones
+* Place the surviving nodes of the control plane first in the "etcd" and "kube\_control\_plane" groups
+* Add the new nodes below the surviving control plane nodes in the "etcd" and "kube\_control\_plane" groups
+
+Examples of what broken means in this context:
+
+* One or more bare metal node(s) suffer from unrecoverable hardware failure
+* One or more node(s) fail during patching or upgrading
+* Etcd database corruption
+* Other node related failures leaving your control plane degraded or nonfunctional
+
+__Note that you need at least one functional node to be able to recover using this method.__
+
+## Runbook
+
+* Move any broken etcd nodes into the "broken\_etcd" group, make sure the "etcd\_member\_name" variable is set.
+* Move any broken control plane nodes into the "broken\_kube\_control\_plane" group.
+
+Then run the playbook with ```--limit etcd,kube_control_plane``` and increase the number of ETCD retries by setting ```-e etcd_retries=10``` or something even larger. The amount of retries required is difficult to predict.
+
+When finished you should have a fully working control plane again.
+
+## Recover from lost quorum
+
+The playbook attempts to figure out it the etcd quorum is intact. If quorum is lost it will attempt to take a snapshot from the first node in the "etcd" group and restore from that. If you would like to restore from an alternate snapshot set the path to that snapshot in the "etcd\_snapshot" variable.
+
+```-e etcd_snapshot=/tmp/etcd_snapshot```
+
+## Caveats
+
+* The playbook has only been tested with fairly small etcd databases.
+* If your new control plane nodes have new ip addresses you may have to change settings in various places.
+* There may be disruptions while running the playbook.
+* There are absolutely no guarantees.
+
+If possible try to break a cluster in the same way that your target cluster is broken and test to recover that before trying on the real target cluster.
--- /dev/null
+# Red Hat Enterprise Linux (RHEL)
+
+## RHEL Support Subscription Registration
+
+In order to install packages via yum or dnf, RHEL 7/8 hosts are required to be registered for a valid Red Hat support subscription.
+
+You can apply for a 1-year Development support subscription by creating a [Red Hat Developers](https://developers.redhat.com/) account. Be aware though that as the Red Hat Developers subscription is limited to only 1 year, it should not be used to register RHEL 7/8 hosts provisioned in Production environments.
+
+Once you have a Red Hat support account, simply add the credentials to the Ansible inventory parameters `rh_subscription_username` and `rh_subscription_password` prior to deploying Kubespray. If your company has a Corporate Red Hat support account, then obtain an **Organization ID** and **Activation Key**, and add these to the Ansible inventory parameters `rh_subscription_org_id` and `rh_subscription_activation_key` instead of using your Red Hat support account credentials.
+
+```ini
+rh_subscription_username: ""
+rh_subscription_password: ""
+# rh_subscription_org_id: ""
+# rh_subscription_activation_key: ""
+```
+
+Either the Red Hat support account username/password, or Organization ID/Activation Key combination must be specified in the Ansible inventory in order for the Red Hat subscription registration to complete successfully during the deployment of Kubespray.
+
+Update the Ansible inventory parameters `rh_subscription_usage`, `rh_subscription_role` and `rh_subscription_sla` if necessary to suit your specific requirements.
+
+```ini
+rh_subscription_usage: "Development"
+rh_subscription_role: "Red Hat Enterprise Server"
+rh_subscription_sla: "Self-Support"
+```
+
+If the RHEL 7/8 hosts are already registered to a valid Red Hat support subscription via an alternative configuration management approach prior to the deployment of Kubespray, the successful RHEL `subscription-manager` status check will simply result in the RHEL subscription registration tasks being skipped.
+
+## RHEL 8
+
+RHEL 8 ships only with iptables-nft (ie without iptables-legacy)
+The only tested configuration for now is using Calico CNI
+You need to use K8S 1.17+ and to add `calico_iptables_backend: "NFT"` to your configuration
+
+If you have containers that are using iptables in the host network namespace (`hostNetwork=true`),
+you need to ensure they are using iptables-nft.
+An example how k8s do the autodetection can be found [in this PR](https://github.com/kubernetes/kubernetes/pull/82966)
--- /dev/null
+# Kubespray's roadmap
+
+We are tracking the evolution towards Kubespray 3.0 in [#6400](https://github.com/kubernetes-sigs/kubespray/issues/6400) as well as in other open issue in our [github issues](https://github.com/kubernetes-sigs/kubespray/issues/) section.
--- /dev/null
+# Setting up your first cluster with Kubespray
+
+This tutorial walks you through the detailed steps for setting up Kubernetes
+with [Kubespray](https://kubespray.io/).
+
+The guide is inspired on the tutorial [Kubernetes The Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way), with the
+difference that here we want to showcase how to spin up a Kubernetes cluster
+in a more managed fashion with Kubespray.
+
+## Target Audience
+
+The target audience for this tutorial is someone looking for a
+hands-on guide to get started with Kubespray.
+
+## Cluster Details
+
+* [kubespray](https://github.com/kubernetes-sigs/kubespray) v2.17.x
+* [kubernetes](https://github.com/kubernetes/kubernetes) v1.17.9
+
+## Prerequisites
+
+* Google Cloud Platform: This tutorial leverages the [Google Cloud Platform](https://cloud.google.com/) to streamline provisioning of the compute infrastructure required to bootstrap a Kubernetes cluster from the ground up. [Sign up](https://cloud.google.com/free/) for $300 in free credits.
+* Google Cloud Platform SDK: Follow the Google Cloud SDK [documentation](https://cloud.google.com/sdk/) to install and configure the `gcloud` command
+ line utility. Make sure to set a default compute region and compute zone.
+* The [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) command line utility is used to interact with the Kubernetes
+ API Server.
+* Linux or Mac environment with Python 3
+
+## Provisioning Compute Resources
+
+Kubernetes requires a set of machines to host the Kubernetes control plane and the worker nodes where containers are ultimately run. In this lab you will provision the compute resources required for running a secure and highly available Kubernetes cluster across a single [compute zone](https://cloud.google.com/compute/docs/regions-zones/regions-zones).
+
+### Networking
+
+The Kubernetes [networking model](https://kubernetes.io/docs/concepts/cluster-administration/networking/#kubernetes-model) assumes a flat network in which containers and nodes can communicate with each other. In cases where this is not desired [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) can limit how groups of containers are allowed to communicate with each other and external network endpoints.
+
+> Setting up network policies is out of scope for this tutorial.
+
+#### Virtual Private Cloud Network
+
+In this section a dedicated [Virtual Private Cloud](https://cloud.google.com/compute/docs/networks-and-firewalls#networks) (VPC) network will be setup to host the Kubernetes cluster.
+
+Create the `kubernetes-the-kubespray-way` custom VPC network:
+
+```ShellSession
+gcloud compute networks create kubernetes-the-kubespray-way --subnet-mode custom
+```
+
+A [subnet](https://cloud.google.com/compute/docs/vpc/#vpc_networks_and_subnets) must be provisioned with an IP address range large enough to assign a private IP address to each node in the Kubernetes cluster.
+
+Create the `kubernetes` subnet in the `kubernetes-the-kubespray-way` VPC network:
+
+```ShellSession
+gcloud compute networks subnets create kubernetes \
+ --network kubernetes-the-kubespray-way \
+ --range 10.240.0.0/24
+ ```
+
+> The `10.240.0.0/24` IP address range can host up to 254 compute instances.
+
+#### Firewall Rules
+
+Create a firewall rule that allows internal communication across all protocols.
+It is important to note that the vxlan protocol has to be allowed in order for
+the calico (see later) networking plugin to work.
+
+```ShellSession
+gcloud compute firewall-rules create kubernetes-the-kubespray-way-allow-internal \
+ --allow tcp,udp,icmp,vxlan \
+ --network kubernetes-the-kubespray-way \
+ --source-ranges 10.240.0.0/24
+```
+
+Create a firewall rule that allows external SSH, ICMP, and HTTPS:
+
+```ShellSession
+gcloud compute firewall-rules create kubernetes-the-kubespray-way-allow-external \
+ --allow tcp:80,tcp:6443,tcp:443,tcp:22,icmp \
+ --network kubernetes-the-kubespray-way \
+ --source-ranges 0.0.0.0/0
+```
+
+It is not feasible to restrict the firewall to a specific IP address from
+where you are accessing the cluster as the nodes also communicate over the public internet and would otherwise run into
+this firewall. Technically you could limit the firewall to the (fixed) IP
+addresses of the cluster nodes and the remote IP addresses for accessing the
+cluster.
+
+### Compute Instances
+
+The compute instances in this lab will be provisioned using [Ubuntu Server](https://www.ubuntu.com/server) 18.04.
+Each compute instance will be provisioned with a fixed private IP address and
+ a public IP address (that can be fixed - see [guide](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address)).
+Using fixed public IP addresses has the advantage that our cluster node
+configuration does not need to be updated with new public IP addresses every
+time the machines are shut down and later on restarted.
+
+Create three compute instances which will host the Kubernetes control plane:
+
+```ShellSession
+for i in 0 1 2; do
+ gcloud compute instances create controller-${i} \
+ --async \
+ --boot-disk-size 200GB \
+ --can-ip-forward \
+ --image-family ubuntu-1804-lts \
+ --image-project ubuntu-os-cloud \
+ --machine-type e2-standard-2 \
+ --private-network-ip 10.240.0.1${i} \
+ --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
+ --subnet kubernetes \
+ --tags kubernetes-the-kubespray-way,controller
+done
+```
+
+> Do not forget to fix the IP addresses if you plan on re-using the cluster
+after temporarily shutting down the VMs - see [guide](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address)
+
+Create three compute instances which will host the Kubernetes worker nodes:
+
+```ShellSession
+for i in 0 1 2; do
+ gcloud compute instances create worker-${i} \
+ --async \
+ --boot-disk-size 200GB \
+ --can-ip-forward \
+ --image-family ubuntu-1804-lts \
+ --image-project ubuntu-os-cloud \
+ --machine-type e2-standard-2 \
+ --private-network-ip 10.240.0.2${i} \
+ --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
+ --subnet kubernetes \
+ --tags kubernetes-the-kubespray-way,worker
+done
+```
+
+> Do not forget to fix the IP addresses if you plan on re-using the cluster
+after temporarily shutting down the VMs - see [guide](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address)
+
+List the compute instances in your default compute zone:
+
+```ShellSession
+gcloud compute instances list --filter="tags.items=kubernetes-the-kubespray-way"
+```
+
+> Output
+
+```ShellSession
+NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
+controller-0 us-west1-c e2-standard-2 10.240.0.10 XX.XX.XX.XXX RUNNING
+controller-1 us-west1-c e2-standard-2 10.240.0.11 XX.XXX.XXX.XX RUNNING
+controller-2 us-west1-c e2-standard-2 10.240.0.12 XX.XXX.XX.XXX RUNNING
+worker-0 us-west1-c e2-standard-2 10.240.0.20 XX.XX.XXX.XXX RUNNING
+worker-1 us-west1-c e2-standard-2 10.240.0.21 XX.XX.XX.XXX RUNNING
+worker-2 us-west1-c e2-standard-2 10.240.0.22 XX.XXX.XX.XX RUNNING
+```
+
+### Configuring SSH Access
+
+Kubespray is relying on SSH to configure the controller and worker instances.
+
+Test SSH access to the `controller-0` compute instance:
+
+```ShellSession
+IP_CONTROLLER_0=$(gcloud compute instances list --filter="tags.items=kubernetes-the-kubespray-way AND name:controller-0" --format="value(EXTERNAL_IP)")
+USERNAME=$(whoami)
+ssh $USERNAME@$IP_CONTROLLER_0
+```
+
+If this is your first time connecting to a compute instance SSH keys will be
+generated for you. In this case you will need to enter a passphrase at the
+prompt to continue.
+
+> If you get a 'Remote host identification changed!' warning, you probably
+already connected to that IP address in the past with another host key. You
+can remove the old host key by running `ssh-keygen -R $IP_CONTROLLER_0`
+
+Please repeat this procedure for all the controller and worker nodes, to
+ensure that SSH access is properly functioning for all nodes.
+
+## Set-up Kubespray
+
+The following set of instruction is based on the [Quick Start](https://github.com/kubernetes-sigs/kubespray) but slightly altered for our
+set-up.
+
+As Ansible is a python application, we will create a fresh virtual
+environment to install the dependencies for the Kubespray playbook:
+
+```ShellSession
+python3 -m venv venv
+source venv/bin/activate
+```
+
+Next, we will git clone the Kubespray code into our working directory:
+
+```ShellSession
+git clone https://github.com/kubernetes-sigs/kubespray.git
+cd kubespray
+git checkout release-2.17
+```
+
+Now we need to install the dependencies for Ansible to run the Kubespray
+playbook:
+
+```ShellSession
+pip install -r requirements.txt
+```
+
+Copy ``inventory/sample`` as ``inventory/mycluster``:
+
+```ShellSession
+cp -rfp inventory/sample inventory/mycluster
+```
+
+Update Ansible inventory file with inventory builder:
+
+```ShellSession
+declare -a IPS=($(gcloud compute instances list --filter="tags.items=kubernetes-the-kubespray-way" --format="value(EXTERNAL_IP)" | tr '\n' ' '))
+CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+```
+
+Open the generated `inventory/mycluster/hosts.yaml` file and adjust it so
+that controller-0, controller-1 and controller-2 are control plane nodes and
+worker-0, worker-1 and worker-2 are worker nodes. Also update the `ip` to the respective local VPC IP and
+remove the `access_ip`.
+
+The main configuration for the cluster is stored in
+`inventory/mycluster/group_vars/k8s_cluster/k8s_cluster.yml`. In this file we
+ will update the `supplementary_addresses_in_ssl_keys` with a list of the IP
+ addresses of the controller nodes. In that way we can access the
+ kubernetes API server as an administrator from outside the VPC network. You
+ can also see that the `kube_network_plugin` is by default set to 'calico'.
+ If you set this to 'cloud', it did not work on GCP at the time of testing.
+
+Kubespray also offers to easily enable popular kubernetes add-ons. You can
+modify the
+list of add-ons in `inventory/mycluster/group_vars/k8s_cluster/addons.yml`.
+Let's enable the metrics server as this is a crucial monitoring element for
+the kubernetes cluster, just change the 'false' to 'true' for
+`metrics_server_enabled`.
+
+Now we will deploy the configuration:
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.yaml -u $USERNAME -b -v --private-key=~/.ssh/id_rsa cluster.yml
+```
+
+Ansible will now execute the playbook, this can take up to 20 minutes.
+
+## Access the kubernetes cluster
+
+We will leverage a kubeconfig file from one of the controller nodes to access
+ the cluster as administrator from our local workstation.
+
+> In this simplified set-up, we did not include a load balancer that usually
+ sits on top of the
+three controller nodes for a high available API server endpoint. In this
+ simplified tutorial we connect directly to one of the three
+ controllers.
+
+First, we need to edit the permission of the kubeconfig file on one of the
+controller nodes:
+
+```ShellSession
+ssh $USERNAME@$IP_CONTROLLER_0
+USERNAME=$(whoami)
+sudo chown -R $USERNAME:$USERNAME /etc/kubernetes/admin.conf
+exit
+```
+
+Now we will copy over the kubeconfig file:
+
+```ShellSession
+scp $USERNAME@$IP_CONTROLLER_0:/etc/kubernetes/admin.conf kubespray-do.conf
+```
+
+This kubeconfig file uses the internal IP address of the controller node to
+access the API server. This kubeconfig file will thus not work of from
+outside of the VPC network. We will need to change the API server IP address
+to the controller node his external IP address. The external IP address will be
+accepted in the
+TLS negotiation as we added the controllers external IP addresses in the SSL
+certificate configuration.
+Open the file and modify the server IP address from the local IP to the
+external IP address of controller-0, as stored in $IP_CONTROLLER_0.
+
+> Example
+
+```ShellSession
+apiVersion: v1
+clusters:
+- cluster:
+ certificate-authority-data: XXX
+ server: https://35.205.205.80:6443
+ name: cluster.local
+...
+```
+
+Now, we load the configuration for `kubectl`:
+
+```ShellSession
+export KUBECONFIG=$PWD/kubespray-do.conf
+```
+
+We should be all set to communicate with our cluster from our local workstation:
+
+```ShellSession
+kubectl get nodes
+```
+
+> Output
+
+```ShellSession
+NAME STATUS ROLES AGE VERSION
+controller-0 Ready master 47m v1.17.9
+controller-1 Ready master 46m v1.17.9
+controller-2 Ready master 46m v1.17.9
+worker-0 Ready <none> 45m v1.17.9
+worker-1 Ready <none> 45m v1.17.9
+worker-2 Ready <none> 45m v1.17.9
+```
+
+## Smoke tests
+
+### Metrics
+
+Verify if the metrics server addon was correctly installed and works:
+
+```ShellSession
+kubectl top nodes
+```
+
+> Output
+
+```ShellSession
+NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
+controller-0 191m 10% 1956Mi 26%
+controller-1 190m 10% 1828Mi 24%
+controller-2 182m 10% 1839Mi 24%
+worker-0 87m 4% 1265Mi 16%
+worker-1 102m 5% 1268Mi 16%
+worker-2 108m 5% 1299Mi 17%
+```
+
+Please note that metrics might not be available at first and need a couple of
+ minutes before you can actually retrieve them.
+
+### Network
+
+Let's verify if the network layer is properly functioning and pods can reach
+each other:
+
+```ShellSession
+kubectl run myshell1 -it --rm --image busybox -- sh
+hostname -i
+# launch myshell2 in separate terminal (see next code block) and ping the hostname of myshell2
+ping <hostname myshell2>
+```
+
+```ShellSession
+kubectl run myshell2 -it --rm --image busybox -- sh
+hostname -i
+ping <hostname myshell1>
+```
+
+> Output
+
+```ShellSession
+PING 10.233.108.2 (10.233.108.2): 56 data bytes
+64 bytes from 10.233.108.2: seq=0 ttl=62 time=2.876 ms
+64 bytes from 10.233.108.2: seq=1 ttl=62 time=0.398 ms
+64 bytes from 10.233.108.2: seq=2 ttl=62 time=0.378 ms
+^C
+--- 10.233.108.2 ping statistics ---
+3 packets transmitted, 3 packets received, 0% packet loss
+round-trip min/avg/max = 0.378/1.217/2.876 ms
+```
+
+### Deployments
+
+In this section you will verify the ability to create and manage [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/).
+
+Create a deployment for the [nginx](https://nginx.org/en/) web server:
+
+```ShellSession
+kubectl create deployment nginx --image=nginx
+```
+
+List the pod created by the `nginx` deployment:
+
+```ShellSession
+kubectl get pods -l app=nginx
+```
+
+> Output
+
+```ShellSession
+NAME READY STATUS RESTARTS AGE
+nginx-86c57db685-bmtt8 1/1 Running 0 18s
+```
+
+#### Port Forwarding
+
+In this section you will verify the ability to access applications remotely using [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/).
+
+Retrieve the full name of the `nginx` pod:
+
+```ShellSession
+POD_NAME=$(kubectl get pods -l app=nginx -o jsonpath="{.items[0].metadata.name}")
+```
+
+Forward port `8080` on your local machine to port `80` of the `nginx` pod:
+
+```ShellSession
+kubectl port-forward $POD_NAME 8080:80
+```
+
+> Output
+
+```ShellSession
+Forwarding from 127.0.0.1:8080 -> 80
+Forwarding from [::1]:8080 -> 80
+```
+
+In a new terminal make an HTTP request using the forwarding address:
+
+```ShellSession
+curl --head http://127.0.0.1:8080
+```
+
+> Output
+
+```ShellSession
+HTTP/1.1 200 OK
+Server: nginx/1.19.1
+Date: Thu, 13 Aug 2020 11:12:04 GMT
+Content-Type: text/html
+Content-Length: 612
+Last-Modified: Tue, 07 Jul 2020 15:52:25 GMT
+Connection: keep-alive
+ETag: "5f049a39-264"
+Accept-Ranges: bytes
+```
+
+Switch back to the previous terminal and stop the port forwarding to the `nginx` pod:
+
+```ShellSession
+Forwarding from 127.0.0.1:8080 -> 80
+Forwarding from [::1]:8080 -> 80
+Handling connection for 8080
+^C
+```
+
+#### Logs
+
+In this section you will verify the ability to [retrieve container logs](https://kubernetes.io/docs/concepts/cluster-administration/logging/).
+
+Print the `nginx` pod logs:
+
+```ShellSession
+kubectl logs $POD_NAME
+```
+
+> Output
+
+```ShellSession
+...
+127.0.0.1 - - [13/Aug/2020:11:12:04 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.1" "-"
+```
+
+#### Exec
+
+In this section you will verify the ability to [execute commands in a container](https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/#running-individual-commands-in-a-container).
+
+Print the nginx version by executing the `nginx -v` command in the `nginx` container:
+
+```ShellSession
+kubectl exec -ti $POD_NAME -- nginx -v
+```
+
+> Output
+
+```ShellSession
+nginx version: nginx/1.19.1
+```
+
+### Kubernetes services
+
+#### Expose outside of the cluster
+
+In this section you will verify the ability to expose applications using a [Service](https://kubernetes.io/docs/concepts/services-networking/service/).
+
+Expose the `nginx` deployment using a [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) service:
+
+```ShellSession
+kubectl expose deployment nginx --port 80 --type NodePort
+```
+
+> The LoadBalancer service type can not be used because your cluster is not configured with [cloud provider integration](https://kubernetes.io/docs/getting-started-guides/scratch/#cloud-provider). Setting up cloud provider integration is out of scope for this tutorial.
+
+Retrieve the node port assigned to the `nginx` service:
+
+```ShellSession
+NODE_PORT=$(kubectl get svc nginx \
+ --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
+```
+
+Create a firewall rule that allows remote access to the `nginx` node port:
+
+```ShellSession
+gcloud compute firewall-rules create kubernetes-the-kubespray-way-allow-nginx-service \
+ --allow=tcp:${NODE_PORT} \
+ --network kubernetes-the-kubespray-way
+```
+
+Retrieve the external IP address of a worker instance:
+
+```ShellSession
+EXTERNAL_IP=$(gcloud compute instances describe worker-0 \
+ --format 'value(networkInterfaces[0].accessConfigs[0].natIP)')
+```
+
+Make an HTTP request using the external IP address and the `nginx` node port:
+
+```ShellSession
+curl -I http://${EXTERNAL_IP}:${NODE_PORT}
+```
+
+> Output
+
+```ShellSession
+HTTP/1.1 200 OK
+Server: nginx/1.19.1
+Date: Thu, 13 Aug 2020 11:15:02 GMT
+Content-Type: text/html
+Content-Length: 612
+Last-Modified: Tue, 07 Jul 2020 15:52:25 GMT
+Connection: keep-alive
+ETag: "5f049a39-264"
+Accept-Ranges: bytes
+```
+
+#### Local DNS
+
+We will now also verify that kubernetes built-in DNS works across namespaces.
+Create a namespace:
+
+```ShellSession
+kubectl create namespace dev
+```
+
+Create an nginx deployment and expose it within the cluster:
+
+```ShellSession
+kubectl create deployment nginx --image=nginx -n dev
+kubectl expose deployment nginx --port 80 --type ClusterIP -n dev
+```
+
+Run a temporary container to see if we can reach the service from the default
+namespace:
+
+```ShellSession
+kubectl run curly -it --rm --image curlimages/curl:7.70.0 -- /bin/sh
+curl --head http://nginx.dev:80
+```
+
+> Output
+
+```ShellSession
+HTTP/1.1 200 OK
+Server: nginx/1.19.1
+Date: Thu, 13 Aug 2020 11:15:59 GMT
+Content-Type: text/html
+Content-Length: 612
+Last-Modified: Tue, 07 Jul 2020 15:52:25 GMT
+Connection: keep-alive
+ETag: "5f049a39-264"
+Accept-Ranges: bytes
+```
+
+Type `exit` to leave the shell.
+
+## Cleaning Up
+
+### Kubernetes resources
+
+Delete the dev namespace, the nginx deployment and service:
+
+```ShellSession
+kubectl delete namespace dev
+kubectl delete deployment nginx
+kubectl delete svc/nginx
+```
+
+### Kubernetes state
+
+Note: you can skip this step if you want to entirely remove the machines.
+
+If you want to keep the VMs and just remove the cluster state, you can simply
+ run another Ansible playbook:
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.yaml -u $USERNAME -b -v --private-key=~/.ssh/id_rsa reset.yml
+```
+
+Resetting the cluster to the VMs original state usually takes about a couple
+of minutes.
+
+### Compute instances
+
+Delete the controller and worker compute instances:
+
+```ShellSession
+gcloud -q compute instances delete \
+ controller-0 controller-1 controller-2 \
+ worker-0 worker-1 worker-2 \
+ --zone $(gcloud config get-value compute/zone)
+ ```
+
+<!-- markdownlint-disable no-duplicate-heading -->
+### Network
+<!-- markdownlint-enable no-duplicate-heading -->
+
+Delete the fixed IP addresses (assuming you named them equal to the VM names),
+if any:
+
+```ShellSession
+gcloud -q compute addresses delete controller-0 controller-1 controller-2 \
+ worker-0 worker-1 worker-2
+```
+
+Delete the `kubernetes-the-kubespray-way` firewall rules:
+
+```ShellSession
+gcloud -q compute firewall-rules delete \
+ kubernetes-the-kubespray-way-allow-nginx-service \
+ kubernetes-the-kubespray-way-allow-internal \
+ kubernetes-the-kubespray-way-allow-external
+```
+
+Delete the `kubernetes-the-kubespray-way` network VPC:
+
+```ShellSession
+gcloud -q compute networks subnets delete kubernetes
+gcloud -q compute networks delete kubernetes-the-kubespray-way
+```
--- /dev/null
+# Node Layouts
+
+There are four node layout types: `default`, `separate`, `ha`, and `scale`.
+
+`default` is a non-HA two nodes setup with one separate `kube_node`
+and the `etcd` group merged with the `kube_control_plane`.
+
+`separate` layout is when there is only node of each type, which includes
+ a kube_control_plane, kube_node, and etcd cluster member.
+
+`ha` layout consists of two etcd nodes, two control planes and a single worker node,
+with role intersection.
+
+`scale` layout can be combined with above layouts (`ha-scale`, `separate-scale`). It includes 200 fake hosts
+in the Ansible inventory. This helps test TLS certificate generation at scale
+to prevent regressions and profile certain long-running tasks. These nodes are
+never actually deployed, but certificates are generated for them.
+
+Note, the canal network plugin deploys flannel as well plus calico policy controller.
+
+## Test cases
+
+The [CI Matrix](/docs/ci.md) displays OS, Network Plugin and Container Manager tested.
+
+All tests are breakdown into 3 "stages" ("Stage" means a build step of the build pipeline) as follows:
+
+- _unit_tests_: Linting, markdown, vagrant & terraform validation etc...
+- _part1_: Molecule and AIO tests
+- _part2_: Standard tests with different layouts and OS/Runtime/Network
+- _part3_: Upgrade jobs, terraform jobs and recover control plane tests
+- _special_: Other jobs (manuals)
+
+The steps are ordered as `unit_tests->part1->part2->part3->special`.
--- /dev/null
+# Upgrading Kubernetes in Kubespray
+
+Kubespray handles upgrades the same way it handles initial deployment. That is to
+say that each component is laid down in a fixed order.
+
+You can also individually control versions of components by explicitly defining their
+versions. Here are all version vars for each component:
+
+* docker_version
+* docker_containerd_version (relevant when `container_manager` == `docker`)
+* containerd_version (relevant when `container_manager` == `containerd`)
+* kube_version
+* etcd_version
+* calico_version
+* calico_cni_version
+* weave_version
+* flannel_version
+* kubedns_version
+
+:warning: [Attempting to upgrade from an older release straight to the latest release is unsupported and likely to break something](https://github.com/kubernetes-sigs/kubespray/issues/3849#issuecomment-451386515) :warning:
+
+See [Multiple Upgrades](#multiple-upgrades) for how to upgrade from older Kubespray release to the latest release
+
+## Unsafe upgrade example
+
+If you wanted to upgrade just kube_version from v1.18.10 to v1.19.7, you could
+deploy the following way:
+
+```ShellSession
+ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.18.10 -e upgrade_cluster_setup=true
+```
+
+And then repeat with v1.19.7 as kube_version:
+
+```ShellSession
+ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.19.7 -e upgrade_cluster_setup=true
+```
+
+The var ```-e upgrade_cluster_setup=true``` is needed to be set in order to migrate the deploys of e.g kube-apiserver inside the cluster immediately which is usually only done in the graceful upgrade. (Refer to [#4139](https://github.com/kubernetes-sigs/kubespray/issues/4139) and [#4736](https://github.com/kubernetes-sigs/kubespray/issues/4736))
+
+## Graceful upgrade
+
+Kubespray also supports cordon, drain and uncordoning of nodes when performing
+a cluster upgrade. There is a separate playbook used for this purpose. It is
+important to note that upgrade-cluster.yml can only be used for upgrading an
+existing cluster. That means there must be at least 1 kube_control_plane already
+deployed.
+
+```ShellSession
+ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.19.7
+```
+
+After a successful upgrade, the Server Version should be updated:
+
+```ShellSession
+$ kubectl version
+Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
+Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
+```
+
+### Pausing the upgrade
+
+If you want to manually control the upgrade procedure, you can set some variables to pause the upgrade playbook. Pausing *before* upgrading each upgrade may be useful for inspecting pods running on that node, or performing manual actions on the node:
+
+* `upgrade_node_confirm: true` - This will pause the playbook execution prior to upgrading each node. The play will resume when manually approved by typing "yes" at the terminal.
+* `upgrade_node_pause_seconds: 60` - This will pause the playbook execution for 60 seconds prior to upgrading each node. The play will resume automatically after 60 seconds.
+
+Pausing *after* upgrading each node may be useful for rebooting the node to apply kernel updates, or testing the still-cordoned node:
+
+* `upgrade_node_post_upgrade_confirm: true` - This will pause the playbook execution after upgrading each node, but before the node is uncordoned. The play will resume when manually approved by typing "yes" at the terminal.
+* `upgrade_node_post_upgrade_pause_seconds: 60` - This will pause the playbook execution for 60 seconds after upgrading each node, but before the node is uncordoned. The play will resume automatically after 60 seconds.
+
+## Node-based upgrade
+
+If you don't want to upgrade all nodes in one run, you can use `--limit` [patterns](https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#patterns-and-ansible-playbook-flags).
+
+Before using `--limit` run playbook `facts.yml` without the limit to refresh facts cache for all nodes:
+
+```ShellSession
+ansible-playbook facts.yml -b -i inventory/sample/hosts.ini
+```
+
+After this upgrade control plane and etcd groups [#5147](https://github.com/kubernetes-sigs/kubespray/issues/5147):
+
+```ShellSession
+ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "kube_control_plane:etcd"
+```
+
+Now you can upgrade other nodes in any order and quantity:
+
+```ShellSession
+ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node4:node6:node7:node12"
+ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node5*"
+```
+
+## Multiple upgrades
+
+:warning: [Do not skip releases when upgrading--upgrade by one tag at a time.](https://github.com/kubernetes-sigs/kubespray/issues/3849#issuecomment-451386515) :warning:
+
+For instance, if you're on v2.6.0, then check out v2.7.0, run the upgrade, check out the next tag, and run the next upgrade, etc.
+
+Assuming you don't explicitly define a kubernetes version in your k8s_cluster.yml, you simply check out the next tag and run the upgrade-cluster.yml playbook
+
+* If you do define kubernetes version in your inventory (e.g. group_vars/k8s_cluster.yml) then either make sure to update it before running upgrade-cluster, or specify the new version you're upgrading to: `ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml -e kube_version=v1.11.3`
+
+ Otherwise, the upgrade will leave your cluster at the same k8s version defined in your inventory vars.
+
+The below example shows taking a cluster that was set up for v2.6.0 up to v2.10.0
+
+```ShellSession
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 1h v1.10.4
+boomer Ready master,node 42m v1.10.4
+caprica Ready master,node 42m v1.10.4
+
+$ git describe --tags
+v2.6.0
+
+$ git tag
+...
+v2.6.0
+v2.7.0
+v2.8.0
+v2.8.1
+v2.8.2
+...
+
+$ git checkout v2.7.0
+Previous HEAD position was 8b3ce6e4 bump upgrade tests to v2.5.0 commit (#3087)
+HEAD is now at 05dabb7e Fix Bionic networking restart error #3430 (#3431)
+
+# NOTE: May need to `pip3 install -r requirements.txt` when upgrading.
+
+ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 1h v1.11.3
+boomer Ready master,node 1h v1.11.3
+caprica Ready master,node 1h v1.11.3
+
+$ git checkout v2.8.0
+Previous HEAD position was 05dabb7e Fix Bionic networking restart error #3430 (#3431)
+HEAD is now at 9051aa52 Fix ubuntu-contiv test failed (#3808)
+```
+
+:info: NOTE: Review changes between the sample inventory and your inventory when upgrading versions. :info:
+
+Some deprecations between versions that mean you can't just upgrade straight from 2.7.0 to 2.8.0 if you started with the sample inventory.
+
+In this case, I set "kubeadm_enabled" to false, knowing that it is deprecated and removed by 2.9.0, to delay converting the cluster to kubeadm as long as I could.
+
+```ShellSession
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 114m v1.12.3
+boomer Ready master,node 114m v1.12.3
+caprica Ready master,node 114m v1.12.3
+
+$ git checkout v2.8.1
+Previous HEAD position was 9051aa52 Fix ubuntu-contiv test failed (#3808)
+HEAD is now at 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
+
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 2h36m v1.12.4
+boomer Ready master,node 2h36m v1.12.4
+caprica Ready master,node 2h36m v1.12.4
+
+$ git checkout v2.8.2
+Previous HEAD position was 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
+HEAD is now at 4167807f Upgrade to 1.12.5 (#4066)
+
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 3h3m v1.12.5
+boomer Ready master,node 3h3m v1.12.5
+caprica Ready master,node 3h3m v1.12.5
+
+$ git checkout v2.8.3
+Previous HEAD position was 4167807f Upgrade to 1.12.5 (#4066)
+HEAD is now at ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
+
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 5h18m v1.12.5
+boomer Ready master,node 5h18m v1.12.5
+caprica Ready master,node 5h18m v1.12.5
+
+$ git checkout v2.8.4
+Previous HEAD position was ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
+HEAD is now at 3901480b go to k8s 1.12.7 (#4400)
+
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 5h37m v1.12.7
+boomer Ready master,node 5h37m v1.12.7
+caprica Ready master,node 5h37m v1.12.7
+
+$ git checkout v2.8.5
+Previous HEAD position was 3901480b go to k8s 1.12.7 (#4400)
+HEAD is now at 6f97687d Release 2.8 robust san handling (#4478)
+
+$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+...
+ "msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
+...
+Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
+yes
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 5h45m v1.12.7
+boomer Ready master,node 5h45m v1.12.7
+caprica Ready master,node 5h45m v1.12.7
+
+$ git checkout v2.9.0
+Previous HEAD position was 6f97687d Release 2.8 robust san handling (#4478)
+HEAD is now at a4e65c7c Upgrade to Ansible >2.7.0 (#4471)
+```
+
+:warning: IMPORTANT: Some of the variable formats changed in the k8s_cluster.yml between 2.8.5 and 2.9.0 :warning:
+
+If you do not keep your inventory copy up to date, **your upgrade will fail** and your first master will be left non-functional until fixed and re-run.
+
+It is at this point the cluster was upgraded from non-kubeadm to kubeadm as per the deprecation warning.
+
+```ShellSession
+ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 6h54m v1.13.5
+boomer Ready master,node 6h55m v1.13.5
+caprica Ready master,node 6h54m v1.13.5
+
+# Watch out: 2.10.0 is hiding between 2.1.2 and 2.2.0
+
+$ git tag
+...
+v2.1.0
+v2.1.1
+v2.1.2
+v2.10.0
+v2.2.0
+...
+
+$ git checkout v2.10.0
+Previous HEAD position was a4e65c7c Upgrade to Ansible >2.7.0 (#4471)
+HEAD is now at dcd9c950 Add etcd role dependency on kube user to avoid etcd role failure when running scale.yml with a fresh node. (#3240) (#4479)
+
+ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
+
+...
+
+$ kubectl get node
+NAME STATUS ROLES AGE VERSION
+apollo Ready master,node 7h40m v1.14.1
+boomer Ready master,node 7h40m v1.14.1
+caprica Ready master,node 7h40m v1.14.1
+
+
+```
+
+## Upgrading to v2.19
+
+`etcd_kubeadm_enabled` is being deprecated at v2.19. The same functionality is achievable by setting `etcd_deployment_type` to `kubeadm`.
+Deploying etcd using kubeadm is experimental and is only available for either new or deployments where `etcd_kubeadm_enabled` was set to `true` while deploying the cluster.
+
+From 2.19 and onward `etcd_deployment_type` variable will be placed in `group_vars/all/etcd.yml` instead of `group_vars/etcd.yml`, due to scope issues.
+The placement of the variable is only important for `etcd_deployment_type: kubeadm` right now. However, since this might change in future updates, it is recommended to move the variable.
+
+Upgrading is straightforward; no changes are required if `etcd_kubeadm_enabled` was not set to `true` when deploying.
+
+If you have a cluster where `etcd` was deployed using `kubeadm`, you will need to remove `etcd_kubeadm_enabled` the variable. Then move `etcd_deployment_type` variable from `group_vars/etcd.yml` to `group_vars/all/etcd.yml` due to scope issues and set `etcd_deployment_type` to `kubeadm`.
+
+## Upgrade order
+
+As mentioned above, components are upgraded in the order in which they were
+installed in the Ansible playbook. The order of component installation is as
+follows:
+
+* Docker
+* Containerd
+* etcd
+* kubelet and kube-proxy
+* network_plugin (such as Calico or Weave)
+* kube-apiserver, kube-scheduler, and kube-controller-manager
+* Add-ons (such as KubeDNS)
+
+### Component-based upgrades
+
+A deployer may want to upgrade specific components in order to minimize risk
+or save time. This strategy is not covered by CI as of this writing, so it is
+not guaranteed to work.
+
+These commands are useful only for upgrading fully-deployed, healthy, existing
+hosts. This will definitely not work for undeployed or partially deployed
+hosts.
+
+Upgrade docker:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=docker
+```
+
+Upgrade etcd:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd
+```
+
+Upgrade etcd without rotating etcd certs:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd --limit=etcd --skip-tags=etcd-secrets
+```
+
+Upgrade kubelet:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=node --skip-tags=k8s-gen-certs,k8s-gen-tokens
+```
+
+Upgrade Kubernetes master components:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=master
+```
+
+Upgrade network plugins:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=network
+```
+
+Upgrade all add-ons:
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=apps
+```
+
+Upgrade just helm (assuming `helm_enabled` is true):
+
+```ShellSession
+ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=helm
+```
+
+## Migrate from Docker to Containerd
+
+Please note that **migrating container engines is not officially supported by Kubespray**. While this procedure can be used to migrate your cluster, it applies to one particular scenario and will likely evolve over time. At the moment, they are intended as an additional resource to provide insight into how these steps can be officially integrated into the Kubespray playbooks.
+
+As of Kubespray 2.18.0, containerd is already the default container engine. If you have the chance, it is advisable and safer to reset and redeploy the entire cluster with a new container engine.
+
+* [Migrating from Docker do Containerd](upgrades/migrate_docker2containerd.md)
--- /dev/null
+# Migrating from Docker to Containerd
+
+❗MAKE SURE YOU READ BEFORE PROCEEDING❗
+
+**Migrating container engines is not officially supported by Kubespray**. The following procedure covers one particular scenario and involves manual steps, along with multiple runs of `cluster.yml`. It provides no guarantees that it will actually work or that any further action is needed. Please, consider these instructions as experimental guidelines. While they can be used to migrate your cluster, they will likely evolve over time. At the moment, they are intended as an additional resource to provide insight into how these steps can be officially integrated into the Kubespray playbooks.
+
+As of Kubespray 2.18.0, containerd is already the default container engine. If you have the chance, it is still advisable and safer to reset and redeploy the entire cluster with a new container engine.
+
+Input and feedback are always appreciated.
+
+## Tested environment
+
+Nodes: Ubuntu 18.04 LTS\
+Cloud Provider: None (baremetal or VMs)\
+Kubernetes version: 1.21.5\
+Kubespray version: 2.18.0
+
+## Important considerations
+
+If you require minimum downtime, nodes need to be cordoned and drained before being processed, one by one. If you wish to run `cluster.yml` only once and get it all done in one swoop, downtime will be significantly higher. Docker will need to be manually removed from all nodes before the playbook runs (see [#8431](https://github.com/kubernetes-sigs/kubespray/issues/8431)). For minimum downtime, the following steps will be executed multiple times, once for each node.
+
+Processing nodes one by one also means you will not be able to update any other cluster configuration using Kubespray before this procedure is finished and the cluster is fully migrated.
+
+Everything done here requires full root access to every node.
+
+## Migration steps
+
+Before you begin, adjust your inventory:
+
+```yaml
+# Filename: k8s_cluster/k8s-cluster.yml
+resolvconf_mode: host_resolvconf
+container_manager: containerd
+
+# Filename: etcd.yml
+etcd_deployment_type: host
+```
+
+### 1) Pick one or more nodes for processing
+
+It is still unclear how the order might affect this procedure. So, to be sure, it might be best to start with the control plane and etcd nodes all together, followed by each worker node individually.
+
+### 2) Cordon and drain the node
+
+... because, downtime.
+
+### 3) Stop docker and kubelet daemons
+
+```commandline
+service kubelet stop
+service docker stop
+```
+
+### 4) Uninstall docker + dependencies
+
+```commandline
+apt-get remove -y --allow-change-held-packages containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras
+```
+
+In some cases, there might a `pigz` missing dependency. Some image layers need this to be extracted.
+
+```shell
+apt-get install pigz
+```
+
+### 5) Run `cluster.yml` playbook with `--limit`
+
+```commandline
+ansible-playbook cluster.yml -i inventory/sample/hosts.ini cluster.yml --limit=NODENAME
+```
+
+This effectively reinstalls containerd and seems to place all config files in the right place. When this completes, kubelet will immediately pick up the new container engine and start spinning up DaemonSets and kube-system Pods.
+
+Optionally, if you feel confident, you can remove `/var/lib/docker` anytime after this step.
+
+```commandline
+rm -fr /var/lib/docker
+```
+
+You can watch new containers using `crictl`.
+
+```commandline
+crictl ps -a
+```
+
+### 6) Replace the cri-socket node annotation
+
+Node annotations need to be adjusted. Kubespray will not do this, but a simple kubectl is enough.
+
+```commandline
+kubectl annotate node NODENAME --overwrite kubeadm.alpha.kubernetes.io/cri-socket=/var/run/containerd/containerd.sock
+```
+
+The annotation is required by kubeadm to follow through future cluster upgrades.
+
+### 7) Reboot the node
+
+Reboot, just to make sure everything restarts fresh before the node is uncordoned.
+
+## After thoughts
+
+If your cluster runs a log aggregator, like fluentd+Graylog, you will likely need to adjust collection filters and parsers. While docker generates Json logs, containerd has its own space delimited format. Example:
+
+```text
+2020-01-10T18:10:40.01576219Z stdout F application log message...
+```
--- /dev/null
+# Vagrant
+
+Assuming you have Vagrant 2.0+ installed with virtualbox, libvirt/qemu or
+vmware, but is untested) you should be able to launch a 3 node Kubernetes
+cluster by simply running `vagrant up`.
+
+This will spin up 3 VMs and install kubernetes on them.
+Once they are completed you can connect to any of them by running `vagrant ssh k8s-[1..3]`.
+
+To give an estimate of the expected duration of a provisioning run:
+On a dual core i5-6300u laptop with an SSD, provisioning takes around 13
+to 15 minutes, once the container images and other files are cached.
+Note that libvirt/qemu is recommended over virtualbox as it is quite a bit
+faster, especially during boot-up time.
+
+For proper performance a minimum of 12GB RAM is recommended.
+It is possible to run a 3 node cluster on a laptop with 8GB of RAM using
+the default Vagrantfile, provided you have 8GB zram swap configured and
+not much more than a browser and a mail client running.
+If you decide to run on such a machine, then also make sure that any tmpfs
+devices, that are mounted, are mostly empty and disable any swapfiles
+mounted on HDD/SSD or you will be in for some serious swap-madness.
+Things can get a bit sluggish during provisioning, but when that's done,
+the system will actually be able to perform quite well.
+
+## Customize Vagrant
+
+You can override the default settings in the `Vagrantfile` either by
+directly modifying the `Vagrantfile` or through an override file.
+In the same directory as the `Vagrantfile`, create a folder called
+`vagrant` and create `config.rb` file in it.
+An example of how to configure this file is given below.
+
+## Use alternative OS for Vagrant
+
+By default, Vagrant uses Ubuntu 18.04 box to provision a local cluster.
+You may use an alternative supported operating system for your local cluster.
+
+Customize `$os` variable in `Vagrantfile` or as override, e.g.,:
+
+```ShellSession
+echo '$os = "flatcar-stable"' >> vagrant/config.rb
+```
+
+The supported operating systems for vagrant are defined in the `SUPPORTED_OS`
+constant in the `Vagrantfile`.
+
+## File and image caching
+
+Kubespray can take quite a while to start on a laptop. To improve provisioning
+speed, the variable 'download_run_once' is set. This will make kubespray
+download all files and containers just once and then redistributes them to
+the other nodes and as a bonus, also cache all downloads locally and re-use
+them on the next provisioning run. For more information on download settings
+see [download documentation](/docs/downloads.md).
+
+## Example use of Vagrant
+
+The following is an example of setting up and running kubespray using `vagrant`.
+For repeated runs, you could save the script to a file in the root of the
+kubespray and run it by executing 'source <name_of_the_file>.
+
+```ShellSession
+# use virtualenv to install all python requirements
+VENVDIR=venv
+virtualenv --python=/usr/bin/python3.7 $VENVDIR
+source $VENVDIR/bin/activate
+pip install -r requirements.txt
+
+# prepare an inventory to test with
+INV=inventory/my_lab
+rm -rf ${INV}.bak &> /dev/null
+mv ${INV} ${INV}.bak &> /dev/null
+cp -a inventory/sample ${INV}
+rm -f ${INV}/hosts.ini
+
+# customize the vagrant environment
+mkdir vagrant
+cat << EOF > vagrant/config.rb
+\$instance_name_prefix = "kub"
+\$vm_cpus = 1
+\$num_instances = 3
+\$os = "centos-bento"
+\$subnet = "10.0.20"
+\$network_plugin = "flannel"
+\$inventory = "$INV"
+\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" }
+EOF
+
+# make the rpm cache
+mkdir -p temp/docker_rpms
+
+vagrant up
+
+# make a copy of the downloaded docker rpm, to speed up the next provisioning run
+scp kub-1:/var/cache/yum/x86_64/7/docker-ce/packages/* temp/docker_rpms/
+
+# copy kubectl access configuration in place
+mkdir $HOME/.kube/ &> /dev/null
+ln -s $PWD/$INV/artifacts/admin.conf $HOME/.kube/config
+# make the kubectl binary available
+sudo ln -s $PWD/$INV/artifacts/kubectl /usr/local/bin/kubectl
+#or
+export PATH=$PATH:$PWD/$INV/artifacts
+```
+
+If a vagrant run failed and you've made some changes to fix the issue causing
+the fail, here is how you would re-run ansible:
+
+```ShellSession
+ansible-playbook -vvv -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory cluster.yml
+```
+
+If all went well, you check if it's all working as expected:
+
+```ShellSession
+kubectl get nodes
+```
+
+The output should look like this:
+
+```ShellSession
+$ kubectl get nodes
+NAME STATUS ROLES AGE VERSION
+kub-1 Ready control-plane,master 4m37s v1.22.5
+kub-2 Ready control-plane,master 4m7s v1.22.5
+kub-3 Ready <none> 3m7s v1.22.5
+```
+
+Another nice test is the following:
+
+```ShellSession
+kubectl get pods --all-namespaces -o wide
+```
+
+Which should yield something like the following:
+
+```ShellSession
+$ kubectl get pods --all-namespaces -o wide
+NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
+kube-system coredns-8474476ff8-m2469 1/1 Running 0 2m45s 10.233.65.2 kub-2 <none> <none>
+kube-system coredns-8474476ff8-v5wzj 1/1 Running 0 2m41s 10.233.64.3 kub-1 <none> <none>
+kube-system dns-autoscaler-5ffdc7f89d-76tnv 1/1 Running 0 2m43s 10.233.64.2 kub-1 <none> <none>
+kube-system kube-apiserver-kub-1 1/1 Running 1 4m54s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-apiserver-kub-2 1/1 Running 1 4m33s 10.0.20.102 kub-2 <none> <none>
+kube-system kube-controller-manager-kub-1 1/1 Running 1 5m1s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-controller-manager-kub-2 1/1 Running 1 4m33s 10.0.20.102 kub-2 <none> <none>
+kube-system kube-flannel-9xgf5 1/1 Running 0 3m10s 10.0.20.102 kub-2 <none> <none>
+kube-system kube-flannel-l8jbl 1/1 Running 0 3m10s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-flannel-zss4t 1/1 Running 0 3m10s 10.0.20.103 kub-3 <none> <none>
+kube-system kube-multus-ds-amd64-bhpc9 1/1 Running 0 3m2s 10.0.20.103 kub-3 <none> <none>
+kube-system kube-multus-ds-amd64-n6vl8 1/1 Running 0 3m2s 10.0.20.102 kub-2 <none> <none>
+kube-system kube-multus-ds-amd64-qttgs 1/1 Running 0 3m2s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-proxy-2x4jl 1/1 Running 0 3m33s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-proxy-d48r7 1/1 Running 0 3m33s 10.0.20.103 kub-3 <none> <none>
+kube-system kube-proxy-f45lp 1/1 Running 0 3m33s 10.0.20.102 kub-2 <none> <none>
+kube-system kube-scheduler-kub-1 1/1 Running 1 4m54s 10.0.20.101 kub-1 <none> <none>
+kube-system kube-scheduler-kub-2 1/1 Running 1 4m33s 10.0.20.102 kub-2 <none> <none>
+kube-system nginx-proxy-kub-3 1/1 Running 0 3m33s 10.0.20.103 kub-3 <none> <none>
+kube-system nodelocaldns-cg9tz 1/1 Running 0 2m41s 10.0.20.102 kub-2 <none> <none>
+kube-system nodelocaldns-htswt 1/1 Running 0 2m41s 10.0.20.103 kub-3 <none> <none>
+kube-system nodelocaldns-nsp7s 1/1 Running 0 2m41s 10.0.20.101 kub-1 <none> <none>
+local-path-storage local-path-provisioner-66df45bfdd-km4zg 1/1 Running 0 2m54s 10.233.66.2 kub-3 <none> <none>
+```
--- /dev/null
+# Configurable Parameters in Kubespray
+
+## Generic Ansible variables
+
+You can view facts gathered by Ansible automatically
+[here](https://docs.ansible.com/ansible/latest/user_guide/playbooks_vars_facts.html#ansible-facts).
+
+Some variables of note include:
+
+* *ansible_user*: user to connect to via SSH
+* *ansible_default_ipv4.address*: IP address Ansible automatically chooses.
+ Generated based on the output from the command ``ip -4 route get 8.8.8.8``
+
+## Common vars that are used in Kubespray
+
+* *calico_version* - Specify version of Calico to use
+* *calico_cni_version* - Specify version of Calico CNI plugin to use
+* *docker_version* - Specify version of Docker to use (should be quoted
+ string). Must match one of the keys defined for *docker_versioned_pkg*
+ in `roles/container-engine/docker/vars/*.yml`.
+* *containerd_version* - Specify version of containerd to use when setting `container_manager` to `containerd`
+* *docker_containerd_version* - Specify which version of containerd to use when setting `container_manager` to `docker`
+* *etcd_version* - Specify version of ETCD to use
+* *calico_ipip_mode* - Configures Calico ipip encapsulation - valid values are 'Never', 'Always' and 'CrossSubnet' (default 'Never')
+* *calico_vxlan_mode* - Configures Calico vxlan encapsulation - valid values are 'Never', 'Always' and 'CrossSubnet' (default 'Always')
+* *calico_network_backend* - Configures Calico network backend - valid values are 'none', 'bird' and 'vxlan' (default 'vxlan')
+* *kube_network_plugin* - Sets k8s network plugin (default Calico)
+* *kube_proxy_mode* - Changes k8s proxy mode to iptables mode
+* *kube_version* - Specify a given Kubernetes version
+* *searchdomains* - Array of DNS domains to search when looking up hostnames
+* *nameservers* - Array of nameservers to use for DNS lookup
+* *preinstall_selinux_state* - Set selinux state, permitted values are permissive, enforcing and disabled.
+
+## Addressing variables
+
+* *ip* - IP to use for binding services (host var)
+* *access_ip* - IP for other hosts to use to connect to. Often required when
+ deploying from a cloud, such as OpenStack or GCE and you have separate
+ public/floating and private IPs.
+* *ansible_default_ipv4.address* - Not Kubespray-specific, but it is used if ip
+ and access_ip are undefined
+* *ip6* - IPv6 address to use for binding services. (host var)
+ If *enable_dual_stack_networks* is set to ``true`` and *ip6* is defined,
+ kubelet's ``--node-ip`` and node's ``InternalIP`` will be the combination of *ip* and *ip6*.
+* *loadbalancer_apiserver* - If defined, all hosts will connect to this
+ address instead of localhost for kube_control_planes and kube_control_plane[0] for
+ kube_nodes. See more details in the
+ [HA guide](/docs/ha-mode.md).
+* *loadbalancer_apiserver_localhost* - makes all hosts to connect to
+ the apiserver internally load balanced endpoint. Mutual exclusive to the
+ `loadbalancer_apiserver`. See more details in the
+ [HA guide](/docs/ha-mode.md).
+
+## Cluster variables
+
+Kubernetes needs some parameters in order to get deployed. These are the
+following default cluster parameters:
+
+* *cluster_name* - Name of cluster (default is cluster.local)
+
+* *container_manager* - Container Runtime to install in the nodes (default is containerd)
+
+* *image_command_tool* - Tool used to pull images (default depends on `container_manager`
+ and is `nerdctl` for `containerd`, `crictl` for `crio`, `docker` for `docker`)
+
+* *image_command_tool_on_localhost* - Tool used to pull images on localhost
+ (default is equal to `image_command_tool`)
+
+* *dns_domain* - Name of cluster DNS domain (default is cluster.local)
+
+* *kube_network_plugin* - Plugin to use for container networking
+
+* *kube_service_addresses* - Subnet for cluster IPs (default is
+ 10.233.0.0/18). Must not overlap with kube_pods_subnet
+
+* *kube_pods_subnet* - Subnet for Pod IPs (default is 10.233.64.0/18). Must not
+ overlap with kube_service_addresses.
+
+* *kube_network_node_prefix* - Subnet allocated per-node for pod IPs. Remaining
+ bits in kube_pods_subnet dictates how many kube_nodes can be in cluster. Setting this > 25 will
+ raise an assertion in playbooks if the `kubelet_max_pods` var also isn't adjusted accordingly
+ (assertion not applicable to calico which doesn't use this as a hard limit, see
+ [Calico IP block sizes](https://docs.projectcalico.org/reference/resources/ippool#block-sizes).
+
+* *enable_dual_stack_networks* - Setting this to true will provision both IPv4 and IPv6 networking for pods and services.
+
+* *kube_service_addresses_ipv6* - Subnet for cluster IPv6 IPs (default is ``fd85:ee78:d8a6:8607::1000/116``). Must not overlap with ``kube_pods_subnet_ipv6``.
+
+* *kube_pods_subnet_ipv6* - Subnet for Pod IPv6 IPs (default is ``fd85:ee78:d8a6:8607::1:0000/112``). Must not overlap with ``kube_service_addresses_ipv6``.
+
+* *kube_network_node_prefix_ipv6* - Subnet allocated per-node for pod IPv6 IPs. Remaining bits in ``kube_pods_subnet_ipv6`` dictates how many kube_nodes can be in cluster.
+
+* *skydns_server* - Cluster IP for DNS (default is 10.233.0.3)
+
+* *skydns_server_secondary* - Secondary Cluster IP for CoreDNS used with coredns_dual deployment (default is 10.233.0.4)
+
+* *enable_coredns_k8s_external* - If enabled, it configures the [k8s_external plugin](https://coredns.io/plugins/k8s_external/)
+ on the CoreDNS service.
+
+* *coredns_k8s_external_zone* - Zone that will be used when CoreDNS k8s_external plugin is enabled
+ (default is k8s_external.local)
+
+* *enable_coredns_k8s_endpoint_pod_names* - If enabled, it configures endpoint_pod_names option for kubernetes plugin.
+ on the CoreDNS service.
+
+* *cloud_provider* - Enable extra Kubelet option if operating inside GCE or
+ OpenStack (default is unset)
+
+* *kube_feature_gates* - A list of key=value pairs that describe feature gates for
+ alpha/experimental Kubernetes features. (defaults is `[]`).
+ Additionally, you can use also the following variables to individually customize your kubernetes components installation (they works exactly like `kube_feature_gates`):
+ * *kube_apiserver_feature_gates*
+ * *kube_controller_feature_gates*
+ * *kube_scheduler_feature_gates*
+ * *kube_proxy_feature_gates*
+ * *kubelet_feature_gates*
+
+* *kubeadm_feature_gates* - A list of key=value pairs that describe feature gates for
+ alpha/experimental Kubeadm features. (defaults is `[]`)
+
+* *authorization_modes* - A list of [authorization mode](
+ https://kubernetes.io/docs/admin/authorization/#using-flags-for-your-authorization-module)
+ that the cluster should be configured for. Defaults to `['Node', 'RBAC']`
+ (Node and RBAC authorizers).
+ Note: `Node` and `RBAC` are enabled by default. Previously deployed clusters can be
+ converted to RBAC mode. However, your apps which rely on Kubernetes API will
+ require a service account and cluster role bindings. You can override this
+ setting by setting authorization_modes to `[]`.
+
+* *kube_apiserver_admission_control_config_file* - Enable configuration for `kube-apiserver` admission plugins.
+ Currently this variable allow you to configure the `EventRateLimit` admission plugin.
+
+ To configure the **EventRateLimit** plugin you have to define a data structure like this:
+
+```yml
+kube_apiserver_admission_event_rate_limits:
+ limit_1:
+ type: Namespace
+ qps: 50
+ burst: 100
+ cache_size: 2000
+ limit_2:
+ type: User
+ qps: 50
+ burst: 100
+ ...
+```
+
+* *kube_apiserver_service_account_lookup* - Enable validation service account before validating token. Default `true`.
+
+Note, if cloud providers have any use of the ``10.233.0.0/16``, like instances'
+private addresses, make sure to pick another values for ``kube_service_addresses``
+and ``kube_pods_subnet``, for example from the ``172.18.0.0/16``.
+
+## Enabling Dual Stack (IPV4 + IPV6) networking
+
+If *enable_dual_stack_networks* is set to ``true``, Dual Stack networking will be enabled in the cluster. This will use the default IPv4 and IPv6 subnets specified in the defaults file in the ``kubespray-defaults`` role, unless overridden of course. The default config will give you room for up to 256 nodes with 126 pods per node, and up to 4096 services.
+
+## DNS variables
+
+By default, hosts are set up with 8.8.8.8 as an upstream DNS server and all
+other settings from your existing /etc/resolv.conf are lost. Set the following
+variables to match your requirements.
+
+* *upstream_dns_servers* - Array of upstream DNS servers configured on host in
+ addition to Kubespray deployed DNS
+* *nameservers* - Array of DNS servers configured for use by hosts
+* *searchdomains* - Array of up to 4 search domains
+* *dns_etchosts* - Content of hosts file for coredns and nodelocaldns
+
+For more information, see [DNS
+Stack](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/dns-stack.md).
+
+## Other service variables
+
+* *docker_options* - Commonly used to set
+ ``--insecure-registry=myregistry.mydomain:5000``
+* *docker_plugins* - This list can be used to define [Docker plugins](https://docs.docker.com/engine/extend/) to install.
+* *containerd_default_runtime* - If defined, changes the default Containerd runtime used by the Kubernetes CRI plugin.
+* *containerd_additional_runtimes* - Sets the additional Containerd runtimes used by the Kubernetes CRI plugin.
+ [Default config](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/container-engine/containerd/defaults/main.yml) can be overriden in inventory vars.
+* *http_proxy/https_proxy/no_proxy/no_proxy_exclude_workers/additional_no_proxy* - Proxy variables for deploying behind a
+ proxy. Note that no_proxy defaults to all internal cluster IPs and hostnames
+ that correspond to each node.
+* *kubelet_cgroup_driver* - Allows manual override of the cgroup-driver option for Kubelet.
+ By default autodetection is used to match container manager configuration.
+ `systemd` is the preferred driver for `containerd` though it can have issues with `cgroups v1` and `kata-containers` in which case you may want to change to `cgroupfs`.
+* *kubelet_rotate_certificates* - Auto rotate the kubelet client certificates by requesting new certificates
+ from the kube-apiserver when the certificate expiration approaches.
+* *kubelet_rotate_server_certificates* - Auto rotate the kubelet server certificates by requesting new certificates
+ from the kube-apiserver when the certificate expiration approaches.
+ **Note** that server certificates are **not** approved automatically. Approve them manually
+ (`kubectl get csr`, `kubectl certificate approve`) or implement custom approving controller like
+ [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp).
+* *kubelet_streaming_connection_idle_timeout* - Set the maximum time a streaming connection can be idle before the connection is automatically closed.
+* *kubelet_make_iptables_util_chains* - If `true`, causes the kubelet ensures a set of `iptables` rules are present on host.
+* *node_labels* - Labels applied to nodes via kubelet --node-labels parameter.
+ For example, labels can be set in the inventory as variables or more widely in group_vars.
+ *node_labels* can only be defined as a dict:
+
+```yml
+node_labels:
+ label1_name: label1_value
+ label2_name: label2_value
+```
+
+* *node_taints* - Taints applied to nodes via kubelet --register-with-taints parameter.
+ For example, taints can be set in the inventory as variables or more widely in group_vars.
+ *node_taints* has to be defined as a list of strings in format `key=value:effect`, e.g.:
+
+```yml
+node_taints:
+ - "node.example.com/external=true:NoSchedule"
+```
+
+* *podsecuritypolicy_enabled* - When set to `true`, enables the PodSecurityPolicy admission controller and defines two policies `privileged` (applying to all resources in `kube-system` namespace and kubelet) and `restricted` (applying all other namespaces).
+ Addons deployed in kube-system namespaces are handled.
+* *kubernetes_audit* - When set to `true`, enables Auditing.
+ The auditing parameters can be tuned via the following variables (which default values are shown below):
+ * `audit_log_path`: /var/log/audit/kube-apiserver-audit.log
+ * `audit_log_maxage`: 30
+ * `audit_log_maxbackups`: 1
+ * `audit_log_maxsize`: 100
+ * `audit_policy_file`: "{{ kube_config_dir }}/audit-policy/apiserver-audit-policy.yaml"
+
+ By default, the `audit_policy_file` contains [default rules](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubernetes/control-plane/templates/apiserver-audit-policy.yaml.j2) that can be overridden with the `audit_policy_custom_rules` variable.
+* *kubernetes_audit_webhook* - When set to `true`, enables the webhook audit backend.
+ The webhook parameters can be tuned via the following variables (which default values are shown below):
+ * `audit_webhook_config_file`: "{{ kube_config_dir }}/audit-policy/apiserver-audit-webhook-config.yaml"
+ * `audit_webhook_server_url`: `"https://audit.app"`
+ * `audit_webhook_server_extra_args`: {}
+ * `audit_webhook_mode`: batch
+ * `audit_webhook_batch_max_size`: 100
+ * `audit_webhook_batch_max_wait`: 1s
+
+### Custom flags for Kube Components
+
+For all kube components, custom flags can be passed in. This allows for edge cases where users need changes to the default deployment that may not be applicable to all deployments.
+
+Extra flags for the kubelet can be specified using these variables,
+in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubelet YAML config file. The `kubelet_node_config_extra_args` apply kubelet settings only to nodes and not control planes. Example:
+
+```yml
+kubelet_config_extra_args:
+ evictionHard:
+ memory.available: "100Mi"
+ evictionSoftGracePeriod:
+ memory.available: "30s"
+ evictionSoft:
+ memory.available: "300Mi"
+```
+
+The possible vars are:
+
+* *kubelet_config_extra_args*
+* *kubelet_node_config_extra_args*
+
+Previously, the same parameters could be passed as flags to kubelet binary with the following vars:
+
+* *kubelet_custom_flags*
+* *kubelet_node_custom_flags*
+
+The `kubelet_node_custom_flags` apply kubelet settings only to nodes and not control planes. Example:
+
+```yml
+kubelet_custom_flags:
+ - "--eviction-hard=memory.available<100Mi"
+ - "--eviction-soft-grace-period=memory.available=30s"
+ - "--eviction-soft=memory.available<300Mi"
+```
+
+This alternative is deprecated and will remain until the flags are completely removed from kubelet
+
+Extra flags for the API server, controller, and scheduler components can be specified using these variables,
+in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubeadm YAML config file:
+
+* *kube_kubeadm_apiserver_extra_args*
+* *kube_kubeadm_controller_extra_args*
+* *kube_kubeadm_scheduler_extra_args*
+
+## App variables
+
+* *helm_version* - Only supports v3.x. Existing v2 installs (with Tiller) will not be modified and need to be removed manually.
--- /dev/null
+# vSphere CSI Driver
+
+vSphere CSI driver allows you to provision volumes over a vSphere deployment. The Kubernetes historic in-tree cloud provider is deprecated and will be removed in future versions.
+
+## Prerequisites
+
+The vSphere user for CSI driver requires a set of privileges to perform Cloud Native Storage operations. Follow the [official guide](https://vsphere-csi-driver.sigs.k8s.io/driver-deployment/prerequisites.html#roles_and_privileges) to configure those.
+
+## Kubespray configuration
+
+To enable vSphere CSI driver, uncomment the `vsphere_csi_enabled` option in `group_vars/all/vsphere.yml` and set it to `true`.
+
+To set the number of replicas for the vSphere CSI controller, you can change `vsphere_csi_controller_replicas` option in `group_vars/all/vsphere.yml`.
+
+You need to source the vSphere credentials you use to deploy your machines that will host Kubernetes.
+
+| Variable | Required | Type | Choices | Default | Comment |
+|---------------------------------------------|----------|---------|----------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------|
+| external_vsphere_vcenter_ip | TRUE | string | | | IP/URL of the vCenter |
+| external_vsphere_vcenter_port | TRUE | string | | "443" | Port of the vCenter API |
+| external_vsphere_insecure | TRUE | string | "true", "false" | "true" | set to "true" if the host above uses a self-signed cert |
+| external_vsphere_user | TRUE | string | | | User name for vCenter with required privileges (Can also be specified with the `VSPHERE_USER` environment variable) |
+| external_vsphere_password | TRUE | string | | | Password for vCenter (Can also be specified with the `VSPHERE_PASSWORD` environment variable) |
+| external_vsphere_datacenter | TRUE | string | | | Datacenter name to use |
+| external_vsphere_kubernetes_cluster_id | TRUE | string | | "kubernetes-cluster-id" | Kubernetes cluster ID to use |
+| external_vsphere_version | TRUE | string | | "6.7u3" | Vmware Vsphere version where located all VMs |
+| external_vsphere_cloud_controller_image_tag | TRUE | string | | "latest" | Kubernetes cluster ID to use |
+| vsphere_syncer_image_tag | TRUE | string | | "v2.2.1" | Syncer image tag to use |
+| vsphere_csi_attacher_image_tag | TRUE | string | | "v3.1.0" | CSI attacher image tag to use |
+| vsphere_csi_controller | TRUE | string | | "v2.2.1" | CSI controller image tag to use |
+| vsphere_csi_controller_replicas | TRUE | integer | | 1 | Number of pods Kubernetes should deploy for the CSI controller |
+| vsphere_csi_liveness_probe_image_tag | TRUE | string | | "v2.2.0" | CSI liveness probe image tag to use |
+| vsphere_csi_provisioner_image_tag | TRUE | string | | "v2.1.0" | CSI provisioner image tag to use |
+| vsphere_csi_node_driver_registrar_image_tag | TRUE | string | | "v1.1.0" | CSI node driver registrar image tag to use |
+| vsphere_csi_driver_image_tag | TRUE | string | | "v1.0.2" | CSI driver image tag to use |
+| vsphere_csi_resizer_tag | TRUE | string | | "v1.1.0" | CSI resizer image tag to use
+| vsphere_csi_aggressive_node_drain | FALSE | boolean | | false | Enable aggressive node drain strategy |
+| vsphere_csi_aggressive_node_unreachable_timeout | FALSE | int | 300 | | Timeout till node will be drained when it in an unreachable state |
+| vsphere_csi_aggressive_node_not_ready_timeout | FALSE | int | 300 | | Timeout till node will be drained when it in not-ready state |
+
+## Usage example
+
+To test the dynamic provisioning using vSphere CSI driver, make sure to create a [storage policy](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#create-a-storage-policy) and [storage class](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#create-a-storageclass), then apply the following manifest:
+
+```yml
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: csi-pvc-vsphere
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 1Gi
+ storageClassName: Space-Efficient
+
+---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: nginx
+spec:
+ containers:
+ - image: nginx
+ imagePullPolicy: IfNotPresent
+ name: nginx
+ ports:
+ - containerPort: 80
+ protocol: TCP
+ volumeMounts:
+ - mountPath: /usr/share/nginx/html
+ name: csi-data-vsphere
+ volumes:
+ - name: csi-data-vsphere
+ persistentVolumeClaim:
+ claimName: csi-pvc-vsphere
+ readOnly: false
+```
+
+Apply this conf to your cluster: ```kubectl apply -f nginx.yml```
+
+You should see the PVC provisioned and bound:
+
+```ShellSession
+$ kubectl get pvc
+NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
+csi-pvc-vsphere Bound pvc-dc7b1d21-ee41-45e1-98d9-e877cc1533ac 1Gi RWO Space-Efficient 10s
+```
+
+And the volume mounted to the Nginx Pod (wait until the Pod is Running):
+
+```ShellSession
+kubectl exec -it nginx -- df -h | grep /usr/share/nginx/html
+/dev/sdb 976M 2.6M 907M 1% /usr/share/nginx/html
+```
+
+## More info
+
+For further information about the vSphere CSI Driver, you can refer to the official [vSphere Cloud Provider documentation](https://cloud-provider-vsphere.sigs.k8s.io/container_storage_interface.html).
--- /dev/null
+# vSphere
+
+Kubespray can be deployed with vSphere as Cloud provider. This feature supports:
+
+- Volumes
+- Persistent Volumes
+- Storage Classes and provisioning of volumes
+- vSphere Storage Policy Based Management for Containers orchestrated by Kubernetes
+
+## Out-of-tree vSphere cloud provider
+
+### Prerequisites
+
+You need at first to configure your vSphere environment by following the [official documentation](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#prerequisites).
+
+After this step you should have:
+
+- vSphere upgraded to 6.7 U3 or later
+- VM hardware upgraded to version 15 or higher
+- UUID activated for each VM where Kubernetes will be deployed
+
+### Kubespray configuration
+
+First in `inventory/sample/group_vars/all.yml` you must set the cloud provider to `external` and external_cloud_provider to `external_cloud_provider`.
+
+```yml
+cloud_provider: "external"
+external_cloud_provider: "vsphere"
+```
+
+Then, `inventory/sample/group_vars/vsphere.yml`, you need to declare your vCenter credentials and enable the vSphere CSI following the description below.
+
+| Variable | Required | Type | Choices | Default | Comment |
+|----------------------------------------|----------|---------|----------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------|
+| external_vsphere_vcenter_ip | TRUE | string | | | IP/URL of the vCenter |
+| external_vsphere_vcenter_port | TRUE | string | | "443" | Port of the vCenter API |
+| external_vsphere_insecure | TRUE | string | "true", "false" | "true" | set to "true" if the host above uses a self-signed cert |
+| external_vsphere_user | TRUE | string | | | User name for vCenter with required privileges (Can also be specified with the `VSPHERE_USER` environment variable) |
+| external_vsphere_password | TRUE | string | | | Password for vCenter (Can also be specified with the `VSPHERE_PASSWORD` environment variable) |
+| external_vsphere_datacenter | TRUE | string | | | Datacenter name to use |
+| external_vsphere_kubernetes_cluster_id | TRUE | string | | "kubernetes-cluster-id" | Kubernetes cluster ID to use |
+| vsphere_csi_enabled | TRUE | boolean | | false | Enable vSphere CSI |
+
+Example configuration:
+
+```yml
+external_vsphere_vcenter_ip: "myvcenter.domain.com"
+external_vsphere_vcenter_port: "443"
+external_vsphere_insecure: "true"
+external_vsphere_user: "administrator@vsphere.local"
+external_vsphere_password: "K8s_admin"
+external_vsphere_datacenter: "DATACENTER_name"
+external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
+vsphere_csi_enabled: true
+```
+
+For a more fine-grained CSI setup, refer to the [vsphere-csi](/docs/vsphere-csi.md) documentation.
+
+### Deployment
+
+Once the configuration is set, you can execute the playbook again to apply the new configuration:
+
+```ShellSession
+cd kubespray
+ansible-playbook -i inventory/sample/hosts.ini -b -v cluster.yml
+```
+
+You'll find some useful examples [here](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#sample-manifests-to-test-csi-driver-functionality) to test your configuration.
+
+## In-tree vSphere cloud provider ([deprecated](https://cloud-provider-vsphere.sigs.k8s.io/concepts/in_tree_vs_out_of_tree.html))
+
+### Prerequisites (deprecated)
+
+You need at first to configure your vSphere environment by following the [official documentation](https://kubernetes.io/docs/getting-started-guides/vsphere/#vsphere-cloud-provider).
+
+After this step you should have:
+
+- UUID activated for each VM where Kubernetes will be deployed
+- A vSphere account with required privileges
+
+If you intend to leverage the [zone and region node labeling](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domain-beta-kubernetes-io-region), create a tag category for both the zone and region in vCenter. The tags can then be applied at the host, cluster, datacenter, or folder level, and the cloud provider will walk the hierarchy to extract and apply the labels to the Kubernetes nodes.
+
+### Kubespray configuration (deprecated)
+
+First you must define the cloud provider in `inventory/sample/group_vars/all.yml` and set it to `vsphere`.
+
+```yml
+cloud_provider: vsphere
+```
+
+Then, in the same file, you need to declare your vCenter credentials following the description below.
+
+| Variable | Required | Type | Choices | Default | Comment |
+|------------------------------|----------|---------|----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| vsphere_vcenter_ip | TRUE | string | | | IP/URL of the vCenter |
+| vsphere_vcenter_port | TRUE | integer | | | Port of the vCenter API. Commonly 443 |
+| vsphere_insecure | TRUE | integer | 1, 0 | | set to 1 if the host above uses a self-signed cert |
+| vsphere_user | TRUE | string | | | User name for vCenter with required privileges |
+| vsphere_password | TRUE | string | | | Password for vCenter |
+| vsphere_datacenter | TRUE | string | | | Datacenter name to use |
+| vsphere_datastore | TRUE | string | | | Datastore name to use |
+| vsphere_working_dir | TRUE | string | | | Working directory from the view "VMs and template" in the vCenter where VM are placed |
+| vsphere_scsi_controller_type | TRUE | string | buslogic, pvscsi, parallel | pvscsi | SCSI controller name. Commonly "pvscsi". |
+| vsphere_vm_uuid | FALSE | string | | | VM Instance UUID of virtual machine that host K8s master. Can be retrieved from instanceUuid property in VmConfigInfo, or as vc.uuid in VMX file or in `/sys/class/dmi/id/product_serial` (Optional, only used for Kubernetes <= 1.9.2) |
+| vsphere_public_network | FALSE | string | | Blank | Name of the network the VMs are joined to |
+| vsphere_resource_pool | FALSE | string | | Blank | Name of the Resource pool where the VMs are located (Optional, only used for Kubernetes >= 1.9.2) |
+| vsphere_zone_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/zone` label on nodes (Optional, only used for Kubernetes >= 1.12.0) |
+| vsphere_region_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/region` label on nodes (Optional, only used for Kubernetes >= 1.12.0) |
+
+Example configuration:
+
+```yml
+vsphere_vcenter_ip: "myvcenter.domain.com"
+vsphere_vcenter_port: 443
+vsphere_insecure: 1
+vsphere_user: "k8s@vsphere.local"
+vsphere_password: "K8s_admin"
+vsphere_datacenter: "DATACENTER_name"
+vsphere_datastore: "DATASTORE_name"
+vsphere_working_dir: "Docker_hosts"
+vsphere_scsi_controller_type: "pvscsi"
+vsphere_resource_pool: "K8s-Pool"
+```
+
+### Deployment (deprecated)
+
+Once the configuration is set, you can execute the playbook again to apply the new configuration:
+
+```ShellSession
+cd kubespray
+ansible-playbook -i inventory/sample/hosts.ini -b -v cluster.yml
+```
+
+You'll find some useful examples [here](https://github.com/kubernetes/examples/tree/master/staging/volumes/vsphere) to test your configuration.
--- /dev/null
+# Weave
+
+Weave 2.0.1 is supported by kubespray
+
+Weave uses [**consensus**](https://www.weave.works/docs/net/latest/ipam/##consensus) mode (default mode) and [**seed**](https://www.weave.works/docs/net/latest/ipam/#seed) mode.
+
+`Consensus` mode is best to use on static size cluster and `seed` mode is best to use on dynamic size cluster
+
+Weave encryption is supported for all communication
+
+* To use Weave encryption, specify a strong password (if no password, no encryption)
+
+```ShellSession
+# In file ./inventory/sample/group_vars/k8s_cluster.yml
+weave_password: EnterPasswordHere
+```
+
+This password is used to set an environment variable inside weave container.
+
+Weave is deployed by kubespray using a daemonSet
+
+* Check the status of Weave containers
+
+```ShellSession
+# From client
+kubectl -n kube-system get pods | grep weave
+# output
+weave-net-50wd2 2/2 Running 0 2m
+weave-net-js9rb 2/2 Running 0 2m
+```
+
+There must be as many pods as nodes (here kubernetes have 2 nodes so there are 2 weave pods).
+
+* Check status of weave (connection,encryption ...) for each node
+
+```ShellSession
+# On nodes
+curl http://127.0.0.1:6784/status
+# output on node1
+Version: 2.0.1 (up to date; next check at 2017/08/01 13:51:34)
+
+ Service: router
+ Protocol: weave 1..2
+ Name: fa:16:3e:b3:d6:b2(node1)
+ Encryption: enabled
+ PeerDiscovery: enabled
+ Targets: 2
+ Connections: 2 (1 established, 1 failed)
+ Peers: 2 (with 2 established connections)
+ TrustedSubnets: none
+
+ Service: ipam
+ Status: ready
+ Range: 10.233.64.0/18
+ DefaultSubnet: 10.233.64.0/18
+```
+
+* Check parameters of weave for each node
+
+```ShellSession
+# On nodes
+ps -aux | grep weaver
+# output on node1 (here its use seed mode)
+root 8559 0.2 3.0 365280 62700 ? Sl 08:25 0:00 /home/weave/weaver --name=fa:16:3e:b3:d6:b2 --port=6783 --datapath=datapath --host-root=/host --http-addr=127.0.0.1:6784 --status-addr=0.0.0.0:6782 --docker-api= --no-dns --db-prefix=/weavedb/weave-net --ipalloc-range=10.233.64.0/18 --nickname=node1 --ipalloc-init seed=fa:16:3e:b3:d6:b2,fa:16:3e:f0:50:53 --conn-limit=30 --expect-npc 192.168.208.28 192.168.208.19
+```
+
+## Consensus mode (default mode)
+
+This mode is best to use on static size cluster
+
+### Seed mode
+
+This mode is best to use on dynamic size cluster
+
+The seed mode also allows multi-clouds and hybrid on-premise/cloud clusters deployment.
+
+* Switch from consensus mode to seed/Observation mode
+
+See [weave ipam documentation](https://www.weave.works/docs/net/latest/tasks/ipam/ipam/) and use `weave_extra_args` to enable.
--- /dev/null
+---
+- name: Gather facts
+ hosts: k8s_cluster:etcd:calico_rr
+ gather_facts: False
+ tags: always
+ tasks:
+ - name: Gather minimal facts
+ setup:
+ gather_subset: '!all'
+
+ # filter match the following variables:
+ # ansible_default_ipv4
+ # ansible_default_ipv6
+ # ansible_all_ipv4_addresses
+ # ansible_all_ipv6_addresses
+ - name: Gather necessary facts (network)
+ setup:
+ gather_subset: '!all,!min,network'
+ filter: "ansible_*_ipv[46]*"
+
+ # filter match the following variables:
+ # ansible_memtotal_mb
+ # ansible_swaptotal_mb
+ - name: Gather necessary facts (hardware)
+ setup:
+ gather_subset: '!all,!min,hardware'
+ filter: "ansible_*total_mb"
--- /dev/null
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <title>Kubespray - Deploy a Production Ready Kubernetes Cluster</title>
+ <meta name="description" content="Deploy a Production Ready Kubernetes Cluster">
+ <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
+ <link rel="stylesheet" href="//unpkg.com/docsify-themeable/dist/css/theme-simple.css">
+ <style>
+ :root {
+ --base-font-size: 16px;
+ --theme-color: rgb(104, 118, 52);
+ --link-color: rgb(104, 118, 52);
+ --link-color--hover: rgb(137, 152, 100);
+ --sidebar-name-margin: 0;
+ --sidebar-name-padding: 0;
+ --code-font-size: .9em;
+ }
+ .sidebar > h1 {
+ margin-bottom: -.75em;
+ margin-top: .75em;
+ }
+ .markdown-section a code {
+ color: var(--link-color)!important;
+ }
+ .markdown-section code:not([class*="lang-"]):not([class*="language-"]) {
+ white-space: unset
+ }
+ </style>
+</head>
+<body>
+ <div id="app"></div>
+</body>
+<script>
+ window.$docsify = {
+ name: 'Kubespray',
+ loadSidebar: 'docs/_sidebar.md',
+ repo: 'https://github.com/kubernetes-sigs/kubespray',
+ auto2top: true,
+ logo: '/logo/logo-clear.png'
+ }
+</script>
+<script src="//unpkg.com/docsify/lib/docsify.min.js"></script>
+<script src="//unpkg.com/docsify/lib/plugins/search.min.js"></script>
+<script src="//unpkg.com/docsify/lib/plugins/ga.min.js"></script>
+
+</html>
--- /dev/null
+---
+## Directory where the binaries will be installed
+bin_dir: /usr/local/bin
+
+## The access_ip variable is used to define how other nodes should access
+## the node. This is used in flannel to allow other flannel nodes to see
+## this node for example. The access_ip is really useful AWS and Google
+## environments where the nodes are accessed remotely by the "public" ip,
+## but don't know about that address themselves.
+# access_ip: 1.1.1.1
+
+
+## External LB example config
+## apiserver_loadbalancer_domain_name: "elb.some.domain"
+# loadbalancer_apiserver:
+# address: 1.2.3.4
+# port: 1234
+
+## Internal loadbalancers for apiservers
+# loadbalancer_apiserver_localhost: true
+# valid options are "nginx" or "haproxy"
+# loadbalancer_apiserver_type: nginx # valid values "nginx" or "haproxy"
+
+## If the cilium is going to be used in strict mode, we can use the
+## localhost connection and not use the external LB. If this parameter is
+## not specified, the first node to connect to kubeapi will be used.
+# use_localhost_as_kubeapi_loadbalancer: true
+
+## Local loadbalancer should use this port
+## And must be set port 6443
+loadbalancer_apiserver_port: 6443
+
+## If loadbalancer_apiserver_healthcheck_port variable defined, enables proxy liveness check for nginx.
+loadbalancer_apiserver_healthcheck_port: 8081
+
+### OTHER OPTIONAL VARIABLES
+
+## Upstream dns servers
+# upstream_dns_servers:
+# - 8.8.8.8
+# - 8.8.4.4
+
+## There are some changes specific to the cloud providers
+## for instance we need to encapsulate packets with some network plugins
+## If set the possible values are either 'gce', 'aws', 'azure', 'openstack', 'vsphere', 'oci', or 'external'
+## When openstack is used make sure to source in the openstack credentials
+## like you would do when using openstack-client before starting the playbook.
+# cloud_provider:
+
+## When cloud_provider is set to 'external', you can set the cloud controller to deploy
+## Supported cloud controllers are: 'openstack', 'vsphere' and 'hcloud'
+## When openstack or vsphere are used make sure to source in the required fields
+# external_cloud_provider:
+
+## Set these proxy values in order to update package manager and docker daemon to use proxies
+# http_proxy: ""
+# https_proxy: ""
+
+## Refer to roles/kubespray-defaults/defaults/main.yml before modifying no_proxy
+# no_proxy: ""
+
+## Some problems may occur when downloading files over https proxy due to ansible bug
+## https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
+## SSL validation of get_url module. Note that kubespray will still be performing checksum validation.
+# download_validate_certs: False
+
+## If you need exclude all cluster nodes from proxy and other resources, add other resources here.
+# additional_no_proxy: ""
+
+## If you need to disable proxying of os package repositories but are still behind an http_proxy set
+## skip_http_proxy_on_os_packages to true
+## This will cause kubespray not to set proxy environment in /etc/yum.conf for centos and in /etc/apt/apt.conf for debian/ubuntu
+## Special information for debian/ubuntu - you have to set the no_proxy variable, then apt package will install from your source of wish
+# skip_http_proxy_on_os_packages: false
+
+## Since workers are included in the no_proxy variable by default, docker engine will be restarted on all nodes (all
+## pods will restart) when adding or removing workers. To override this behaviour by only including master nodes in the
+## no_proxy variable, set below to true:
+no_proxy_exclude_workers: false
+
+## Certificate Management
+## This setting determines whether certs are generated via scripts.
+## Chose 'none' if you provide your own certificates.
+## Option is "script", "none"
+# cert_management: script
+
+## Set to true to allow pre-checks to fail and continue deployment
+# ignore_assert_errors: false
+
+## The read-only port for the Kubelet to serve on with no authentication/authorization. Uncomment to enable.
+# kube_read_only_port: 10255
+
+## Set true to download and cache container
+# download_container: true
+
+## Deploy container engine
+# Set false if you want to deploy container engine manually.
+# deploy_container_engine: true
+
+## Red Hat Enterprise Linux subscription registration
+## Add either RHEL subscription Username/Password or Organization ID/Activation Key combination
+## Update RHEL subscription purpose usage, role and SLA if necessary
+# rh_subscription_username: ""
+# rh_subscription_password: ""
+# rh_subscription_org_id: ""
+# rh_subscription_activation_key: ""
+# rh_subscription_usage: "Development"
+# rh_subscription_role: "Red Hat Enterprise Server"
+# rh_subscription_sla: "Self-Support"
+
+## Check if access_ip responds to ping. Set false if your firewall blocks ICMP.
+# ping_access_ip: true
+
+# sysctl_file_path to add sysctl conf to
+# sysctl_file_path: "/etc/sysctl.d/99-sysctl.conf"
+
+## Variables for webhook token auth https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication
+kube_webhook_token_auth: false
+kube_webhook_token_auth_url_skip_tls_verify: false
+# kube_webhook_token_auth_url: https://...
+## base64-encoded string of the webhook's CA certificate
+# kube_webhook_token_auth_ca_data: "LS0t..."
+
+## NTP Settings
+# Start the ntpd or chrony service and enable it at system boot.
+ntp_enabled: false
+ntp_manage_config: false
+ntp_servers:
+ - "0.pool.ntp.org iburst"
+ - "1.pool.ntp.org iburst"
+ - "2.pool.ntp.org iburst"
+ - "3.pool.ntp.org iburst"
+
+## Used to control no_log attribute
+unsafe_show_logs: false
--- /dev/null
+## To use AWS EBS CSI Driver to provision volumes, uncomment the first value
+## and configure the parameters below
+# aws_ebs_csi_enabled: true
+# aws_ebs_csi_enable_volume_scheduling: true
+# aws_ebs_csi_enable_volume_snapshot: false
+# aws_ebs_csi_enable_volume_resizing: false
+# aws_ebs_csi_controller_replicas: 1
+# aws_ebs_csi_plugin_image_tag: latest
+# aws_ebs_csi_extra_volume_tags: "Owner=owner,Team=team,Environment=environment'
--- /dev/null
+## When azure is used, you need to also set the following variables.
+## see docs/azure.md for details on how to get these values
+
+# azure_cloud:
+# azure_tenant_id:
+# azure_subscription_id:
+# azure_aad_client_id:
+# azure_aad_client_secret:
+# azure_resource_group:
+# azure_location:
+# azure_subnet_name:
+# azure_security_group_name:
+# azure_security_group_resource_group:
+# azure_vnet_name:
+# azure_vnet_resource_group:
+# azure_route_table_name:
+# azure_route_table_resource_group:
+# supported values are 'standard' or 'vmss'
+# azure_vmtype: standard
+
+## Azure Disk CSI credentials and parameters
+## see docs/azure-csi.md for details on how to get these values
+
+# azure_csi_tenant_id:
+# azure_csi_subscription_id:
+# azure_csi_aad_client_id:
+# azure_csi_aad_client_secret:
+# azure_csi_location:
+# azure_csi_resource_group:
+# azure_csi_vnet_name:
+# azure_csi_vnet_resource_group:
+# azure_csi_subnet_name:
+# azure_csi_security_group_name:
+# azure_csi_use_instance_metadata:
+# azure_csi_tags: "Owner=owner,Team=team,Environment=environment'
+
+## To enable Azure Disk CSI, uncomment below
+# azure_csi_enabled: true
+# azure_csi_controller_replicas: 1
+# azure_csi_plugin_image_tag: latest
--- /dev/null
+---
+# Please see roles/container-engine/containerd/defaults/main.yml for more configuration options
+
+# containerd_storage_dir: "/var/lib/containerd"
+# containerd_state_dir: "/run/containerd"
+# containerd_oom_score: 0
+
+# containerd_default_runtime: "runc"
+# containerd_snapshotter: "native"
+
+# containerd_runc_runtime:
+# name: runc
+# type: "io.containerd.runc.v2"
+# engine: ""
+# root: ""
+
+# containerd_additional_runtimes:
+# Example for Kata Containers as additional runtime:
+# - name: kata
+# type: "io.containerd.kata.v2"
+# engine: ""
+# root: ""
+
+# containerd_grpc_max_recv_message_size: 16777216
+# containerd_grpc_max_send_message_size: 16777216
+
+# containerd_debug_level: "info"
+
+# containerd_metrics_address: ""
+
+# containerd_metrics_grpc_histogram: false
+
+## An obvious use case is allowing insecure-registry access to self hosted registries.
+## Can be ipaddress and domain_name.
+## example define mirror.registry.io or 172.19.16.11:5000
+## set "name": "url". insecure url must be started http://
+## Port number is also needed if the default HTTPS port is not used.
+# containerd_insecure_registries:
+# "localhost": "http://127.0.0.1"
+# "172.19.16.11:5000": "http://172.19.16.11:5000"
+
+# containerd_registries:
+# "docker.io": "https://registry-1.docker.io"
+
+# containerd_max_container_log_line_size: -1
+
+# containerd_registry_auth:
+# - registry: 10.0.0.2:5000
+# username: user
+# password: pass
--- /dev/null
+## Does coreos need auto upgrade, default is true
+# coreos_auto_upgrade: true
--- /dev/null
+# crio_insecure_registries:
+# - 10.0.0.2:5000
+# crio_registry_auth:
+# - registry: 10.0.0.2:5000
+# username: user
+# password: pass
--- /dev/null
+---
+## Uncomment this if you want to force overlay/overlay2 as docker storage driver
+## Please note that overlay2 is only supported on newer kernels
+# docker_storage_options: -s overlay2
+
+## Enable docker_container_storage_setup, it will configure devicemapper driver on Centos7 or RedHat7.
+docker_container_storage_setup: false
+
+## It must be define a disk path for docker_container_storage_setup_devs.
+## Otherwise docker-storage-setup will be executed incorrectly.
+# docker_container_storage_setup_devs: /dev/vdb
+
+## Uncomment this if you want to change the Docker Cgroup driver (native.cgroupdriver)
+## Valid options are systemd or cgroupfs, default is systemd
+# docker_cgroup_driver: systemd
+
+## Only set this if you have more than 3 nameservers:
+## If true Kubespray will only use the first 3, otherwise it will fail
+docker_dns_servers_strict: false
+
+# Path used to store Docker data
+docker_daemon_graph: "/var/lib/docker"
+
+## Used to set docker daemon iptables options to true
+docker_iptables_enabled: "false"
+
+# Docker log options
+# Rotate container stderr/stdout logs at 50m and keep last 5
+docker_log_opts: "--log-opt max-size=50m --log-opt max-file=5"
+
+# define docker bin_dir
+docker_bin_dir: "/usr/bin"
+
+# keep docker packages after installation; speeds up repeated ansible provisioning runs when '1'
+# kubespray deletes the docker package on each run, so caching the package makes sense
+docker_rpm_keepcache: 1
+
+## An obvious use case is allowing insecure-registry access to self hosted registries.
+## Can be ipaddress and domain_name.
+## example define 172.19.16.11 or mirror.registry.io
+# docker_insecure_registries:
+# - mirror.registry.io
+# - 172.19.16.11
+
+## Add other registry,example China registry mirror.
+# docker_registry_mirrors:
+# - https://registry.docker-cn.com
+# - https://mirror.aliyuncs.com
+
+## If non-empty will override default system MountFlags value.
+## This option takes a mount propagation flag: shared, slave
+## or private, which control whether mounts in the file system
+## namespace set up for docker will receive or propagate mounts
+## and unmounts. Leave empty for system default
+# docker_mount_flags:
+
+## A string of extra options to pass to the docker daemon.
+## This string should be exactly as you wish it to appear.
+# docker_options: ""
--- /dev/null
+---
+## Directory where etcd data stored
+etcd_data_dir: /var/lib/etcd
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
+## Kubeadm etcd deployment is experimental and only available for new deployments
+## If this is not set, container manager will be inherited from the Kubespray defaults
+## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
+## Also this makes possible to use different container manager for etcd nodes.
+# container_manager: containerd
+
+## Settings for etcd deployment type
+# Set this to docker if you are using container_manager: docker
+etcd_deployment_type: host
\ No newline at end of file
--- /dev/null
+## GCP compute Persistent Disk CSI Driver credentials and parameters
+## See docs/gcp-pd-csi.md for information about the implementation
+
+## Specify the path to the file containing the service account credentials
+# gcp_pd_csi_sa_cred_file: "/my/safe/credentials/directory/cloud-sa.json"
+
+## To enable GCP Persistent Disk CSI driver, uncomment below
+# gcp_pd_csi_enabled: true
+# gcp_pd_csi_controller_replicas: 1
+# gcp_pd_csi_driver_image_tag: "v0.7.0-gke.0"
--- /dev/null
+## Values for the external Hcloud Cloud Controller
+# external_hcloud_cloud:
+# hcloud_api_token: ""
+# token_secret_name: hcloud
+# with_networks: false # Use the hcloud controller-manager with networks support https://github.com/hetznercloud/hcloud-cloud-controller-manager#networks-support
+# service_account_name: cloud-controller-manager
+#
+# controller_image_tag: "latest"
+# ## A dictionary of extra arguments to add to the openstack cloud controller manager daemonset
+# ## Format:
+# ## external_hcloud_cloud.controller_extra_args:
+# ## arg1: "value1"
+# ## arg2: "value2"
+# controller_extra_args: {}
--- /dev/null
+## When Oracle Cloud Infrastructure is used, set these variables
+# oci_private_key:
+# oci_region_id:
+# oci_tenancy_id:
+# oci_user_id:
+# oci_user_fingerprint:
+# oci_compartment_id:
+# oci_vnc_id:
+# oci_subnet1_id:
+# oci_subnet2_id:
+## Override these default/optional behaviors if you wish
+# oci_security_list_management: All
+## If you would like the controller to manage specific lists per subnet. This is a mapping of subnet ocids to security list ocids. Below are examples.
+# oci_security_lists:
+# ocid1.subnet.oc1.phx.aaaaaaaasa53hlkzk6nzksqfccegk2qnkxmphkblst3riclzs4rhwg7rg57q: ocid1.securitylist.oc1.iad.aaaaaaaaqti5jsfvyw6ejahh7r4okb2xbtuiuguswhs746mtahn72r7adt7q
+# ocid1.subnet.oc1.phx.aaaaaaaahuxrgvs65iwdz7ekwgg3l5gyah7ww5klkwjcso74u3e4i64hvtvq: ocid1.securitylist.oc1.iad.aaaaaaaaqti5jsfvyw6ejahh7r4okb2xbtuiuguswhs746mtahn72r7adt7q
+## If oci_use_instance_principals is true, you do not need to set the region, tenancy, user, key, passphrase, or fingerprint
+# oci_use_instance_principals: false
+# oci_cloud_controller_version: 0.6.0
+## If you would like to control OCI query rate limits for the controller
+# oci_rate_limit:
+# rate_limit_qps_read:
+# rate_limit_qps_write:
+# rate_limit_bucket_read:
+# rate_limit_bucket_write:
+## Other optional variables
+# oci_cloud_controller_pull_source: (default iad.ocir.io/oracle/cloud-provider-oci)
+# oci_cloud_controller_pull_secret: (name of pull secret to use if you define your own mirror above)
--- /dev/null
+---
+## Global Offline settings
+### Private Container Image Registry
+# registry_host: "myprivateregisry.com"
+# files_repo: "http://myprivatehttpd"
+### If using CentOS, RedHat, AlmaLinux or Fedora
+# yum_repo: "http://myinternalyumrepo"
+### If using Debian
+# debian_repo: "http://myinternaldebianrepo"
+### If using Ubuntu
+# ubuntu_repo: "http://myinternalubunturepo"
+
+## Container Registry overrides
+# kube_image_repo: "{{ registry_host }}"
+# gcr_image_repo: "{{ registry_host }}"
+# github_image_repo: "{{ registry_host }}"
+# docker_image_repo: "{{ registry_host }}"
+# quay_image_repo: "{{ registry_host }}"
+
+## Kubernetes components
+# kubeadm_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
+# kubectl_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
+# kubelet_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
+
+## CNI Plugins
+# cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
+
+## cri-tools
+# crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+
+## [Optional] etcd: only if you **DON'T** use etcd_deployment=host
+# etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
+
+# [Optional] Calico: If using Calico network plugin
+# calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
+# calicoctl_alternate_download_url: "{{ files_repo }}/github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
+# [Optional] Calico with kdd: If using Calico network plugin with kdd datastore
+# calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"
+
+# [Optional] Flannel: If using Falnnel network plugin
+# flannel_cni_download_url: "{{ files_repo }}/kubernetes/flannel/{{ flannel_cni_version }}/flannel-{{ image_arch }}"
+
+# [Optional] helm: only if you set helm_enabled: true
+# helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
+
+# [Optional] crun: only if you set crun_enabled: true
+# crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
+
+# [Optional] kata: only if you set kata_containers_enabled: true
+# kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"
+
+# [Optional] cri-dockerd: only if you set container_manager: docker
+# cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz"
+
+# [Optional] cri-o: only if you set container_manager: crio
+# crio_download_base: "download.opensuse.org/repositories/devel:kubic:libcontainers:stable"
+# crio_download_crio: "http://{{ crio_download_base }}:/cri-o:/"
+
+# [Optional] runc,containerd: only if you set container_runtime: containerd
+# runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}"
+# containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
+# nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+
+## CentOS/Redhat/AlmaLinux
+### For EL7, base and extras repo must be available, for EL8, baseos and appstream
+### By default we enable those repo automatically
+# rhel_enable_repos: false
+### Docker / Containerd
+# docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
+# docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+## Fedora
+### Docker
+# docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
+# docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
+# containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+## Debian
+### Docker
+# docker_debian_repo_base_url: "{{ debian_repo }}/docker-ce"
+# docker_debian_repo_gpgkey: "{{ debian_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_debian_repo_base_url: "{{ ubuntu_repo }}/containerd"
+# containerd_debian_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+# containerd_debian_repo_repokey: 'YOURREPOKEY'
+
+## Ubuntu
+### Docker
+# docker_ubuntu_repo_base_url: "{{ ubuntu_repo }}/docker-ce"
+# docker_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_ubuntu_repo_base_url: "{{ ubuntu_repo }}/containerd"
+# containerd_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+# containerd_ubuntu_repo_repokey: 'YOURREPOKEY'
--- /dev/null
+## When OpenStack is used, Cinder version can be explicitly specified if autodetection fails (Fixed in 1.9: https://github.com/kubernetes/kubernetes/issues/50461)
+# openstack_blockstorage_version: "v1/v2/auto (default)"
+# openstack_blockstorage_ignore_volume_az: yes
+## When OpenStack is used, if LBaaSv2 is available you can enable it with the following 2 variables.
+# openstack_lbaas_enabled: True
+# openstack_lbaas_subnet_id: "Neutron subnet ID (not network ID) to create LBaaS VIP"
+## To enable automatic floating ip provisioning, specify a subnet.
+# openstack_lbaas_floating_network_id: "Neutron network ID (not subnet ID) to get floating IP from, disabled by default"
+## Override default LBaaS behavior
+# openstack_lbaas_use_octavia: False
+# openstack_lbaas_method: "ROUND_ROBIN"
+# openstack_lbaas_provider: "haproxy"
+# openstack_lbaas_create_monitor: "yes"
+# openstack_lbaas_monitor_delay: "1m"
+# openstack_lbaas_monitor_timeout: "30s"
+# openstack_lbaas_monitor_max_retries: "3"
+
+## Values for the external OpenStack Cloud Controller
+# external_openstack_lbaas_network_id: "Neutron network ID to create LBaaS VIP"
+# external_openstack_lbaas_subnet_id: "Neutron subnet ID to create LBaaS VIP"
+# external_openstack_lbaas_floating_network_id: "Neutron network ID to get floating IP from"
+# external_openstack_lbaas_floating_subnet_id: "Neutron subnet ID to get floating IP from"
+# external_openstack_lbaas_method: "ROUND_ROBIN"
+# external_openstack_lbaas_provider: "octavia"
+# external_openstack_lbaas_create_monitor: false
+# external_openstack_lbaas_monitor_delay: "1m"
+# external_openstack_lbaas_monitor_timeout: "30s"
+# external_openstack_lbaas_monitor_max_retries: "3"
+# external_openstack_lbaas_manage_security_groups: false
+# external_openstack_lbaas_internal_lb: false
+# external_openstack_network_ipv6_disabled: false
+# external_openstack_network_internal_networks: []
+# external_openstack_network_public_networks: []
+# external_openstack_metadata_search_order: "configDrive,metadataService"
+
+## Application credentials to authenticate against Keystone API
+## Those settings will take precedence over username and password that might be set your environment
+## All of them are required
+# external_openstack_application_credential_name:
+# external_openstack_application_credential_id:
+# external_openstack_application_credential_secret:
+
+## The tag of the external OpenStack Cloud Controller image
+# external_openstack_cloud_controller_image_tag: "latest"
+
+## To use Cinder CSI plugin to provision volumes set this value to true
+## Make sure to source in the openstack credentials
+# cinder_csi_enabled: true
+# cinder_csi_controller_replicas: 1
--- /dev/null
+## Repo for UpClouds csi-driver: https://github.com/UpCloudLtd/upcloud-csi
+## To use UpClouds CSI plugin to provision volumes set this value to true
+## Remember to set UPCLOUD_USERNAME and UPCLOUD_PASSWORD
+# upcloud_csi_enabled: true
+# upcloud_csi_controller_replicas: 1
+## Override used image tags
+# upcloud_csi_provisioner_image_tag: "v3.1.0"
+# upcloud_csi_attacher_image_tag: "v3.4.0"
+# upcloud_csi_resizer_image_tag: "v1.4.0"
+# upcloud_csi_plugin_image_tag: "v0.2.1"
+# upcloud_csi_node_image_tag: "v2.5.0"
+# upcloud_tolerations: []
+## Storage class options
+# expand_persistent_volumes: true
+# parameters:
+# tier: maxiops # or hdd
+# storage_classes:
+# - name: standard
+# is_default: true
--- /dev/null
+## Values for the external vSphere Cloud Provider
+# external_vsphere_vcenter_ip: "myvcenter.domain.com"
+# external_vsphere_vcenter_port: "443"
+# external_vsphere_insecure: "true"
+# external_vsphere_user: "administrator@vsphere.local" # Can also be set via the `VSPHERE_USER` environment variable
+# external_vsphere_password: "K8s_admin" # Can also be set via the `VSPHERE_PASSWORD` environment variable
+# external_vsphere_datacenter: "DATACENTER_name"
+# external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
+
+## Vsphere version where located VMs
+# external_vsphere_version: "6.7u3"
+
+## Tags for the external vSphere Cloud Provider images
+## gcr.io/cloud-provider-vsphere/cpi/release/manager
+# external_vsphere_cloud_controller_image_tag: "latest"
+## gcr.io/cloud-provider-vsphere/csi/release/syncer
+# vsphere_syncer_image_tag: "v2.5.1"
+## registry.k8s.io/sig-storage/csi-attacher
+# vsphere_csi_attacher_image_tag: "v3.4.0"
+## gcr.io/cloud-provider-vsphere/csi/release/driver
+# vsphere_csi_controller: "v2.5.1"
+## registry.k8s.io/sig-storage/livenessprobe
+# vsphere_csi_liveness_probe_image_tag: "v2.6.0"
+## registry.k8s.io/sig-storage/csi-provisioner
+# vsphere_csi_provisioner_image_tag: "v3.1.0"
+## registry.k8s.io/sig-storage/csi-resizer
+## makes sense only for vSphere version >=7.0
+# vsphere_csi_resizer_tag: "v1.3.0"
+
+## To use vSphere CSI plugin to provision volumes set this value to true
+# vsphere_csi_enabled: true
+# vsphere_csi_controller_replicas: 1
--- /dev/null
+---
+## Etcd auto compaction retention for mvcc key value store in hour
+# etcd_compaction_retention: 0
+
+## Set level of detail for etcd exported metrics, specify 'extensive' to include histogram metrics.
+# etcd_metrics: basic
+
+## Etcd is restricted by default to 512M on systems under 4GB RAM, 512MB is not enough for much more than testing.
+## Set this if your etcd nodes have less than 4GB but you want more RAM for etcd. Set to 0 for unrestricted RAM.
+## This value is only relevant when deploying etcd with `etcd_deployment_type: docker`
+# etcd_memory_limit: "512M"
+
+## Etcd has a default of 2G for its space quota. If you put a value in etcd_memory_limit which is less than
+## etcd_quota_backend_bytes, you may encounter out of memory terminations of the etcd cluster. Please check
+## etcd documentation for more information.
+# 8G is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.
+# etcd_quota_backend_bytes: "2147483648"
+
+# Maximum client request size in bytes the server will accept.
+# etcd is designed to handle small key value pairs typical for metadata.
+# Larger requests will work, but may increase the latency of other requests
+# etcd_max_request_bytes: "1572864"
+
+### ETCD: disable peer client cert authentication.
+# This affects ETCD_PEER_CLIENT_CERT_AUTH variable
+# etcd_peer_client_auth: true
--- /dev/null
+---
+# Kubernetes dashboard
+# RBAC required. see docs/getting-started.md for access details.
+# dashboard_enabled: false
+
+# Helm deployment
+helm_enabled: false
+
+# Registry deployment
+registry_enabled: false
+# registry_namespace: kube-system
+# registry_storage_class: ""
+# registry_disk_size: "10Gi"
+
+# Metrics Server deployment
+metrics_server_enabled: false
+# metrics_server_container_port: 4443
+# metrics_server_kubelet_insecure_tls: true
+# metrics_server_metric_resolution: 15s
+# metrics_server_kubelet_preferred_address_types: "InternalIP,ExternalIP,Hostname"
+
+# Rancher Local Path Provisioner
+local_path_provisioner_enabled: false
+# local_path_provisioner_namespace: "local-path-storage"
+# local_path_provisioner_storage_class: "local-path"
+# local_path_provisioner_reclaim_policy: Delete
+# local_path_provisioner_claim_root: /opt/local-path-provisioner/
+# local_path_provisioner_debug: false
+# local_path_provisioner_image_repo: "rancher/local-path-provisioner"
+# local_path_provisioner_image_tag: "v0.0.22"
+# local_path_provisioner_helper_image_repo: "busybox"
+# local_path_provisioner_helper_image_tag: "latest"
+
+# Local volume provisioner deployment
+local_volume_provisioner_enabled: false
+# local_volume_provisioner_namespace: kube-system
+# local_volume_provisioner_nodelabels:
+# - kubernetes.io/hostname
+# - topology.kubernetes.io/region
+# - topology.kubernetes.io/zone
+# local_volume_provisioner_storage_classes:
+# local-storage:
+# host_dir: /mnt/disks
+# mount_dir: /mnt/disks
+# volume_mode: Filesystem
+# fs_type: ext4
+# fast-disks:
+# host_dir: /mnt/fast-disks
+# mount_dir: /mnt/fast-disks
+# block_cleaner_command:
+# - "/scripts/shred.sh"
+# - "2"
+# volume_mode: Filesystem
+# fs_type: ext4
+# local_volume_provisioner_tolerations:
+# - effect: NoSchedule
+# operator: Exists
+
+# CSI Volume Snapshot Controller deployment, set this to true if your CSI is able to manage snapshots
+# currently, setting cinder_csi_enabled=true would automatically enable the snapshot controller
+# Longhorn is an extenal CSI that would also require setting this to true but it is not included in kubespray
+# csi_snapshot_controller_enabled: false
+# csi snapshot namespace
+# snapshot_controller_namespace: kube-system
+
+# CephFS provisioner deployment
+cephfs_provisioner_enabled: false
+# cephfs_provisioner_namespace: "cephfs-provisioner"
+# cephfs_provisioner_cluster: ceph
+# cephfs_provisioner_monitors: "172.24.0.1:6789,172.24.0.2:6789,172.24.0.3:6789"
+# cephfs_provisioner_admin_id: admin
+# cephfs_provisioner_secret: secret
+# cephfs_provisioner_storage_class: cephfs
+# cephfs_provisioner_reclaim_policy: Delete
+# cephfs_provisioner_claim_root: /volumes
+# cephfs_provisioner_deterministic_names: true
+
+# RBD provisioner deployment
+rbd_provisioner_enabled: false
+# rbd_provisioner_namespace: rbd-provisioner
+# rbd_provisioner_replicas: 2
+# rbd_provisioner_monitors: "172.24.0.1:6789,172.24.0.2:6789,172.24.0.3:6789"
+# rbd_provisioner_pool: kube
+# rbd_provisioner_admin_id: admin
+# rbd_provisioner_secret_name: ceph-secret-admin
+# rbd_provisioner_secret: ceph-key-admin
+# rbd_provisioner_user_id: kube
+# rbd_provisioner_user_secret_name: ceph-secret-user
+# rbd_provisioner_user_secret: ceph-key-user
+# rbd_provisioner_user_secret_namespace: rbd-provisioner
+# rbd_provisioner_fs_type: ext4
+# rbd_provisioner_image_format: "2"
+# rbd_provisioner_image_features: layering
+# rbd_provisioner_storage_class: rbd
+# rbd_provisioner_reclaim_policy: Delete
+
+# Nginx ingress controller deployment
+ingress_nginx_enabled: false
+# ingress_nginx_host_network: false
+ingress_publish_status_address: ""
+# ingress_nginx_nodeselector:
+# kubernetes.io/os: "linux"
+# ingress_nginx_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# ingress_nginx_namespace: "ingress-nginx"
+# ingress_nginx_insecure_port: 80
+# ingress_nginx_secure_port: 443
+# ingress_nginx_configmap:
+# map-hash-bucket-size: "128"
+# ssl-protocols: "TLSv1.2 TLSv1.3"
+# ingress_nginx_configmap_tcp_services:
+# 9000: "default/example-go:8080"
+# ingress_nginx_configmap_udp_services:
+# 53: "kube-system/coredns:53"
+# ingress_nginx_extra_args:
+# - --default-ssl-certificate=default/foo-tls
+# ingress_nginx_termination_grace_period_seconds: 300
+# ingress_nginx_class: nginx
+
+# ALB ingress controller deployment
+ingress_alb_enabled: false
+# alb_ingress_aws_region: "us-east-1"
+# alb_ingress_restrict_scheme: "false"
+# Enables logging on all outbound requests sent to the AWS API.
+# If logging is desired, set to true.
+# alb_ingress_aws_debug: "false"
+
+# Cert manager deployment
+cert_manager_enabled: false
+# cert_manager_namespace: "cert-manager"
+# cert_manager_tolerations:
+# - key: node-role.kubernetes.io/master
+# effect: NoSchedule
+# - key: node-role.kubernetes.io/control-plane
+# effect: NoSchedule
+# cert_manager_affinity:
+# nodeAffinity:
+# preferredDuringSchedulingIgnoredDuringExecution:
+# - weight: 100
+# preference:
+# matchExpressions:
+# - key: node-role.kubernetes.io/control-plane
+# operator: In
+# values:
+# - ""
+# cert_manager_nodeselector:
+# kubernetes.io/os: "linux"
+
+# cert_manager_trusted_internal_ca: |
+# -----BEGIN CERTIFICATE-----
+# [REPLACE with your CA certificate]
+# -----END CERTIFICATE-----
+# cert_manager_leader_election_namespace: kube-system
+
+# MetalLB deployment
+metallb_enabled: false
+metallb_speaker_enabled: true
+# metallb_ip_range:
+# - "10.5.0.50-10.5.0.99"
+# metallb_pool_name: "loadbalanced"
+# metallb_auto_assign: true
+# metallb_avoid_buggy_ips: false
+# metallb_speaker_nodeselector:
+# kubernetes.io/os: "linux"
+# metallb_controller_nodeselector:
+# kubernetes.io/os: "linux"
+# metallb_speaker_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# metallb_controller_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# metallb_version: v0.12.1
+# metallb_protocol: "layer2"
+# metallb_port: "7472"
+# metallb_memberlist_port: "7946"
+# metallb_additional_address_pools:
+# kube_service_pool:
+# ip_range:
+# - "10.5.1.50-10.5.1.99"
+# protocol: "layer2"
+# auto_assign: false
+# avoid_buggy_ips: false
+# metallb_protocol: "bgp"
+# metallb_peers:
+# - peer_address: 192.0.2.1
+# peer_asn: 64512
+# my_asn: 4200000000
+# - peer_address: 192.0.2.2
+# peer_asn: 64513
+# my_asn: 4200000000
+
+argocd_enabled: false
+# argocd_version: v2.4.7
+# argocd_namespace: argocd
+# Default password:
+# - https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli
+# ---
+# The initial password is autogenerated to be the pod name of the Argo CD API server. This can be retrieved with the command:
+# kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
+# ---
+# Use the following var to set admin password
+# argocd_admin_password: "password"
+
+# The plugin manager for kubectl
+krew_enabled: false
+krew_root_dir: "/usr/local/krew"
--- /dev/null
+---
+# Kubernetes configuration dirs and system namespace.
+# Those are where all the additional config stuff goes
+# the kubernetes normally puts in /srv/kubernetes.
+# This puts them in a sane location and namespace.
+# Editing those values will almost surely break something.
+kube_config_dir: /etc/kubernetes
+kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
+kube_manifest_dir: "{{ kube_config_dir }}/manifests"
+
+# This is where all the cert scripts and certs will be located
+kube_cert_dir: "{{ kube_config_dir }}/ssl"
+
+# This is where all of the bearer tokens will be stored
+kube_token_dir: "{{ kube_config_dir }}/tokens"
+
+kube_api_anonymous_auth: true
+
+## Change this to use another Kubernetes version, e.g. a current beta release
+kube_version: v1.24.4
+
+# Where the binaries will be downloaded.
+# Note: ensure that you've enough disk space (about 1G)
+local_release_dir: "/tmp/releases"
+# Random shifts for retrying failed ops like pushing/downloading
+retry_stagger: 5
+
+# This is the user that owns tha cluster installation.
+kube_owner: kube
+
+# This is the group that the cert creation scripts chgrp the
+# cert files to. Not really changeable...
+kube_cert_group: kube-cert
+
+# Cluster Loglevel configuration
+kube_log_level: 2
+
+# Directory where credentials will be stored
+credentials_dir: "{{ inventory_dir }}/credentials"
+
+## It is possible to activate / deactivate selected authentication methods (oidc, static token auth)
+# kube_oidc_auth: false
+# kube_token_auth: false
+
+
+## Variables for OpenID Connect Configuration https://kubernetes.io/docs/admin/authentication/
+## To use OpenID you have to deploy additional an OpenID Provider (e.g Dex, Keycloak, ...)
+
+# kube_oidc_url: https:// ...
+# kube_oidc_client_id: kubernetes
+## Optional settings for OIDC
+# kube_oidc_ca_file: "{{ kube_cert_dir }}/ca.pem"
+# kube_oidc_username_claim: sub
+# kube_oidc_username_prefix: 'oidc:'
+# kube_oidc_groups_claim: groups
+# kube_oidc_groups_prefix: 'oidc:'
+
+## Variables to control webhook authn/authz
+# kube_webhook_token_auth: false
+# kube_webhook_token_auth_url: https://...
+# kube_webhook_token_auth_url_skip_tls_verify: false
+
+## For webhook authorization, authorization_modes must include Webhook
+# kube_webhook_authorization: false
+# kube_webhook_authorization_url: https://...
+# kube_webhook_authorization_url_skip_tls_verify: false
+
+# Choose network plugin (cilium, calico, kube-ovn, weave or flannel. Use cni for generic cni plugin)
+# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
+kube_network_plugin: calico
+
+# Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni
+kube_network_plugin_multus: false
+
+# Kubernetes internal network for services, unused block of space.
+kube_service_addresses: 10.233.0.0/18
+
+# internal network. When used, it will assign IP
+# addresses from this range to individual pods.
+# This network must be unused in your network infrastructure!
+kube_pods_subnet: 10.233.64.0/18
+
+# internal network node size allocation (optional). This is the size allocated
+# to each node for pod IP address allocation. Note that the number of pods per node is
+# also limited by the kubelet_max_pods variable which defaults to 110.
+#
+# Example:
+# Up to 64 nodes and up to 254 or kubelet_max_pods (the lowest of the two) pods per node:
+# - kube_pods_subnet: 10.233.64.0/18
+# - kube_network_node_prefix: 24
+# - kubelet_max_pods: 110
+#
+# Example:
+# Up to 128 nodes and up to 126 or kubelet_max_pods (the lowest of the two) pods per node:
+# - kube_pods_subnet: 10.233.64.0/18
+# - kube_network_node_prefix: 25
+# - kubelet_max_pods: 110
+kube_network_node_prefix: 24
+
+# Configure Dual Stack networking (i.e. both IPv4 and IPv6)
+enable_dual_stack_networks: false
+
+# Kubernetes internal network for IPv6 services, unused block of space.
+# This is only used if enable_dual_stack_networks is set to true
+# This provides 4096 IPv6 IPs
+kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
+
+# Internal network. When used, it will assign IPv6 addresses from this range to individual pods.
+# This network must not already be in your network infrastructure!
+# This is only used if enable_dual_stack_networks is set to true.
+# This provides room for 256 nodes with 254 pods per node.
+kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+
+# IPv6 subnet size allocated to each for pods.
+# This is only used if enable_dual_stack_networks is set to true
+# This provides room for 254 pods per node.
+kube_network_node_prefix_ipv6: 120
+
+# The port the API Server will be listening on.
+kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
+kube_apiserver_port: 6443 # (https)
+
+# Kube-proxy proxyMode configuration.
+# Can be ipvs, iptables
+kube_proxy_mode: ipvs
+
+# configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
+# must be set to true for MetalLB to work
+kube_proxy_strict_arp: false
+
+# A string slice of values which specify the addresses to use for NodePorts.
+# Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32).
+# The default empty string slice ([]) means to use all local addresses.
+# kube_proxy_nodeport_addresses_cidr is retained for legacy config
+kube_proxy_nodeport_addresses: >-
+ {%- if kube_proxy_nodeport_addresses_cidr is defined -%}
+ [{{ kube_proxy_nodeport_addresses_cidr }}]
+ {%- else -%}
+ []
+ {%- endif -%}
+
+# If non-empty, will use this string as identification instead of the actual hostname
+# kube_override_hostname: >-
+# {%- if cloud_provider is defined and cloud_provider in [ 'aws' ] -%}
+# {%- else -%}
+# {{ inventory_hostname }}
+# {%- endif -%}
+
+## Encrypting Secret Data at Rest
+kube_encrypt_secret_data: false
+
+# Graceful Node Shutdown (Kubernetes >= 1.21.0), see https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/
+# kubelet_shutdown_grace_period had to be greater than kubelet_shutdown_grace_period_critical_pods to allow
+# non-critical podsa to also terminate gracefully
+# kubelet_shutdown_grace_period: 60s
+# kubelet_shutdown_grace_period_critical_pods: 20s
+
+# DNS configuration.
+# Kubernetes cluster name, also will be used as DNS domain
+cluster_name: cluster.local
+# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
+ndots: 2
+# Can be coredns, coredns_dual, manual or none
+dns_mode: coredns
+# Set manual server if using a custom cluster DNS server
+# manual_dns_server: 10.x.x.x
+# Enable nodelocal dns cache
+enable_nodelocaldns: true
+enable_nodelocaldns_secondary: false
+nodelocaldns_ip: 169.254.25.10
+nodelocaldns_health_port: 9254
+nodelocaldns_second_health_port: 9256
+nodelocaldns_bind_metrics_host_ip: false
+nodelocaldns_secondary_skew_seconds: 5
+# nodelocaldns_external_zones:
+# - zones:
+# - example.com
+# - example.io:1053
+# nameservers:
+# - 1.1.1.1
+# - 2.2.2.2
+# cache: 5
+# - zones:
+# - https://mycompany.local:4453
+# nameservers:
+# - 192.168.0.53
+# cache: 0
+# Enable k8s_external plugin for CoreDNS
+enable_coredns_k8s_external: false
+coredns_k8s_external_zone: k8s_external.local
+# Enable endpoint_pod_names option for kubernetes plugin
+enable_coredns_k8s_endpoint_pod_names: false
+
+# Can be docker_dns, host_resolvconf or none
+resolvconf_mode: host_resolvconf
+# Deploy netchecker app to verify DNS resolve as an HTTP service
+deploy_netchecker: false
+# Ip address of the kubernetes skydns service
+skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
+skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
+dns_domain: "{{ cluster_name }}"
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Default: containerd
+container_manager: containerd
+
+# Additional container runtimes
+kata_containers_enabled: false
+
+kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
+
+# K8s image pull policy (imagePullPolicy)
+k8s_image_pull_policy: IfNotPresent
+
+# audit log for kubernetes
+kubernetes_audit: false
+
+# define kubelet config dir for dynamic kubelet
+# kubelet_config_dir:
+default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
+
+# pod security policy (RBAC must be enabled either by having 'RBAC' in authorization_modes or kubeadm enabled)
+podsecuritypolicy_enabled: false
+
+# Custom PodSecurityPolicySpec for restricted policy
+# podsecuritypolicy_restricted_spec: {}
+
+# Custom PodSecurityPolicySpec for privileged policy
+# podsecuritypolicy_privileged_spec: {}
+
+# Make a copy of kubeconfig on the host that runs Ansible in {{ inventory_dir }}/artifacts
+# kubeconfig_localhost: false
+# Use ansible_host as external api ip when copying over kubeconfig.
+# kubeconfig_localhost_ansible_host: false
+# Download kubectl onto the host that runs Ansible in {{ bin_dir }}
+# kubectl_localhost: false
+
+# A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.
+# Acceptable options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".
+# kubelet_enforce_node_allocatable: pods
+
+## Optionally reserve resources for OS system daemons.
+# system_reserved: true
+## Uncomment to override default values
+# system_memory_reserved: 512Mi
+# system_cpu_reserved: 500m
+# system_ephemeral_storage_reserved: 2Gi
+## Reservation for master hosts
+# system_master_memory_reserved: 256Mi
+# system_master_cpu_reserved: 250m
+# system_master_ephemeral_storage_reserved: 2Gi
+
+## Eviction Thresholds to avoid system OOMs
+# https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#eviction-thresholds
+# eviction_hard: {}
+# eviction_hard_control_plane: {}
+
+# An alternative flexvolume plugin directory
+# kubelet_flexvolumes_plugins_dir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
+
+## Supplementary addresses that can be added in kubernetes ssl keys.
+## That can be useful for example to setup a keepalived virtual IP
+# supplementary_addresses_in_ssl_keys: [10.0.0.1, 10.0.0.2, 10.0.0.3]
+
+## Running on top of openstack vms with cinder enabled may lead to unschedulable pods due to NoVolumeZoneConflict restriction in kube-scheduler.
+## See https://github.com/kubernetes-sigs/kubespray/issues/2141
+## Set this variable to true to get rid of this issue
+volume_cross_zone_attachment: false
+## Add Persistent Volumes Storage Class for corresponding cloud provider (supported: in-tree OpenStack, Cinder CSI,
+## AWS EBS CSI, Azure Disk CSI, GCP Persistent Disk CSI)
+persistent_volumes_enabled: false
+
+## Container Engine Acceleration
+## Enable container acceleration feature, for example use gpu acceleration in containers
+# nvidia_accelerator_enabled: true
+## Nvidia GPU driver install. Install will by done by a (init) pod running as a daemonset.
+## Important: if you use Ubuntu then you should set in all.yml 'docker_storage_options: -s overlay2'
+## Array with nvida_gpu_nodes, leave empty or comment if you don't want to install drivers.
+## Labels and taints won't be set to nodes if they are not in the array.
+# nvidia_gpu_nodes:
+# - kube-gpu-001
+# nvidia_driver_version: "384.111"
+## flavor can be tesla or gtx
+# nvidia_gpu_flavor: gtx
+## NVIDIA driver installer images. Change them if you have trouble accessing gcr.io.
+# nvidia_driver_install_centos_container: atzedevries/nvidia-centos-driver-installer:2
+# nvidia_driver_install_ubuntu_container: gcr.io/google-containers/ubuntu-nvidia-driver-installer@sha256:7df76a0f0a17294e86f691c81de6bbb7c04a1b4b3d4ea4e7e2cccdc42e1f6d63
+## NVIDIA GPU device plugin image.
+# nvidia_gpu_device_plugin_container: "registry.k8s.io/nvidia-gpu-device-plugin@sha256:0842734032018be107fa2490c98156992911e3e1f2a21e059ff0105b07dd8e9e"
+
+## Support tls min version, Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13.
+# tls_min_version: ""
+
+## Support tls cipher suites.
+# tls_cipher_suites: {}
+# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
+# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+# - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
+# - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
+# - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
+# - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
+# - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
+# - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
+# - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
+# - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
+# - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
+# - TLS_ECDHE_RSA_WITH_RC4_128_SHA
+# - TLS_RSA_WITH_3DES_EDE_CBC_SHA
+# - TLS_RSA_WITH_AES_128_CBC_SHA
+# - TLS_RSA_WITH_AES_128_CBC_SHA256
+# - TLS_RSA_WITH_AES_128_GCM_SHA256
+# - TLS_RSA_WITH_AES_256_CBC_SHA
+# - TLS_RSA_WITH_AES_256_GCM_SHA384
+# - TLS_RSA_WITH_RC4_128_SHA
+
+## Amount of time to retain events. (default 1h0m0s)
+event_ttl_duration: "1h0m0s"
+
+## Automatically renew K8S control plane certificates on first Monday of each month
+auto_renew_certificates: false
+# First Monday of each month
+# auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"
--- /dev/null
+---
+# see roles/network_plugin/calico/defaults/main.yml
+
+# the default value of name
+calico_cni_name: k8s-pod-network
+
+## With calico it is possible to distributed routes with border routers of the datacenter.
+## Warning : enabling router peering will disable calico's default behavior ('node mesh').
+## The subnets of each nodes will be distributed by the datacenter router
+# peer_with_router: false
+
+# Enables Internet connectivity from containers
+# nat_outgoing: true
+
+# Enables Calico CNI "host-local" IPAM plugin
+# calico_ipam_host_local: true
+
+# add default ippool name
+# calico_pool_name: "default-pool"
+
+# add default ippool blockSize (defaults kube_network_node_prefix)
+calico_pool_blocksize: 26
+
+# add default ippool CIDR (must be inside kube_pods_subnet, defaults to kube_pods_subnet otherwise)
+# calico_pool_cidr: 1.2.3.4/5
+
+# add default ippool CIDR to CNI config
+# calico_cni_pool: true
+
+# Add default IPV6 IPPool CIDR. Must be inside kube_pods_subnet_ipv6. Defaults to kube_pods_subnet_ipv6 if not set.
+# calico_pool_cidr_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+
+# Add default IPV6 IPPool CIDR to CNI config
+# calico_cni_pool_ipv6: true
+
+# Global as_num (/calico/bgp/v1/global/as_num)
+# global_as_num: "64512"
+
+# If doing peering with node-assigned asn where the globas does not match your nodes, you want this
+# to be true. All other cases, false.
+# calico_no_global_as_num: false
+
+# You can set MTU value here. If left undefined or empty, it will
+# not be specified in calico CNI config, so Calico will use built-in
+# defaults. The value should be a number, not a string.
+# calico_mtu: 1500
+
+# Configure the MTU to use for workload interfaces and tunnels.
+# - If Wireguard is enabled, subtract 60 from your network MTU (i.e 1500-60=1440)
+# - Otherwise, if VXLAN or BPF mode is enabled, subtract 50 from your network MTU (i.e. 1500-50=1450)
+# - Otherwise, if IPIP is enabled, subtract 20 from your network MTU (i.e. 1500-20=1480)
+# - Otherwise, if not using any encapsulation, set to your network MTU (i.e. 1500)
+# calico_veth_mtu: 1440
+
+# Advertise Cluster IPs
+# calico_advertise_cluster_ips: true
+
+# Advertise Service External IPs
+# calico_advertise_service_external_ips:
+# - x.x.x.x/24
+# - y.y.y.y/32
+
+# Adveritse Service LoadBalancer IPs
+# calico_advertise_service_loadbalancer_ips:
+# - x.x.x.x/24
+# - y.y.y.y/16
+
+# Choose data store type for calico: "etcd" or "kdd" (kubernetes datastore)
+# calico_datastore: "kdd"
+
+# Choose Calico iptables backend: "Legacy", "Auto" or "NFT"
+# calico_iptables_backend: "Auto"
+
+# Use typha (only with kdd)
+# typha_enabled: false
+
+# Generate TLS certs for secure typha<->calico-node communication
+# typha_secure: false
+
+# Scaling typha: 1 replica per 100 nodes is adequate
+# Number of typha replicas
+# typha_replicas: 1
+
+# Set max typha connections
+# typha_max_connections_lower_limit: 300
+
+# Set calico network backend: "bird", "vxlan" or "none"
+# bird enable BGP routing, required for ipip and no encapsulation modes
+# calico_network_backend: vxlan
+
+# IP in IP and VXLAN is mutualy exclusive modes.
+# set IP in IP encapsulation mode: "Always", "CrossSubnet", "Never"
+# calico_ipip_mode: 'Never'
+
+# set VXLAN encapsulation mode: "Always", "CrossSubnet", "Never"
+# calico_vxlan_mode: 'Always'
+
+# set VXLAN port and VNI
+# calico_vxlan_vni: 4096
+# calico_vxlan_port: 4789
+
+# Cenable eBPF mode
+# calico_bpf_enabled: false
+
+# If you want to use non default IP_AUTODETECTION_METHOD, IP6_AUTODETECTION_METHOD for calico node set this option to one of:
+# * can-reach=DESTINATION
+# * interface=INTERFACE-REGEX
+# see https://docs.projectcalico.org/reference/node/configuration
+# calico_ip_auto_method: "interface=eth.*"
+# calico_ip6_auto_method: "interface=eth.*"
+
+# Choose the iptables insert mode for Calico: "Insert" or "Append".
+# calico_felix_chaininsertmode: Insert
+
+# If you want use the default route interface when you use multiple interface with dynamique route (iproute2)
+# see https://docs.projectcalico.org/reference/node/configuration : FELIX_DEVICEROUTESOURCEADDRESS
+# calico_use_default_route_src_ipaddr: false
+
+# Enable calico traffic encryption with wireguard
+# calico_wireguard_enabled: false
+
+# Under certain situations liveness and readiness probes may need tunning
+# calico_node_livenessprobe_timeout: 10
+# calico_node_readinessprobe_timeout: 10
+
+# Calico apiserver (only with kdd)
+# calico_apiserver_enabled: false
--- /dev/null
+# see roles/network_plugin/canal/defaults/main.yml
+
+# The interface used by canal for host <-> host communication.
+# If left blank, then the interface is choosing using the node's
+# default route.
+# canal_iface: ""
+
+# Whether or not to masquerade traffic to destinations not within
+# the pod network.
+# canal_masquerade: "true"
--- /dev/null
+---
+# cilium_version: "v1.11.7"
+
+# Log-level
+# cilium_debug: false
+
+# cilium_mtu: ""
+# cilium_enable_ipv4: true
+# cilium_enable_ipv6: false
+
+# Cilium agent health port
+# cilium_agent_health_port: "9879"
+
+# Identity allocation mode selects how identities are shared between cilium
+# nodes by setting how they are stored. The options are "crd" or "kvstore".
+# - "crd" stores identities in kubernetes as CRDs (custom resource definition).
+# These can be queried with:
+# `kubectl get ciliumid`
+# - "kvstore" stores identities in an etcd kvstore.
+# - In order to support External Workloads, "crd" is required
+# - Ref: https://docs.cilium.io/en/stable/gettingstarted/external-workloads/#setting-up-support-for-external-workloads-beta
+# - KVStore operations are only required when cilium-operator is running with any of the below options:
+# - --synchronize-k8s-services
+# - --synchronize-k8s-nodes
+# - --identity-allocation-mode=kvstore
+# - Ref: https://docs.cilium.io/en/stable/internals/cilium_operator/#kvstore-operations
+# cilium_identity_allocation_mode: kvstore
+
+# Etcd SSL dirs
+# cilium_cert_dir: /etc/cilium/certs
+# kube_etcd_cacert_file: ca.pem
+# kube_etcd_cert_file: cert.pem
+# kube_etcd_key_file: cert-key.pem
+
+# Limits for apps
+# cilium_memory_limit: 500M
+# cilium_cpu_limit: 500m
+# cilium_memory_requests: 64M
+# cilium_cpu_requests: 100m
+
+# Overlay Network Mode
+# cilium_tunnel_mode: vxlan
+# Optional features
+# cilium_enable_prometheus: false
+# Enable if you want to make use of hostPort mappings
+# cilium_enable_portmap: false
+# Monitor aggregation level (none/low/medium/maximum)
+# cilium_monitor_aggregation: medium
+# The monitor aggregation flags determine which TCP flags which, upon the
+# first observation, cause monitor notifications to be generated.
+#
+# Only effective when monitor aggregation is set to "medium" or higher.
+# cilium_monitor_aggregation_flags: "all"
+# Kube Proxy Replacement mode (strict/probe/partial)
+# cilium_kube_proxy_replacement: probe
+
+# If upgrading from Cilium < 1.5, you may want to override some of these options
+# to prevent service disruptions. See also:
+# http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
+# cilium_preallocate_bpf_maps: false
+
+# `cilium_tofqdns_enable_poller` is deprecated in 1.8, removed in 1.9
+# cilium_tofqdns_enable_poller: false
+
+# `cilium_enable_legacy_services` is deprecated in 1.6, removed in 1.9
+# cilium_enable_legacy_services: false
+
+# Unique ID of the cluster. Must be unique across all conneted clusters and
+# in the range of 1 and 255. Only relevant when building a mesh of clusters.
+# This value is not defined by default
+# cilium_cluster_id:
+
+# Deploy cilium even if kube_network_plugin is not cilium.
+# This enables to deploy cilium alongside another CNI to replace kube-proxy.
+# cilium_deploy_additionally: false
+
+# Auto direct nodes routes can be used to advertise pods routes in your cluster
+# without any tunelling (with `cilium_tunnel_mode` sets to `disabled`).
+# This works only if you have a L2 connectivity between all your nodes.
+# You wil also have to specify the variable `cilium_native_routing_cidr` to
+# make this work. Please refer to the cilium documentation for more
+# information about this kind of setups.
+# cilium_auto_direct_node_routes: false
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
+# cilium_native_routing_cidr: ""
+
+# Allows to explicitly specify the IPv6 CIDR for native routing.
+# cilium_native_routing_cidr_ipv6: ""
+
+# Enable transparent network encryption.
+# cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+# cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+# cilium_ipsec_node_encryption: false
+
+# If your kernel or distribution does not support WireGuard, Cilium agent can be configured to fall back on the user-space implementation.
+# When this flag is enabled and Cilium detects that the kernel has no native support for WireGuard,
+# it will fallback on the wireguard-go user-space implementation of WireGuard.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+# cilium_wireguard_userspace_fallback: false
+
+# IP Masquerade Agent
+# https://docs.cilium.io/en/stable/concepts/networking/masquerading/
+# By default, all packets from a pod destined to an IP address outside of the cilium_native_routing_cidr range are masqueraded
+# cilium_ip_masq_agent_enable: false
+### A packet sent from a pod to a destination which belongs to any CIDR from the nonMasqueradeCIDRs is not going to be masqueraded
+# cilium_non_masquerade_cidrs:
+# - 10.0.0.0/8
+# - 172.16.0.0/12
+# - 192.168.0.0/16
+# - 100.64.0.0/10
+# - 192.0.0.0/24
+# - 192.0.2.0/24
+# - 192.88.99.0/24
+# - 198.18.0.0/15
+# - 198.51.100.0/24
+# - 203.0.113.0/24
+# - 240.0.0.0/4
+### Indicates whether to masquerade traffic to the link local prefix.
+### If the masqLinkLocal is not set or set to false, then 169.254.0.0/16 is appended to the non-masquerade CIDRs list.
+# cilium_masq_link_local: false
+### A time interval at which the agent attempts to reload config from disk
+# cilium_ip_masq_resync_interval: 60s
+
+# Hubble
+### Enable Hubble without install
+# cilium_enable_hubble: false
+### Enable Hubble Metrics
+# cilium_enable_hubble_metrics: false
+### if cilium_enable_hubble_metrics: true
+# cilium_hubble_metrics: {}
+# - dns
+# - drop
+# - tcp
+# - flow
+# - icmp
+# - http
+### Enable Hubble install
+# cilium_hubble_install: false
+### Enable auto generate certs if cilium_hubble_install: true
+# cilium_hubble_tls_generate: false
+
+# IP address management mode for v1.9+.
+# https://docs.cilium.io/en/v1.9/concepts/networking/ipam/
+# cilium_ipam_mode: kubernetes
+
+# Extra arguments for the Cilium agent
+# cilium_agent_custom_args: []
+
+# For adding and mounting extra volumes to the cilium agent
+# cilium_agent_extra_volumes: []
+# cilium_agent_extra_volume_mounts: []
+
+# cilium_agent_extra_env_vars: []
+
+# cilium_operator_replicas: 2
+
+# The address at which the cillium operator bind health check api
+# cilium_operator_api_serve_addr: "127.0.0.1:9234"
+
+## A dictionary of extra config variables to add to cilium-config, formatted like:
+## cilium_config_extra_vars:
+## var1: "value1"
+## var2: "value2"
+# cilium_config_extra_vars: {}
+
+# For adding and mounting extra volumes to the cilium operator
+# cilium_operator_extra_volumes: []
+# cilium_operator_extra_volume_mounts: []
+
+# Extra arguments for the Cilium Operator
+# cilium_operator_custom_args: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+# cilium_cluster_name: default
+
+# Make Cilium take ownership over the `/etc/cni/net.d` directory on the node, renaming all non-Cilium CNI configurations to `*.cilium_bak`.
+# This ensures no Pods can be scheduled using other CNI plugins during Cilium agent downtime.
+# Available for Cilium v1.10 and up.
+# cilium_cni_exclusive: true
+
+# Configure the log file for CNI logging with retention policy of 7 days.
+# Disable CNI file logging by setting this field to empty explicitly.
+# Available for Cilium v1.12 and up.
+# cilium_cni_log_file: "/var/run/cilium/cilium-cni.log"
+
+# -- Configure cgroup related configuration
+# -- Enable auto mount of cgroup2 filesystem.
+# When `cilium_cgroup_auto_mount` is enabled, cgroup2 filesystem is mounted at
+# `cilium_cgroup_host_root` path on the underlying host and inside the cilium agent pod.
+# If users disable `cilium_cgroup_auto_mount`, it's expected that users have mounted
+# cgroup2 filesystem at the specified `cilium_cgroup_auto_mount` volume, and then the
+# volume will be mounted inside the cilium agent pod at the same path.
+# Available for Cilium v1.11 and up
+# cilium_cgroup_auto_mount: true
+# -- Configure cgroup root where cgroup2 filesystem is mounted on the host
+# cilium_cgroup_host_root: "/run/cilium/cgroupv2"
+
+# Specifies the ratio (0.0-1.0) of total system memory to use for dynamic
+# sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
+# cilium_bpf_map_dynamic_size_ratio: "0.0"
+
+# -- Enables masquerading of IPv4 traffic leaving the node from endpoints.
+# Available for Cilium v1.10 and up
+# cilium_enable_ipv4_masquerade: true
+# -- Enables masquerading of IPv6 traffic leaving the node from endpoints.
+# Available for Cilium v1.10 and up
+# cilium_enable_ipv6_masquerade: true
+
+# -- Enable native IP masquerade support in eBPF
+# cilium_enable_bpf_masquerade: false
+
+# -- Configure whether direct routing mode should route traffic via
+# host stack (true) or directly and more efficiently out of BPF (false) if
+# the kernel supports it. The latter has the implication that it will also
+# bypass netfilter in the host namespace.
+# cilium_enable_host_legacy_routing: true
+
+# -- Enable use of the remote node identity.
+# ref: https://docs.cilium.io/en/v1.7/install/upgrade/#configmap-remote-node-identity
+# cilium_enable_remote_node_identity: true
+
+# -- Enable the use of well-known identities.
+# cilium_enable_well_known_identities: false
+
+# cilium_enable_bpf_clock_probe: true
+
+# -- Whether to enable CNP status updates.
+# cilium_disable_cnp_status_updates: true
--- /dev/null
+# see roles/network_plugin/flannel/defaults/main.yml
+
+## interface that should be used for flannel operations
+## This is actually an inventory cluster-level item
+# flannel_interface:
+
+## Select interface that should be used for flannel operations by regexp on Name or IP
+## This is actually an inventory cluster-level item
+## example: select interface with ip from net 10.0.0.0/23
+## single quote and escape backslashes
+# flannel_interface_regexp: '10\\.0\\.[0-2]\\.\\d{1,3}'
+
+# You can choose what type of flannel backend to use: 'vxlan' or 'host-gw'
+# for experimental backend
+# please refer to flannel's docs : https://github.com/coreos/flannel/blob/master/README.md
+# flannel_backend_type: "vxlan"
+# flannel_vxlan_vni: 1
+# flannel_vxlan_port: 8472
--- /dev/null
+---
+
+# geneve or vlan
+kube_ovn_network_type: geneve
+
+# geneve, vxlan or stt. ATTENTION: some networkpolicy cannot take effect when using vxlan and stt need custom compile ovs kernel module
+kube_ovn_tunnel_type: geneve
+
+## The nic to support container network can be a nic name or a group of regex separated by comma e.g: 'enp6s0f0,eth.*', if empty will use the nic that the default route use.
+# kube_ovn_iface: eth1
+## The MTU used by pod iface in overlay networks (default iface MTU - 100)
+# kube_ovn_mtu: 1333
+
+## Enable hw-offload, disable traffic mirror and set the iface to the physical port. Make sure that there is an IP address bind to the physical port.
+kube_ovn_hw_offload: false
+# traffic mirror
+kube_ovn_traffic_mirror: false
+
+# kube_ovn_pool_cidr_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+# kube_ovn_default_interface_name: eth0
+
+kube_ovn_external_address: 8.8.8.8
+kube_ovn_external_address_ipv6: 2400:3200::1
+kube_ovn_external_dns: alauda.cn
+
+# kube_ovn_default_gateway: 10.233.64.1,fd85:ee78:d8a6:8607::1:0
+kube_ovn_default_gateway_check: true
+kube_ovn_default_logical_gateway: false
+# kube_ovn_default_exclude_ips: 10.16.0.1
+kube_ovn_node_switch_cidr: 100.64.0.0/16
+kube_ovn_node_switch_cidr_ipv6: fd00:100:64::/64
+
+## vlan config, set default interface name and vlan id
+# kube_ovn_default_interface_name: eth0
+kube_ovn_default_vlan_id: 100
+kube_ovn_vlan_name: product
+
+## pod nic type, support: veth-pair or internal-port
+kube_ovn_pod_nic_type: veth_pair
+
+## Enable load balancer
+kube_ovn_enable_lb: true
+
+## Enable network policy support
+kube_ovn_enable_np: true
+
+## Enable external vpc support
+kube_ovn_enable_external_vpc: true
+
+## Enable checksum
+kube_ovn_encap_checksum: true
+
+## enable ssl
+kube_ovn_enable_ssl: false
+
+## dpdk
+kube_ovn_dpdk_enabled: false
--- /dev/null
+# See roles/network_plugin/kube-router//defaults/main.yml
+
+# Enables Pod Networking -- Advertises and learns the routes to Pods via iBGP
+# kube_router_run_router: true
+
+# Enables Network Policy -- sets up iptables to provide ingress firewall for pods
+# kube_router_run_firewall: true
+
+# Enables Service Proxy -- sets up IPVS for Kubernetes Services
+# see docs/kube-router.md "Caveats" section
+# kube_router_run_service_proxy: false
+
+# Add Cluster IP of the service to the RIB so that it gets advertises to the BGP peers.
+# kube_router_advertise_cluster_ip: false
+
+# Add External IP of service to the RIB so that it gets advertised to the BGP peers.
+# kube_router_advertise_external_ip: false
+
+# Add LoadbBalancer IP of service status as set by the LB provider to the RIB so that it gets advertised to the BGP peers.
+# kube_router_advertise_loadbalancer_ip: false
+
+# Adjust manifest of kube-router daemonset template with DSR needed changes
+# kube_router_enable_dsr: false
+
+# Array of arbitrary extra arguments to kube-router, see
+# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md
+# kube_router_extra_args: []
+
+# ASN number of the cluster, used when communicating with external BGP routers
+# kube_router_cluster_asn: ~
+
+# ASN numbers of the BGP peer to which cluster nodes will advertise cluster ip and node's pod cidr.
+# kube_router_peer_router_asns: ~
+
+# The ip address of the external router to which all nodes will peer and advertise the cluster ip and pod cidr's.
+# kube_router_peer_router_ips: ~
+
+# The remote port of the external BGP to which all nodes will peer. If not set, default BGP port (179) will be used.
+# kube_router_peer_router_ports: ~
+
+# Setups node CNI to allow hairpin mode, requires node reboots, see
+# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode
+# kube_router_support_hairpin_mode: false
+
+# Select DNS Policy ClusterFirstWithHostNet, ClusterFirst, etc.
+# kube_router_dns_policy: ClusterFirstWithHostNet
+
+# Array of annotations for master
+# kube_router_annotations_master: []
+
+# Array of annotations for every node
+# kube_router_annotations_node: []
+
+# Array of common annotations for every node
+# kube_router_annotations_all: []
+
+# Enables scraping kube-router metrics with Prometheus
+# kube_router_enable_metrics: false
+
+# Path to serve Prometheus metrics on
+# kube_router_metrics_path: /metrics
+
+# Prometheus metrics port to use
+# kube_router_metrics_port: 9255
--- /dev/null
+---
+# private interface, on a l2-network
+macvlan_interface: "eth1"
+
+# Enable nat in default gateway network interface
+enable_nat_default_gateway: true
--- /dev/null
+# see roles/network_plugin/weave/defaults/main.yml
+
+# Weave's network password for encryption, if null then no network encryption.
+# weave_password: ~
+
+# If set to 1, disable checking for new Weave Net versions (default is blank,
+# i.e. check is enabled)
+# weave_checkpoint_disable: false
+
+# Soft limit on the number of connections between peers. Defaults to 100.
+# weave_conn_limit: 100
+
+# Weave Net defaults to enabling hairpin on the bridge side of the veth pair
+# for containers attached. If you need to disable hairpin, e.g. your kernel is
+# one of those that can panic if hairpin is enabled, then you can disable it by
+# setting `HAIRPIN_MODE=false`.
+# weave_hairpin_mode: true
+
+# The range of IP addresses used by Weave Net and the subnet they are placed in
+# (CIDR format; default 10.32.0.0/12)
+# weave_ipalloc_range: "{{ kube_pods_subnet }}"
+
+# Set to 0 to disable Network Policy Controller (default is on)
+# weave_expect_npc: "{{ enable_network_policy }}"
+
+# List of addresses of peers in the Kubernetes cluster (default is to fetch the
+# list from the api-server)
+# weave_kube_peers: ~
+
+# Set the initialization mode of the IP Address Manager (defaults to consensus
+# amongst the KUBE_PEERS)
+# weave_ipalloc_init: ~
+
+# Set the IP address used as a gateway from the Weave network to the host
+# network - this is useful if you are configuring the addon as a static pod.
+# weave_expose_ip: ~
+
+# Address and port that the Weave Net daemon will serve Prometheus-style
+# metrics on (defaults to 0.0.0.0:6782)
+# weave_metrics_addr: ~
+
+# Address and port that the Weave Net daemon will serve status requests on
+# (defaults to disabled)
+# weave_status_addr: ~
+
+# Weave Net defaults to 1376 bytes, but you can set a smaller size if your
+# underlying network has a tighter limit, or set a larger size for better
+# performance if your network supports jumbo frames (e.g. 8916)
+# weave_mtu: 1376
+
+# Set to 1 to preserve the client source IP address when accessing Service
+# annotated with `service.spec.externalTrafficPolicy=Local`. The feature works
+# only with Weave IPAM (default).
+# weave_no_masq_local: true
+
+# set to nft to use nftables backend for iptables (default is iptables)
+# weave_iptables_backend: iptables
+
+# Extra variables that passing to launch.sh, useful for enabling seed mode, see
+# https://www.weave.works/docs/net/latest/tasks/ipam/ipam/
+# weave_extra_args: ~
+
+# Extra variables for weave_npc that passing to launch.sh, useful for change log level, ex --log-level=error
+# weave_npc_extra_args: ~
--- /dev/null
+node1 ansible_connection=local local_release_dir={{ansible_env.HOME}}/releases
+
+[kube_control_plane]
+node1
+
+[etcd]
+node1
+
+[kube_node]
+node1
+
+[k8s_cluster:children]
+kube_node
+kube_control_plane
--- /dev/null
+---
+## Directory where the binaries will be installed
+bin_dir: /usr/local/bin
+
+## The access_ip variable is used to define how other nodes should access
+## the node. This is used in flannel to allow other flannel nodes to see
+## this node for example. The access_ip is really useful AWS and Google
+## environments where the nodes are accessed remotely by the "public" ip,
+## but don't know about that address themselves.
+# access_ip: 1.1.1.1
+
+
+## External LB example config
+## apiserver_loadbalancer_domain_name: "elb.some.domain"
+# loadbalancer_apiserver:
+# address: 1.2.3.4
+# port: 1234
+
+## Internal loadbalancers for apiservers
+# loadbalancer_apiserver_localhost: true
+# valid options are "nginx" or "haproxy"
+# loadbalancer_apiserver_type: nginx # valid values "nginx" or "haproxy"
+
+## If the cilium is going to be used in strict mode, we can use the
+## localhost connection and not use the external LB. If this parameter is
+## not specified, the first node to connect to kubeapi will be used.
+# use_localhost_as_kubeapi_loadbalancer: true
+
+## Local loadbalancer should use this port
+## And must be set port 6443
+loadbalancer_apiserver_port: 6443
+
+## If loadbalancer_apiserver_healthcheck_port variable defined, enables proxy liveness check for nginx.
+loadbalancer_apiserver_healthcheck_port: 8081
+
+### OTHER OPTIONAL VARIABLES
+
+## Upstream dns servers
+# upstream_dns_servers:
+# - 8.8.8.8
+# - 8.8.4.4
+
+## There are some changes specific to the cloud providers
+## for instance we need to encapsulate packets with some network plugins
+## If set the possible values are either 'gce', 'aws', 'azure', 'openstack', 'vsphere', 'oci', or 'external'
+## When openstack is used make sure to source in the openstack credentials
+## like you would do when using openstack-client before starting the playbook.
+# cloud_provider:
+
+## When cloud_provider is set to 'external', you can set the cloud controller to deploy
+## Supported cloud controllers are: 'openstack', 'vsphere' and 'hcloud'
+## When openstack or vsphere are used make sure to source in the required fields
+# external_cloud_provider:
+
+## Set these proxy values in order to update package manager and docker daemon to use proxies
+# http_proxy: ""
+# https_proxy: ""
+
+## Refer to roles/kubespray-defaults/defaults/main.yml before modifying no_proxy
+# no_proxy: ""
+
+## Some problems may occur when downloading files over https proxy due to ansible bug
+## https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
+## SSL validation of get_url module. Note that kubespray will still be performing checksum validation.
+# download_validate_certs: False
+
+## If you need exclude all cluster nodes from proxy and other resources, add other resources here.
+# additional_no_proxy: ""
+
+## If you need to disable proxying of os package repositories but are still behind an http_proxy set
+## skip_http_proxy_on_os_packages to true
+## This will cause kubespray not to set proxy environment in /etc/yum.conf for centos and in /etc/apt/apt.conf for debian/ubuntu
+## Special information for debian/ubuntu - you have to set the no_proxy variable, then apt package will install from your source of wish
+# skip_http_proxy_on_os_packages: false
+
+## Since workers are included in the no_proxy variable by default, docker engine will be restarted on all nodes (all
+## pods will restart) when adding or removing workers. To override this behaviour by only including master nodes in the
+## no_proxy variable, set below to true:
+no_proxy_exclude_workers: false
+
+## Certificate Management
+## This setting determines whether certs are generated via scripts.
+## Chose 'none' if you provide your own certificates.
+## Option is "script", "none"
+# cert_management: script
+
+## Set to true to allow pre-checks to fail and continue deployment
+# ignore_assert_errors: false
+
+## The read-only port for the Kubelet to serve on with no authentication/authorization. Uncomment to enable.
+# kube_read_only_port: 10255
+
+## Set true to download and cache container
+# download_container: true
+
+## Deploy container engine
+# Set false if you want to deploy container engine manually.
+# deploy_container_engine: true
+
+## Red Hat Enterprise Linux subscription registration
+## Add either RHEL subscription Username/Password or Organization ID/Activation Key combination
+## Update RHEL subscription purpose usage, role and SLA if necessary
+# rh_subscription_username: ""
+# rh_subscription_password: ""
+# rh_subscription_org_id: ""
+# rh_subscription_activation_key: ""
+# rh_subscription_usage: "Development"
+# rh_subscription_role: "Red Hat Enterprise Server"
+# rh_subscription_sla: "Self-Support"
+
+## Check if access_ip responds to ping. Set false if your firewall blocks ICMP.
+# ping_access_ip: true
+
+# sysctl_file_path to add sysctl conf to
+# sysctl_file_path: "/etc/sysctl.d/99-sysctl.conf"
+
+## Variables for webhook token auth https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication
+kube_webhook_token_auth: false
+kube_webhook_token_auth_url_skip_tls_verify: false
+# kube_webhook_token_auth_url: https://...
+## base64-encoded string of the webhook's CA certificate
+# kube_webhook_token_auth_ca_data: "LS0t..."
+
+## NTP Settings
+# Start the ntpd or chrony service and enable it at system boot.
+ntp_enabled: false
+ntp_manage_config: false
+ntp_servers:
+ - "0.pool.ntp.org iburst"
+ - "1.pool.ntp.org iburst"
+ - "2.pool.ntp.org iburst"
+ - "3.pool.ntp.org iburst"
+
+## Used to control no_log attribute
+unsafe_show_logs: false
--- /dev/null
+## To use AWS EBS CSI Driver to provision volumes, uncomment the first value
+## and configure the parameters below
+# aws_ebs_csi_enabled: true
+# aws_ebs_csi_enable_volume_scheduling: true
+# aws_ebs_csi_enable_volume_snapshot: false
+# aws_ebs_csi_enable_volume_resizing: false
+# aws_ebs_csi_controller_replicas: 1
+# aws_ebs_csi_plugin_image_tag: latest
+# aws_ebs_csi_extra_volume_tags: "Owner=owner,Team=team,Environment=environment'
--- /dev/null
+## When azure is used, you need to also set the following variables.
+## see docs/azure.md for details on how to get these values
+
+# azure_cloud:
+# azure_tenant_id:
+# azure_subscription_id:
+# azure_aad_client_id:
+# azure_aad_client_secret:
+# azure_resource_group:
+# azure_location:
+# azure_subnet_name:
+# azure_security_group_name:
+# azure_security_group_resource_group:
+# azure_vnet_name:
+# azure_vnet_resource_group:
+# azure_route_table_name:
+# azure_route_table_resource_group:
+# supported values are 'standard' or 'vmss'
+# azure_vmtype: standard
+
+## Azure Disk CSI credentials and parameters
+## see docs/azure-csi.md for details on how to get these values
+
+# azure_csi_tenant_id:
+# azure_csi_subscription_id:
+# azure_csi_aad_client_id:
+# azure_csi_aad_client_secret:
+# azure_csi_location:
+# azure_csi_resource_group:
+# azure_csi_vnet_name:
+# azure_csi_vnet_resource_group:
+# azure_csi_subnet_name:
+# azure_csi_security_group_name:
+# azure_csi_use_instance_metadata:
+# azure_csi_tags: "Owner=owner,Team=team,Environment=environment'
+
+## To enable Azure Disk CSI, uncomment below
+# azure_csi_enabled: true
+# azure_csi_controller_replicas: 1
+# azure_csi_plugin_image_tag: latest
--- /dev/null
+---
+# Please see roles/container-engine/containerd/defaults/main.yml for more configuration options
+
+# containerd_storage_dir: "/var/lib/containerd"
+# containerd_state_dir: "/run/containerd"
+# containerd_oom_score: 0
+
+# containerd_default_runtime: "runc"
+# containerd_snapshotter: "native"
+
+# containerd_runc_runtime:
+# name: runc
+# type: "io.containerd.runc.v2"
+# engine: ""
+# root: ""
+
+# containerd_additional_runtimes:
+# Example for Kata Containers as additional runtime:
+# - name: kata
+# type: "io.containerd.kata.v2"
+# engine: ""
+# root: ""
+
+# containerd_grpc_max_recv_message_size: 16777216
+# containerd_grpc_max_send_message_size: 16777216
+
+# containerd_debug_level: "info"
+
+# containerd_metrics_address: ""
+
+# containerd_metrics_grpc_histogram: false
+
+## An obvious use case is allowing insecure-registry access to self hosted registries.
+## Can be ipaddress and domain_name.
+## example define mirror.registry.io or 172.19.16.11:5000
+## set "name": "url". insecure url must be started http://
+## Port number is also needed if the default HTTPS port is not used.
+# containerd_insecure_registries:
+# "localhost": "http://127.0.0.1"
+# "172.19.16.11:5000": "http://172.19.16.11:5000"
+
+# containerd_registries:
+# "docker.io": "https://registry-1.docker.io"
+
+# containerd_max_container_log_line_size: -1
+
+# containerd_registry_auth:
+# - registry: 10.0.0.2:5000
+# username: user
+# password: pass
--- /dev/null
+## Does coreos need auto upgrade, default is true
+# coreos_auto_upgrade: true
--- /dev/null
+# crio_insecure_registries:
+# - 10.0.0.2:5000
+# crio_registry_auth:
+# - registry: 10.0.0.2:5000
+# username: user
+# password: pass
--- /dev/null
+---
+## Uncomment this if you want to force overlay/overlay2 as docker storage driver
+## Please note that overlay2 is only supported on newer kernels
+# docker_storage_options: -s overlay2
+
+## Enable docker_container_storage_setup, it will configure devicemapper driver on Centos7 or RedHat7.
+docker_container_storage_setup: false
+
+## It must be define a disk path for docker_container_storage_setup_devs.
+## Otherwise docker-storage-setup will be executed incorrectly.
+# docker_container_storage_setup_devs: /dev/vdb
+
+## Uncomment this if you want to change the Docker Cgroup driver (native.cgroupdriver)
+## Valid options are systemd or cgroupfs, default is systemd
+# docker_cgroup_driver: systemd
+
+## Only set this if you have more than 3 nameservers:
+## If true Kubespray will only use the first 3, otherwise it will fail
+docker_dns_servers_strict: false
+
+# Path used to store Docker data
+docker_daemon_graph: "/var/lib/docker"
+
+## Used to set docker daemon iptables options to true
+docker_iptables_enabled: "false"
+
+# Docker log options
+# Rotate container stderr/stdout logs at 50m and keep last 5
+docker_log_opts: "--log-opt max-size=50m --log-opt max-file=5"
+
+# define docker bin_dir
+docker_bin_dir: "/usr/bin"
+
+# keep docker packages after installation; speeds up repeated ansible provisioning runs when '1'
+# kubespray deletes the docker package on each run, so caching the package makes sense
+docker_rpm_keepcache: 1
+
+## An obvious use case is allowing insecure-registry access to self hosted registries.
+## Can be ipaddress and domain_name.
+## example define 172.19.16.11 or mirror.registry.io
+# docker_insecure_registries:
+# - mirror.registry.io
+# - 172.19.16.11
+
+## Add other registry,example China registry mirror.
+# docker_registry_mirrors:
+# - https://registry.docker-cn.com
+# - https://mirror.aliyuncs.com
+
+## If non-empty will override default system MountFlags value.
+## This option takes a mount propagation flag: shared, slave
+## or private, which control whether mounts in the file system
+## namespace set up for docker will receive or propagate mounts
+## and unmounts. Leave empty for system default
+# docker_mount_flags:
+
+## A string of extra options to pass to the docker daemon.
+## This string should be exactly as you wish it to appear.
+# docker_options: ""
--- /dev/null
+---
+## Directory where etcd data stored
+etcd_data_dir: /var/lib/etcd
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
+## Kubeadm etcd deployment is experimental and only available for new deployments
+## If this is not set, container manager will be inherited from the Kubespray defaults
+## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
+## Also this makes possible to use different container manager for etcd nodes.
+# container_manager: containerd
+
+## Settings for etcd deployment type
+# Set this to docker if you are using container_manager: docker
+etcd_deployment_type: host
\ No newline at end of file
--- /dev/null
+## GCP compute Persistent Disk CSI Driver credentials and parameters
+## See docs/gcp-pd-csi.md for information about the implementation
+
+## Specify the path to the file containing the service account credentials
+# gcp_pd_csi_sa_cred_file: "/my/safe/credentials/directory/cloud-sa.json"
+
+## To enable GCP Persistent Disk CSI driver, uncomment below
+# gcp_pd_csi_enabled: true
+# gcp_pd_csi_controller_replicas: 1
+# gcp_pd_csi_driver_image_tag: "v0.7.0-gke.0"
--- /dev/null
+## Values for the external Hcloud Cloud Controller
+# external_hcloud_cloud:
+# hcloud_api_token: ""
+# token_secret_name: hcloud
+# with_networks: false # Use the hcloud controller-manager with networks support https://github.com/hetznercloud/hcloud-cloud-controller-manager#networks-support
+# service_account_name: cloud-controller-manager
+#
+# controller_image_tag: "latest"
+# ## A dictionary of extra arguments to add to the openstack cloud controller manager daemonset
+# ## Format:
+# ## external_hcloud_cloud.controller_extra_args:
+# ## arg1: "value1"
+# ## arg2: "value2"
+# controller_extra_args: {}
--- /dev/null
+## When Oracle Cloud Infrastructure is used, set these variables
+# oci_private_key:
+# oci_region_id:
+# oci_tenancy_id:
+# oci_user_id:
+# oci_user_fingerprint:
+# oci_compartment_id:
+# oci_vnc_id:
+# oci_subnet1_id:
+# oci_subnet2_id:
+## Override these default/optional behaviors if you wish
+# oci_security_list_management: All
+## If you would like the controller to manage specific lists per subnet. This is a mapping of subnet ocids to security list ocids. Below are examples.
+# oci_security_lists:
+# ocid1.subnet.oc1.phx.aaaaaaaasa53hlkzk6nzksqfccegk2qnkxmphkblst3riclzs4rhwg7rg57q: ocid1.securitylist.oc1.iad.aaaaaaaaqti5jsfvyw6ejahh7r4okb2xbtuiuguswhs746mtahn72r7adt7q
+# ocid1.subnet.oc1.phx.aaaaaaaahuxrgvs65iwdz7ekwgg3l5gyah7ww5klkwjcso74u3e4i64hvtvq: ocid1.securitylist.oc1.iad.aaaaaaaaqti5jsfvyw6ejahh7r4okb2xbtuiuguswhs746mtahn72r7adt7q
+## If oci_use_instance_principals is true, you do not need to set the region, tenancy, user, key, passphrase, or fingerprint
+# oci_use_instance_principals: false
+# oci_cloud_controller_version: 0.6.0
+## If you would like to control OCI query rate limits for the controller
+# oci_rate_limit:
+# rate_limit_qps_read:
+# rate_limit_qps_write:
+# rate_limit_bucket_read:
+# rate_limit_bucket_write:
+## Other optional variables
+# oci_cloud_controller_pull_source: (default iad.ocir.io/oracle/cloud-provider-oci)
+# oci_cloud_controller_pull_secret: (name of pull secret to use if you define your own mirror above)
--- /dev/null
+---
+## Global Offline settings
+### Private Container Image Registry
+# registry_host: "myprivateregisry.com"
+# files_repo: "http://myprivatehttpd"
+### If using CentOS, RedHat, AlmaLinux or Fedora
+# yum_repo: "http://myinternalyumrepo"
+### If using Debian
+# debian_repo: "http://myinternaldebianrepo"
+### If using Ubuntu
+# ubuntu_repo: "http://myinternalubunturepo"
+
+## Container Registry overrides
+# kube_image_repo: "{{ registry_host }}"
+# gcr_image_repo: "{{ registry_host }}"
+# github_image_repo: "{{ registry_host }}"
+# docker_image_repo: "{{ registry_host }}"
+# quay_image_repo: "{{ registry_host }}"
+
+## Kubernetes components
+# kubeadm_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
+# kubectl_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
+# kubelet_download_url: "{{ files_repo }}/storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
+
+## CNI Plugins
+# cni_download_url: "{{ files_repo }}/github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
+
+## cri-tools
+# crictl_download_url: "{{ files_repo }}/github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+
+## [Optional] etcd: only if you **DON'T** use etcd_deployment=host
+# etcd_download_url: "{{ files_repo }}/github.com/etcd-io/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
+
+# [Optional] Calico: If using Calico network plugin
+# calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
+# calicoctl_alternate_download_url: "{{ files_repo }}/github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
+# [Optional] Calico with kdd: If using Calico network plugin with kdd datastore
+# calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/{{ calico_version }}.tar.gz"
+
+# [Optional] Flannel: If using Falnnel network plugin
+# flannel_cni_download_url: "{{ files_repo }}/kubernetes/flannel/{{ flannel_cni_version }}/flannel-{{ image_arch }}"
+
+# [Optional] helm: only if you set helm_enabled: true
+# helm_download_url: "{{ files_repo }}/get.helm.sh/helm-{{ helm_version }}-linux-{{ image_arch }}.tar.gz"
+
+# [Optional] crun: only if you set crun_enabled: true
+# crun_download_url: "{{ files_repo }}/github.com/containers/crun/releases/download/{{ crun_version }}/crun-{{ crun_version }}-linux-{{ image_arch }}"
+
+# [Optional] kata: only if you set kata_containers_enabled: true
+# kata_containers_download_url: "{{ files_repo }}/github.com/kata-containers/kata-containers/releases/download/{{ kata_containers_version }}/kata-static-{{ kata_containers_version }}-{{ ansible_architecture }}.tar.xz"
+
+# [Optional] cri-dockerd: only if you set container_manager: docker
+# cri_dockerd_download_url: "{{ files_repo }}/github.com/Mirantis/cri-dockerd/releases/download/v{{ cri_dockerd_version }}/cri-dockerd-{{ cri_dockerd_version }}.{{ image_arch }}.tgz"
+
+# [Optional] cri-o: only if you set container_manager: crio
+# crio_download_base: "download.opensuse.org/repositories/devel:kubic:libcontainers:stable"
+# crio_download_crio: "http://{{ crio_download_base }}:/cri-o:/"
+
+# [Optional] runc,containerd: only if you set container_runtime: containerd
+# runc_download_url: "{{ files_repo }}/github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.{{ image_arch }}"
+# containerd_download_url: "{{ files_repo }}/github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
+# nerdctl_download_url: "{{ files_repo }}/github.com/containerd/nerdctl/releases/download/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
+
+## CentOS/Redhat/AlmaLinux
+### For EL7, base and extras repo must be available, for EL8, baseos and appstream
+### By default we enable those repo automatically
+# rhel_enable_repos: false
+### Docker / Containerd
+# docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
+# docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+## Fedora
+### Docker
+# docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
+# docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
+# containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
+
+## Debian
+### Docker
+# docker_debian_repo_base_url: "{{ debian_repo }}/docker-ce"
+# docker_debian_repo_gpgkey: "{{ debian_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_debian_repo_base_url: "{{ ubuntu_repo }}/containerd"
+# containerd_debian_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+# containerd_debian_repo_repokey: 'YOURREPOKEY'
+
+## Ubuntu
+### Docker
+# docker_ubuntu_repo_base_url: "{{ ubuntu_repo }}/docker-ce"
+# docker_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/docker-ce/gpg"
+### Containerd
+# containerd_ubuntu_repo_base_url: "{{ ubuntu_repo }}/containerd"
+# containerd_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
+# containerd_ubuntu_repo_repokey: 'YOURREPOKEY'
--- /dev/null
+## When OpenStack is used, Cinder version can be explicitly specified if autodetection fails (Fixed in 1.9: https://github.com/kubernetes/kubernetes/issues/50461)
+# openstack_blockstorage_version: "v1/v2/auto (default)"
+# openstack_blockstorage_ignore_volume_az: yes
+## When OpenStack is used, if LBaaSv2 is available you can enable it with the following 2 variables.
+# openstack_lbaas_enabled: True
+# openstack_lbaas_subnet_id: "Neutron subnet ID (not network ID) to create LBaaS VIP"
+## To enable automatic floating ip provisioning, specify a subnet.
+# openstack_lbaas_floating_network_id: "Neutron network ID (not subnet ID) to get floating IP from, disabled by default"
+## Override default LBaaS behavior
+# openstack_lbaas_use_octavia: False
+# openstack_lbaas_method: "ROUND_ROBIN"
+# openstack_lbaas_provider: "haproxy"
+# openstack_lbaas_create_monitor: "yes"
+# openstack_lbaas_monitor_delay: "1m"
+# openstack_lbaas_monitor_timeout: "30s"
+# openstack_lbaas_monitor_max_retries: "3"
+
+## Values for the external OpenStack Cloud Controller
+# external_openstack_lbaas_network_id: "Neutron network ID to create LBaaS VIP"
+# external_openstack_lbaas_subnet_id: "Neutron subnet ID to create LBaaS VIP"
+# external_openstack_lbaas_floating_network_id: "Neutron network ID to get floating IP from"
+# external_openstack_lbaas_floating_subnet_id: "Neutron subnet ID to get floating IP from"
+# external_openstack_lbaas_method: "ROUND_ROBIN"
+# external_openstack_lbaas_provider: "octavia"
+# external_openstack_lbaas_create_monitor: false
+# external_openstack_lbaas_monitor_delay: "1m"
+# external_openstack_lbaas_monitor_timeout: "30s"
+# external_openstack_lbaas_monitor_max_retries: "3"
+# external_openstack_lbaas_manage_security_groups: false
+# external_openstack_lbaas_internal_lb: false
+# external_openstack_network_ipv6_disabled: false
+# external_openstack_network_internal_networks: []
+# external_openstack_network_public_networks: []
+# external_openstack_metadata_search_order: "configDrive,metadataService"
+
+## Application credentials to authenticate against Keystone API
+## Those settings will take precedence over username and password that might be set your environment
+## All of them are required
+# external_openstack_application_credential_name:
+# external_openstack_application_credential_id:
+# external_openstack_application_credential_secret:
+
+## The tag of the external OpenStack Cloud Controller image
+# external_openstack_cloud_controller_image_tag: "latest"
+
+## To use Cinder CSI plugin to provision volumes set this value to true
+## Make sure to source in the openstack credentials
+# cinder_csi_enabled: true
+# cinder_csi_controller_replicas: 1
--- /dev/null
+## Repo for UpClouds csi-driver: https://github.com/UpCloudLtd/upcloud-csi
+## To use UpClouds CSI plugin to provision volumes set this value to true
+## Remember to set UPCLOUD_USERNAME and UPCLOUD_PASSWORD
+# upcloud_csi_enabled: true
+# upcloud_csi_controller_replicas: 1
+## Override used image tags
+# upcloud_csi_provisioner_image_tag: "v3.1.0"
+# upcloud_csi_attacher_image_tag: "v3.4.0"
+# upcloud_csi_resizer_image_tag: "v1.4.0"
+# upcloud_csi_plugin_image_tag: "v0.2.1"
+# upcloud_csi_node_image_tag: "v2.5.0"
+# upcloud_tolerations: []
+## Storage class options
+# expand_persistent_volumes: true
+# parameters:
+# tier: maxiops # or hdd
+# storage_classes:
+# - name: standard
+# is_default: true
--- /dev/null
+## Values for the external vSphere Cloud Provider
+# external_vsphere_vcenter_ip: "myvcenter.domain.com"
+# external_vsphere_vcenter_port: "443"
+# external_vsphere_insecure: "true"
+# external_vsphere_user: "administrator@vsphere.local" # Can also be set via the `VSPHERE_USER` environment variable
+# external_vsphere_password: "K8s_admin" # Can also be set via the `VSPHERE_PASSWORD` environment variable
+# external_vsphere_datacenter: "DATACENTER_name"
+# external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
+
+## Vsphere version where located VMs
+# external_vsphere_version: "6.7u3"
+
+## Tags for the external vSphere Cloud Provider images
+## gcr.io/cloud-provider-vsphere/cpi/release/manager
+# external_vsphere_cloud_controller_image_tag: "latest"
+## gcr.io/cloud-provider-vsphere/csi/release/syncer
+# vsphere_syncer_image_tag: "v2.5.1"
+## registry.k8s.io/sig-storage/csi-attacher
+# vsphere_csi_attacher_image_tag: "v3.4.0"
+## gcr.io/cloud-provider-vsphere/csi/release/driver
+# vsphere_csi_controller: "v2.5.1"
+## registry.k8s.io/sig-storage/livenessprobe
+# vsphere_csi_liveness_probe_image_tag: "v2.6.0"
+## registry.k8s.io/sig-storage/csi-provisioner
+# vsphere_csi_provisioner_image_tag: "v3.1.0"
+## registry.k8s.io/sig-storage/csi-resizer
+## makes sense only for vSphere version >=7.0
+# vsphere_csi_resizer_tag: "v1.3.0"
+
+## To use vSphere CSI plugin to provision volumes set this value to true
+# vsphere_csi_enabled: true
+# vsphere_csi_controller_replicas: 1
--- /dev/null
+---
+## Etcd auto compaction retention for mvcc key value store in hour
+# etcd_compaction_retention: 0
+
+## Set level of detail for etcd exported metrics, specify 'extensive' to include histogram metrics.
+# etcd_metrics: basic
+
+## Etcd is restricted by default to 512M on systems under 4GB RAM, 512MB is not enough for much more than testing.
+## Set this if your etcd nodes have less than 4GB but you want more RAM for etcd. Set to 0 for unrestricted RAM.
+## This value is only relevant when deploying etcd with `etcd_deployment_type: docker`
+# etcd_memory_limit: "512M"
+
+## Etcd has a default of 2G for its space quota. If you put a value in etcd_memory_limit which is less than
+## etcd_quota_backend_bytes, you may encounter out of memory terminations of the etcd cluster. Please check
+## etcd documentation for more information.
+# 8G is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.
+# etcd_quota_backend_bytes: "2147483648"
+
+# Maximum client request size in bytes the server will accept.
+# etcd is designed to handle small key value pairs typical for metadata.
+# Larger requests will work, but may increase the latency of other requests
+# etcd_max_request_bytes: "1572864"
+
+### ETCD: disable peer client cert authentication.
+# This affects ETCD_PEER_CLIENT_CERT_AUTH variable
+# etcd_peer_client_auth: true
--- /dev/null
+---
+# Kubernetes dashboard
+# RBAC required. see docs/getting-started.md for access details.
+# dashboard_enabled: false
+
+# Helm deployment
+helm_enabled: false
+
+# Registry deployment
+registry_enabled: false
+# registry_namespace: kube-system
+# registry_storage_class: ""
+# registry_disk_size: "10Gi"
+
+# Metrics Server deployment
+metrics_server_enabled: false
+# metrics_server_container_port: 4443
+# metrics_server_kubelet_insecure_tls: true
+# metrics_server_metric_resolution: 15s
+# metrics_server_kubelet_preferred_address_types: "InternalIP,ExternalIP,Hostname"
+
+# Rancher Local Path Provisioner
+local_path_provisioner_enabled: false
+# local_path_provisioner_namespace: "local-path-storage"
+# local_path_provisioner_storage_class: "local-path"
+# local_path_provisioner_reclaim_policy: Delete
+# local_path_provisioner_claim_root: /opt/local-path-provisioner/
+# local_path_provisioner_debug: false
+# local_path_provisioner_image_repo: "rancher/local-path-provisioner"
+# local_path_provisioner_image_tag: "v0.0.22"
+# local_path_provisioner_helper_image_repo: "busybox"
+# local_path_provisioner_helper_image_tag: "latest"
+
+# Local volume provisioner deployment
+local_volume_provisioner_enabled: false
+# local_volume_provisioner_namespace: kube-system
+# local_volume_provisioner_nodelabels:
+# - kubernetes.io/hostname
+# - topology.kubernetes.io/region
+# - topology.kubernetes.io/zone
+# local_volume_provisioner_storage_classes:
+# local-storage:
+# host_dir: /mnt/disks
+# mount_dir: /mnt/disks
+# volume_mode: Filesystem
+# fs_type: ext4
+# fast-disks:
+# host_dir: /mnt/fast-disks
+# mount_dir: /mnt/fast-disks
+# block_cleaner_command:
+# - "/scripts/shred.sh"
+# - "2"
+# volume_mode: Filesystem
+# fs_type: ext4
+# local_volume_provisioner_tolerations:
+# - effect: NoSchedule
+# operator: Exists
+
+# CSI Volume Snapshot Controller deployment, set this to true if your CSI is able to manage snapshots
+# currently, setting cinder_csi_enabled=true would automatically enable the snapshot controller
+# Longhorn is an extenal CSI that would also require setting this to true but it is not included in kubespray
+# csi_snapshot_controller_enabled: false
+# csi snapshot namespace
+# snapshot_controller_namespace: kube-system
+
+# CephFS provisioner deployment
+cephfs_provisioner_enabled: false
+# cephfs_provisioner_namespace: "cephfs-provisioner"
+# cephfs_provisioner_cluster: ceph
+# cephfs_provisioner_monitors: "172.24.0.1:6789,172.24.0.2:6789,172.24.0.3:6789"
+# cephfs_provisioner_admin_id: admin
+# cephfs_provisioner_secret: secret
+# cephfs_provisioner_storage_class: cephfs
+# cephfs_provisioner_reclaim_policy: Delete
+# cephfs_provisioner_claim_root: /volumes
+# cephfs_provisioner_deterministic_names: true
+
+# RBD provisioner deployment
+rbd_provisioner_enabled: false
+# rbd_provisioner_namespace: rbd-provisioner
+# rbd_provisioner_replicas: 2
+# rbd_provisioner_monitors: "172.24.0.1:6789,172.24.0.2:6789,172.24.0.3:6789"
+# rbd_provisioner_pool: kube
+# rbd_provisioner_admin_id: admin
+# rbd_provisioner_secret_name: ceph-secret-admin
+# rbd_provisioner_secret: ceph-key-admin
+# rbd_provisioner_user_id: kube
+# rbd_provisioner_user_secret_name: ceph-secret-user
+# rbd_provisioner_user_secret: ceph-key-user
+# rbd_provisioner_user_secret_namespace: rbd-provisioner
+# rbd_provisioner_fs_type: ext4
+# rbd_provisioner_image_format: "2"
+# rbd_provisioner_image_features: layering
+# rbd_provisioner_storage_class: rbd
+# rbd_provisioner_reclaim_policy: Delete
+
+# Nginx ingress controller deployment
+ingress_nginx_enabled: false
+# ingress_nginx_host_network: false
+ingress_publish_status_address: ""
+# ingress_nginx_nodeselector:
+# kubernetes.io/os: "linux"
+# ingress_nginx_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# ingress_nginx_namespace: "ingress-nginx"
+# ingress_nginx_insecure_port: 80
+# ingress_nginx_secure_port: 443
+# ingress_nginx_configmap:
+# map-hash-bucket-size: "128"
+# ssl-protocols: "TLSv1.2 TLSv1.3"
+# ingress_nginx_configmap_tcp_services:
+# 9000: "default/example-go:8080"
+# ingress_nginx_configmap_udp_services:
+# 53: "kube-system/coredns:53"
+# ingress_nginx_extra_args:
+# - --default-ssl-certificate=default/foo-tls
+# ingress_nginx_termination_grace_period_seconds: 300
+# ingress_nginx_class: nginx
+
+# ALB ingress controller deployment
+ingress_alb_enabled: false
+# alb_ingress_aws_region: "us-east-1"
+# alb_ingress_restrict_scheme: "false"
+# Enables logging on all outbound requests sent to the AWS API.
+# If logging is desired, set to true.
+# alb_ingress_aws_debug: "false"
+
+# Cert manager deployment
+cert_manager_enabled: false
+# cert_manager_namespace: "cert-manager"
+# cert_manager_tolerations:
+# - key: node-role.kubernetes.io/master
+# effect: NoSchedule
+# - key: node-role.kubernetes.io/control-plane
+# effect: NoSchedule
+# cert_manager_affinity:
+# nodeAffinity:
+# preferredDuringSchedulingIgnoredDuringExecution:
+# - weight: 100
+# preference:
+# matchExpressions:
+# - key: node-role.kubernetes.io/control-plane
+# operator: In
+# values:
+# - ""
+# cert_manager_nodeselector:
+# kubernetes.io/os: "linux"
+
+# cert_manager_trusted_internal_ca: |
+# -----BEGIN CERTIFICATE-----
+# [REPLACE with your CA certificate]
+# -----END CERTIFICATE-----
+# cert_manager_leader_election_namespace: kube-system
+
+# MetalLB deployment
+metallb_enabled: false
+metallb_speaker_enabled: true
+# metallb_ip_range:
+# - "10.5.0.50-10.5.0.99"
+# metallb_pool_name: "loadbalanced"
+# metallb_auto_assign: true
+# metallb_avoid_buggy_ips: false
+# metallb_speaker_nodeselector:
+# kubernetes.io/os: "linux"
+# metallb_controller_nodeselector:
+# kubernetes.io/os: "linux"
+# metallb_speaker_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# metallb_controller_tolerations:
+# - key: "node-role.kubernetes.io/master"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# - key: "node-role.kubernetes.io/control-plane"
+# operator: "Equal"
+# value: ""
+# effect: "NoSchedule"
+# metallb_version: v0.12.1
+# metallb_protocol: "layer2"
+# metallb_port: "7472"
+# metallb_memberlist_port: "7946"
+# metallb_additional_address_pools:
+# kube_service_pool:
+# ip_range:
+# - "10.5.1.50-10.5.1.99"
+# protocol: "layer2"
+# auto_assign: false
+# avoid_buggy_ips: false
+# metallb_protocol: "bgp"
+# metallb_peers:
+# - peer_address: 192.0.2.1
+# peer_asn: 64512
+# my_asn: 4200000000
+# - peer_address: 192.0.2.2
+# peer_asn: 64513
+# my_asn: 4200000000
+
+argocd_enabled: false
+# argocd_version: v2.4.7
+# argocd_namespace: argocd
+# Default password:
+# - https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli
+# ---
+# The initial password is autogenerated to be the pod name of the Argo CD API server. This can be retrieved with the command:
+# kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
+# ---
+# Use the following var to set admin password
+# argocd_admin_password: "password"
+
+# The plugin manager for kubectl
+krew_enabled: false
+krew_root_dir: "/usr/local/krew"
--- /dev/null
+---
+# Kubernetes configuration dirs and system namespace.
+# Those are where all the additional config stuff goes
+# the kubernetes normally puts in /srv/kubernetes.
+# This puts them in a sane location and namespace.
+# Editing those values will almost surely break something.
+kube_config_dir: /etc/kubernetes
+kube_script_dir: "{{ bin_dir }}/kubernetes-scripts"
+kube_manifest_dir: "{{ kube_config_dir }}/manifests"
+
+# This is where all the cert scripts and certs will be located
+kube_cert_dir: "{{ kube_config_dir }}/ssl"
+
+# This is where all of the bearer tokens will be stored
+kube_token_dir: "{{ kube_config_dir }}/tokens"
+
+kube_api_anonymous_auth: true
+
+## Change this to use another Kubernetes version, e.g. a current beta release
+kube_version: v1.24.4
+
+# Where the binaries will be downloaded.
+# Note: ensure that you've enough disk space (about 1G)
+local_release_dir: "/tmp/releases"
+# Random shifts for retrying failed ops like pushing/downloading
+retry_stagger: 5
+
+# This is the user that owns tha cluster installation.
+kube_owner: kube
+
+# This is the group that the cert creation scripts chgrp the
+# cert files to. Not really changeable...
+kube_cert_group: kube-cert
+
+# Cluster Loglevel configuration
+kube_log_level: 2
+
+# Directory where credentials will be stored
+credentials_dir: "{{ inventory_dir }}/credentials"
+
+## It is possible to activate / deactivate selected authentication methods (oidc, static token auth)
+# kube_oidc_auth: false
+# kube_token_auth: false
+
+
+## Variables for OpenID Connect Configuration https://kubernetes.io/docs/admin/authentication/
+## To use OpenID you have to deploy additional an OpenID Provider (e.g Dex, Keycloak, ...)
+
+# kube_oidc_url: https:// ...
+# kube_oidc_client_id: kubernetes
+## Optional settings for OIDC
+# kube_oidc_ca_file: "{{ kube_cert_dir }}/ca.pem"
+# kube_oidc_username_claim: sub
+# kube_oidc_username_prefix: 'oidc:'
+# kube_oidc_groups_claim: groups
+# kube_oidc_groups_prefix: 'oidc:'
+
+## Variables to control webhook authn/authz
+# kube_webhook_token_auth: false
+# kube_webhook_token_auth_url: https://...
+# kube_webhook_token_auth_url_skip_tls_verify: false
+
+## For webhook authorization, authorization_modes must include Webhook
+# kube_webhook_authorization: false
+# kube_webhook_authorization_url: https://...
+# kube_webhook_authorization_url_skip_tls_verify: false
+
+# Choose network plugin (cilium, calico, kube-ovn, weave or flannel. Use cni for generic cni plugin)
+# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
+kube_network_plugin: calico
+
+# Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni
+kube_network_plugin_multus: false
+
+# Kubernetes internal network for services, unused block of space.
+kube_service_addresses: 10.233.0.0/18
+
+# internal network. When used, it will assign IP
+# addresses from this range to individual pods.
+# This network must be unused in your network infrastructure!
+kube_pods_subnet: 10.233.64.0/18
+
+# internal network node size allocation (optional). This is the size allocated
+# to each node for pod IP address allocation. Note that the number of pods per node is
+# also limited by the kubelet_max_pods variable which defaults to 110.
+#
+# Example:
+# Up to 64 nodes and up to 254 or kubelet_max_pods (the lowest of the two) pods per node:
+# - kube_pods_subnet: 10.233.64.0/18
+# - kube_network_node_prefix: 24
+# - kubelet_max_pods: 110
+#
+# Example:
+# Up to 128 nodes and up to 126 or kubelet_max_pods (the lowest of the two) pods per node:
+# - kube_pods_subnet: 10.233.64.0/18
+# - kube_network_node_prefix: 25
+# - kubelet_max_pods: 110
+kube_network_node_prefix: 24
+
+# Configure Dual Stack networking (i.e. both IPv4 and IPv6)
+enable_dual_stack_networks: false
+
+# Kubernetes internal network for IPv6 services, unused block of space.
+# This is only used if enable_dual_stack_networks is set to true
+# This provides 4096 IPv6 IPs
+kube_service_addresses_ipv6: fd85:ee78:d8a6:8607::1000/116
+
+# Internal network. When used, it will assign IPv6 addresses from this range to individual pods.
+# This network must not already be in your network infrastructure!
+# This is only used if enable_dual_stack_networks is set to true.
+# This provides room for 256 nodes with 254 pods per node.
+kube_pods_subnet_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+
+# IPv6 subnet size allocated to each for pods.
+# This is only used if enable_dual_stack_networks is set to true
+# This provides room for 254 pods per node.
+kube_network_node_prefix_ipv6: 120
+
+# The port the API Server will be listening on.
+kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
+kube_apiserver_port: 6443 # (https)
+
+# Kube-proxy proxyMode configuration.
+# Can be ipvs, iptables
+kube_proxy_mode: ipvs
+
+# configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
+# must be set to true for MetalLB to work
+kube_proxy_strict_arp: false
+
+# A string slice of values which specify the addresses to use for NodePorts.
+# Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32).
+# The default empty string slice ([]) means to use all local addresses.
+# kube_proxy_nodeport_addresses_cidr is retained for legacy config
+kube_proxy_nodeport_addresses: >-
+ {%- if kube_proxy_nodeport_addresses_cidr is defined -%}
+ [{{ kube_proxy_nodeport_addresses_cidr }}]
+ {%- else -%}
+ []
+ {%- endif -%}
+
+# If non-empty, will use this string as identification instead of the actual hostname
+# kube_override_hostname: >-
+# {%- if cloud_provider is defined and cloud_provider in [ 'aws' ] -%}
+# {%- else -%}
+# {{ inventory_hostname }}
+# {%- endif -%}
+
+## Encrypting Secret Data at Rest
+kube_encrypt_secret_data: false
+
+# Graceful Node Shutdown (Kubernetes >= 1.21.0), see https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/
+# kubelet_shutdown_grace_period had to be greater than kubelet_shutdown_grace_period_critical_pods to allow
+# non-critical podsa to also terminate gracefully
+# kubelet_shutdown_grace_period: 60s
+# kubelet_shutdown_grace_period_critical_pods: 20s
+
+# DNS configuration.
+# Kubernetes cluster name, also will be used as DNS domain
+cluster_name: cluster.local
+# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
+ndots: 2
+# Can be coredns, coredns_dual, manual or none
+dns_mode: coredns
+# Set manual server if using a custom cluster DNS server
+# manual_dns_server: 10.x.x.x
+# Enable nodelocal dns cache
+enable_nodelocaldns: true
+enable_nodelocaldns_secondary: false
+nodelocaldns_ip: 169.254.25.10
+nodelocaldns_health_port: 9254
+nodelocaldns_second_health_port: 9256
+nodelocaldns_bind_metrics_host_ip: false
+nodelocaldns_secondary_skew_seconds: 5
+# nodelocaldns_external_zones:
+# - zones:
+# - example.com
+# - example.io:1053
+# nameservers:
+# - 1.1.1.1
+# - 2.2.2.2
+# cache: 5
+# - zones:
+# - https://mycompany.local:4453
+# nameservers:
+# - 192.168.0.53
+# cache: 0
+# Enable k8s_external plugin for CoreDNS
+enable_coredns_k8s_external: false
+coredns_k8s_external_zone: k8s_external.local
+# Enable endpoint_pod_names option for kubernetes plugin
+enable_coredns_k8s_endpoint_pod_names: false
+
+# Can be docker_dns, host_resolvconf or none
+resolvconf_mode: host_resolvconf
+# Deploy netchecker app to verify DNS resolve as an HTTP service
+deploy_netchecker: false
+# Ip address of the kubernetes skydns service
+skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
+skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
+dns_domain: "{{ cluster_name }}"
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Default: containerd
+container_manager: containerd
+
+# Additional container runtimes
+kata_containers_enabled: false
+
+kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}"
+
+# K8s image pull policy (imagePullPolicy)
+k8s_image_pull_policy: IfNotPresent
+
+# audit log for kubernetes
+kubernetes_audit: false
+
+# define kubelet config dir for dynamic kubelet
+# kubelet_config_dir:
+default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
+
+# pod security policy (RBAC must be enabled either by having 'RBAC' in authorization_modes or kubeadm enabled)
+podsecuritypolicy_enabled: false
+
+# Custom PodSecurityPolicySpec for restricted policy
+# podsecuritypolicy_restricted_spec: {}
+
+# Custom PodSecurityPolicySpec for privileged policy
+# podsecuritypolicy_privileged_spec: {}
+
+# Make a copy of kubeconfig on the host that runs Ansible in {{ inventory_dir }}/artifacts
+# kubeconfig_localhost: false
+# Use ansible_host as external api ip when copying over kubeconfig.
+# kubeconfig_localhost_ansible_host: false
+# Download kubectl onto the host that runs Ansible in {{ bin_dir }}
+# kubectl_localhost: false
+
+# A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.
+# Acceptable options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".
+# kubelet_enforce_node_allocatable: pods
+
+## Optionally reserve resources for OS system daemons.
+# system_reserved: true
+## Uncomment to override default values
+# system_memory_reserved: 512Mi
+# system_cpu_reserved: 500m
+# system_ephemeral_storage_reserved: 2Gi
+## Reservation for master hosts
+# system_master_memory_reserved: 256Mi
+# system_master_cpu_reserved: 250m
+# system_master_ephemeral_storage_reserved: 2Gi
+
+## Eviction Thresholds to avoid system OOMs
+# https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#eviction-thresholds
+# eviction_hard: {}
+# eviction_hard_control_plane: {}
+
+# An alternative flexvolume plugin directory
+# kubelet_flexvolumes_plugins_dir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
+
+## Supplementary addresses that can be added in kubernetes ssl keys.
+## That can be useful for example to setup a keepalived virtual IP
+# supplementary_addresses_in_ssl_keys: [10.0.0.1, 10.0.0.2, 10.0.0.3]
+
+## Running on top of openstack vms with cinder enabled may lead to unschedulable pods due to NoVolumeZoneConflict restriction in kube-scheduler.
+## See https://github.com/kubernetes-sigs/kubespray/issues/2141
+## Set this variable to true to get rid of this issue
+volume_cross_zone_attachment: false
+## Add Persistent Volumes Storage Class for corresponding cloud provider (supported: in-tree OpenStack, Cinder CSI,
+## AWS EBS CSI, Azure Disk CSI, GCP Persistent Disk CSI)
+persistent_volumes_enabled: false
+
+## Container Engine Acceleration
+## Enable container acceleration feature, for example use gpu acceleration in containers
+# nvidia_accelerator_enabled: true
+## Nvidia GPU driver install. Install will by done by a (init) pod running as a daemonset.
+## Important: if you use Ubuntu then you should set in all.yml 'docker_storage_options: -s overlay2'
+## Array with nvida_gpu_nodes, leave empty or comment if you don't want to install drivers.
+## Labels and taints won't be set to nodes if they are not in the array.
+# nvidia_gpu_nodes:
+# - kube-gpu-001
+# nvidia_driver_version: "384.111"
+## flavor can be tesla or gtx
+# nvidia_gpu_flavor: gtx
+## NVIDIA driver installer images. Change them if you have trouble accessing gcr.io.
+# nvidia_driver_install_centos_container: atzedevries/nvidia-centos-driver-installer:2
+# nvidia_driver_install_ubuntu_container: gcr.io/google-containers/ubuntu-nvidia-driver-installer@sha256:7df76a0f0a17294e86f691c81de6bbb7c04a1b4b3d4ea4e7e2cccdc42e1f6d63
+## NVIDIA GPU device plugin image.
+# nvidia_gpu_device_plugin_container: "registry.k8s.io/nvidia-gpu-device-plugin@sha256:0842734032018be107fa2490c98156992911e3e1f2a21e059ff0105b07dd8e9e"
+
+## Support tls min version, Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13.
+# tls_min_version: ""
+
+## Support tls cipher suites.
+# tls_cipher_suites: {}
+# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
+# - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+# - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
+# - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
+# - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
+# - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
+# - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
+# - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
+# - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
+# - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
+# - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
+# - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
+# - TLS_ECDHE_RSA_WITH_RC4_128_SHA
+# - TLS_RSA_WITH_3DES_EDE_CBC_SHA
+# - TLS_RSA_WITH_AES_128_CBC_SHA
+# - TLS_RSA_WITH_AES_128_CBC_SHA256
+# - TLS_RSA_WITH_AES_128_GCM_SHA256
+# - TLS_RSA_WITH_AES_256_CBC_SHA
+# - TLS_RSA_WITH_AES_256_GCM_SHA384
+# - TLS_RSA_WITH_RC4_128_SHA
+
+## Amount of time to retain events. (default 1h0m0s)
+event_ttl_duration: "1h0m0s"
+
+## Automatically renew K8S control plane certificates on first Monday of each month
+auto_renew_certificates: false
+# First Monday of each month
+# auto_renew_certificates_systemd_calendar: "Mon *-*-1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"
--- /dev/null
+---
+# see roles/network_plugin/calico/defaults/main.yml
+
+# the default value of name
+calico_cni_name: k8s-pod-network
+
+## With calico it is possible to distributed routes with border routers of the datacenter.
+## Warning : enabling router peering will disable calico's default behavior ('node mesh').
+## The subnets of each nodes will be distributed by the datacenter router
+# peer_with_router: false
+
+# Enables Internet connectivity from containers
+# nat_outgoing: true
+
+# Enables Calico CNI "host-local" IPAM plugin
+# calico_ipam_host_local: true
+
+# add default ippool name
+# calico_pool_name: "default-pool"
+
+# add default ippool blockSize (defaults kube_network_node_prefix)
+calico_pool_blocksize: 26
+
+# add default ippool CIDR (must be inside kube_pods_subnet, defaults to kube_pods_subnet otherwise)
+# calico_pool_cidr: 1.2.3.4/5
+
+# add default ippool CIDR to CNI config
+# calico_cni_pool: true
+
+# Add default IPV6 IPPool CIDR. Must be inside kube_pods_subnet_ipv6. Defaults to kube_pods_subnet_ipv6 if not set.
+# calico_pool_cidr_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+
+# Add default IPV6 IPPool CIDR to CNI config
+# calico_cni_pool_ipv6: true
+
+# Global as_num (/calico/bgp/v1/global/as_num)
+# global_as_num: "64512"
+
+# If doing peering with node-assigned asn where the globas does not match your nodes, you want this
+# to be true. All other cases, false.
+# calico_no_global_as_num: false
+
+# You can set MTU value here. If left undefined or empty, it will
+# not be specified in calico CNI config, so Calico will use built-in
+# defaults. The value should be a number, not a string.
+# calico_mtu: 1500
+
+# Configure the MTU to use for workload interfaces and tunnels.
+# - If Wireguard is enabled, subtract 60 from your network MTU (i.e 1500-60=1440)
+# - Otherwise, if VXLAN or BPF mode is enabled, subtract 50 from your network MTU (i.e. 1500-50=1450)
+# - Otherwise, if IPIP is enabled, subtract 20 from your network MTU (i.e. 1500-20=1480)
+# - Otherwise, if not using any encapsulation, set to your network MTU (i.e. 1500)
+# calico_veth_mtu: 1440
+
+# Advertise Cluster IPs
+# calico_advertise_cluster_ips: true
+
+# Advertise Service External IPs
+# calico_advertise_service_external_ips:
+# - x.x.x.x/24
+# - y.y.y.y/32
+
+# Adveritse Service LoadBalancer IPs
+# calico_advertise_service_loadbalancer_ips:
+# - x.x.x.x/24
+# - y.y.y.y/16
+
+# Choose data store type for calico: "etcd" or "kdd" (kubernetes datastore)
+# calico_datastore: "kdd"
+
+# Choose Calico iptables backend: "Legacy", "Auto" or "NFT"
+# calico_iptables_backend: "Auto"
+
+# Use typha (only with kdd)
+# typha_enabled: false
+
+# Generate TLS certs for secure typha<->calico-node communication
+# typha_secure: false
+
+# Scaling typha: 1 replica per 100 nodes is adequate
+# Number of typha replicas
+# typha_replicas: 1
+
+# Set max typha connections
+# typha_max_connections_lower_limit: 300
+
+# Set calico network backend: "bird", "vxlan" or "none"
+# bird enable BGP routing, required for ipip and no encapsulation modes
+# calico_network_backend: vxlan
+
+# IP in IP and VXLAN is mutualy exclusive modes.
+# set IP in IP encapsulation mode: "Always", "CrossSubnet", "Never"
+# calico_ipip_mode: 'Never'
+
+# set VXLAN encapsulation mode: "Always", "CrossSubnet", "Never"
+# calico_vxlan_mode: 'Always'
+
+# set VXLAN port and VNI
+# calico_vxlan_vni: 4096
+# calico_vxlan_port: 4789
+
+# Cenable eBPF mode
+# calico_bpf_enabled: false
+
+# If you want to use non default IP_AUTODETECTION_METHOD, IP6_AUTODETECTION_METHOD for calico node set this option to one of:
+# * can-reach=DESTINATION
+# * interface=INTERFACE-REGEX
+# see https://docs.projectcalico.org/reference/node/configuration
+# calico_ip_auto_method: "interface=eth.*"
+# calico_ip6_auto_method: "interface=eth.*"
+
+# Choose the iptables insert mode for Calico: "Insert" or "Append".
+# calico_felix_chaininsertmode: Insert
+
+# If you want use the default route interface when you use multiple interface with dynamique route (iproute2)
+# see https://docs.projectcalico.org/reference/node/configuration : FELIX_DEVICEROUTESOURCEADDRESS
+# calico_use_default_route_src_ipaddr: false
+
+# Enable calico traffic encryption with wireguard
+# calico_wireguard_enabled: false
+
+# Under certain situations liveness and readiness probes may need tunning
+# calico_node_livenessprobe_timeout: 10
+# calico_node_readinessprobe_timeout: 10
+
+# Calico apiserver (only with kdd)
+# calico_apiserver_enabled: false
--- /dev/null
+# see roles/network_plugin/canal/defaults/main.yml
+
+# The interface used by canal for host <-> host communication.
+# If left blank, then the interface is choosing using the node's
+# default route.
+# canal_iface: ""
+
+# Whether or not to masquerade traffic to destinations not within
+# the pod network.
+# canal_masquerade: "true"
--- /dev/null
+---
+# cilium_version: "v1.11.7"
+
+# Log-level
+# cilium_debug: false
+
+# cilium_mtu: ""
+# cilium_enable_ipv4: true
+# cilium_enable_ipv6: false
+
+# Cilium agent health port
+# cilium_agent_health_port: "9879"
+
+# Identity allocation mode selects how identities are shared between cilium
+# nodes by setting how they are stored. The options are "crd" or "kvstore".
+# - "crd" stores identities in kubernetes as CRDs (custom resource definition).
+# These can be queried with:
+# `kubectl get ciliumid`
+# - "kvstore" stores identities in an etcd kvstore.
+# - In order to support External Workloads, "crd" is required
+# - Ref: https://docs.cilium.io/en/stable/gettingstarted/external-workloads/#setting-up-support-for-external-workloads-beta
+# - KVStore operations are only required when cilium-operator is running with any of the below options:
+# - --synchronize-k8s-services
+# - --synchronize-k8s-nodes
+# - --identity-allocation-mode=kvstore
+# - Ref: https://docs.cilium.io/en/stable/internals/cilium_operator/#kvstore-operations
+# cilium_identity_allocation_mode: kvstore
+
+# Etcd SSL dirs
+# cilium_cert_dir: /etc/cilium/certs
+# kube_etcd_cacert_file: ca.pem
+# kube_etcd_cert_file: cert.pem
+# kube_etcd_key_file: cert-key.pem
+
+# Limits for apps
+# cilium_memory_limit: 500M
+# cilium_cpu_limit: 500m
+# cilium_memory_requests: 64M
+# cilium_cpu_requests: 100m
+
+# Overlay Network Mode
+# cilium_tunnel_mode: vxlan
+# Optional features
+# cilium_enable_prometheus: false
+# Enable if you want to make use of hostPort mappings
+# cilium_enable_portmap: false
+# Monitor aggregation level (none/low/medium/maximum)
+# cilium_monitor_aggregation: medium
+# The monitor aggregation flags determine which TCP flags which, upon the
+# first observation, cause monitor notifications to be generated.
+#
+# Only effective when monitor aggregation is set to "medium" or higher.
+# cilium_monitor_aggregation_flags: "all"
+# Kube Proxy Replacement mode (strict/probe/partial)
+# cilium_kube_proxy_replacement: probe
+
+# If upgrading from Cilium < 1.5, you may want to override some of these options
+# to prevent service disruptions. See also:
+# http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
+# cilium_preallocate_bpf_maps: false
+
+# `cilium_tofqdns_enable_poller` is deprecated in 1.8, removed in 1.9
+# cilium_tofqdns_enable_poller: false
+
+# `cilium_enable_legacy_services` is deprecated in 1.6, removed in 1.9
+# cilium_enable_legacy_services: false
+
+# Unique ID of the cluster. Must be unique across all conneted clusters and
+# in the range of 1 and 255. Only relevant when building a mesh of clusters.
+# This value is not defined by default
+# cilium_cluster_id:
+
+# Deploy cilium even if kube_network_plugin is not cilium.
+# This enables to deploy cilium alongside another CNI to replace kube-proxy.
+# cilium_deploy_additionally: false
+
+# Auto direct nodes routes can be used to advertise pods routes in your cluster
+# without any tunelling (with `cilium_tunnel_mode` sets to `disabled`).
+# This works only if you have a L2 connectivity between all your nodes.
+# You wil also have to specify the variable `cilium_native_routing_cidr` to
+# make this work. Please refer to the cilium documentation for more
+# information about this kind of setups.
+# cilium_auto_direct_node_routes: false
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
+# cilium_native_routing_cidr: ""
+
+# Allows to explicitly specify the IPv6 CIDR for native routing.
+# cilium_native_routing_cidr_ipv6: ""
+
+# Enable transparent network encryption.
+# cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+# cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+# cilium_ipsec_node_encryption: false
+
+# If your kernel or distribution does not support WireGuard, Cilium agent can be configured to fall back on the user-space implementation.
+# When this flag is enabled and Cilium detects that the kernel has no native support for WireGuard,
+# it will fallback on the wireguard-go user-space implementation of WireGuard.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+# cilium_wireguard_userspace_fallback: false
+
+# IP Masquerade Agent
+# https://docs.cilium.io/en/stable/concepts/networking/masquerading/
+# By default, all packets from a pod destined to an IP address outside of the cilium_native_routing_cidr range are masqueraded
+# cilium_ip_masq_agent_enable: false
+### A packet sent from a pod to a destination which belongs to any CIDR from the nonMasqueradeCIDRs is not going to be masqueraded
+# cilium_non_masquerade_cidrs:
+# - 10.0.0.0/8
+# - 172.16.0.0/12
+# - 192.168.0.0/16
+# - 100.64.0.0/10
+# - 192.0.0.0/24
+# - 192.0.2.0/24
+# - 192.88.99.0/24
+# - 198.18.0.0/15
+# - 198.51.100.0/24
+# - 203.0.113.0/24
+# - 240.0.0.0/4
+### Indicates whether to masquerade traffic to the link local prefix.
+### If the masqLinkLocal is not set or set to false, then 169.254.0.0/16 is appended to the non-masquerade CIDRs list.
+# cilium_masq_link_local: false
+### A time interval at which the agent attempts to reload config from disk
+# cilium_ip_masq_resync_interval: 60s
+
+# Hubble
+### Enable Hubble without install
+# cilium_enable_hubble: false
+### Enable Hubble Metrics
+# cilium_enable_hubble_metrics: false
+### if cilium_enable_hubble_metrics: true
+# cilium_hubble_metrics: {}
+# - dns
+# - drop
+# - tcp
+# - flow
+# - icmp
+# - http
+### Enable Hubble install
+# cilium_hubble_install: false
+### Enable auto generate certs if cilium_hubble_install: true
+# cilium_hubble_tls_generate: false
+
+# IP address management mode for v1.9+.
+# https://docs.cilium.io/en/v1.9/concepts/networking/ipam/
+# cilium_ipam_mode: kubernetes
+
+# Extra arguments for the Cilium agent
+# cilium_agent_custom_args: []
+
+# For adding and mounting extra volumes to the cilium agent
+# cilium_agent_extra_volumes: []
+# cilium_agent_extra_volume_mounts: []
+
+# cilium_agent_extra_env_vars: []
+
+# cilium_operator_replicas: 2
+
+# The address at which the cillium operator bind health check api
+# cilium_operator_api_serve_addr: "127.0.0.1:9234"
+
+## A dictionary of extra config variables to add to cilium-config, formatted like:
+## cilium_config_extra_vars:
+## var1: "value1"
+## var2: "value2"
+# cilium_config_extra_vars: {}
+
+# For adding and mounting extra volumes to the cilium operator
+# cilium_operator_extra_volumes: []
+# cilium_operator_extra_volume_mounts: []
+
+# Extra arguments for the Cilium Operator
+# cilium_operator_custom_args: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+# cilium_cluster_name: default
+
+# Make Cilium take ownership over the `/etc/cni/net.d` directory on the node, renaming all non-Cilium CNI configurations to `*.cilium_bak`.
+# This ensures no Pods can be scheduled using other CNI plugins during Cilium agent downtime.
+# Available for Cilium v1.10 and up.
+# cilium_cni_exclusive: true
+
+# Configure the log file for CNI logging with retention policy of 7 days.
+# Disable CNI file logging by setting this field to empty explicitly.
+# Available for Cilium v1.12 and up.
+# cilium_cni_log_file: "/var/run/cilium/cilium-cni.log"
+
+# -- Configure cgroup related configuration
+# -- Enable auto mount of cgroup2 filesystem.
+# When `cilium_cgroup_auto_mount` is enabled, cgroup2 filesystem is mounted at
+# `cilium_cgroup_host_root` path on the underlying host and inside the cilium agent pod.
+# If users disable `cilium_cgroup_auto_mount`, it's expected that users have mounted
+# cgroup2 filesystem at the specified `cilium_cgroup_auto_mount` volume, and then the
+# volume will be mounted inside the cilium agent pod at the same path.
+# Available for Cilium v1.11 and up
+# cilium_cgroup_auto_mount: true
+# -- Configure cgroup root where cgroup2 filesystem is mounted on the host
+# cilium_cgroup_host_root: "/run/cilium/cgroupv2"
+
+# Specifies the ratio (0.0-1.0) of total system memory to use for dynamic
+# sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
+# cilium_bpf_map_dynamic_size_ratio: "0.0"
+
+# -- Enables masquerading of IPv4 traffic leaving the node from endpoints.
+# Available for Cilium v1.10 and up
+# cilium_enable_ipv4_masquerade: true
+# -- Enables masquerading of IPv6 traffic leaving the node from endpoints.
+# Available for Cilium v1.10 and up
+# cilium_enable_ipv6_masquerade: true
+
+# -- Enable native IP masquerade support in eBPF
+# cilium_enable_bpf_masquerade: false
+
+# -- Configure whether direct routing mode should route traffic via
+# host stack (true) or directly and more efficiently out of BPF (false) if
+# the kernel supports it. The latter has the implication that it will also
+# bypass netfilter in the host namespace.
+# cilium_enable_host_legacy_routing: true
+
+# -- Enable use of the remote node identity.
+# ref: https://docs.cilium.io/en/v1.7/install/upgrade/#configmap-remote-node-identity
+# cilium_enable_remote_node_identity: true
+
+# -- Enable the use of well-known identities.
+# cilium_enable_well_known_identities: false
+
+# cilium_enable_bpf_clock_probe: true
+
+# -- Whether to enable CNP status updates.
+# cilium_disable_cnp_status_updates: true
--- /dev/null
+# see roles/network_plugin/flannel/defaults/main.yml
+
+## interface that should be used for flannel operations
+## This is actually an inventory cluster-level item
+# flannel_interface:
+
+## Select interface that should be used for flannel operations by regexp on Name or IP
+## This is actually an inventory cluster-level item
+## example: select interface with ip from net 10.0.0.0/23
+## single quote and escape backslashes
+# flannel_interface_regexp: '10\\.0\\.[0-2]\\.\\d{1,3}'
+
+# You can choose what type of flannel backend to use: 'vxlan' or 'host-gw'
+# for experimental backend
+# please refer to flannel's docs : https://github.com/coreos/flannel/blob/master/README.md
+# flannel_backend_type: "vxlan"
+# flannel_vxlan_vni: 1
+# flannel_vxlan_port: 8472
--- /dev/null
+---
+
+# geneve or vlan
+kube_ovn_network_type: geneve
+
+# geneve, vxlan or stt. ATTENTION: some networkpolicy cannot take effect when using vxlan and stt need custom compile ovs kernel module
+kube_ovn_tunnel_type: geneve
+
+## The nic to support container network can be a nic name or a group of regex separated by comma e.g: 'enp6s0f0,eth.*', if empty will use the nic that the default route use.
+# kube_ovn_iface: eth1
+## The MTU used by pod iface in overlay networks (default iface MTU - 100)
+# kube_ovn_mtu: 1333
+
+## Enable hw-offload, disable traffic mirror and set the iface to the physical port. Make sure that there is an IP address bind to the physical port.
+kube_ovn_hw_offload: false
+# traffic mirror
+kube_ovn_traffic_mirror: false
+
+# kube_ovn_pool_cidr_ipv6: fd85:ee78:d8a6:8607::1:0000/112
+# kube_ovn_default_interface_name: eth0
+
+kube_ovn_external_address: 8.8.8.8
+kube_ovn_external_address_ipv6: 2400:3200::1
+kube_ovn_external_dns: alauda.cn
+
+# kube_ovn_default_gateway: 10.233.64.1,fd85:ee78:d8a6:8607::1:0
+kube_ovn_default_gateway_check: true
+kube_ovn_default_logical_gateway: false
+# kube_ovn_default_exclude_ips: 10.16.0.1
+kube_ovn_node_switch_cidr: 100.64.0.0/16
+kube_ovn_node_switch_cidr_ipv6: fd00:100:64::/64
+
+## vlan config, set default interface name and vlan id
+# kube_ovn_default_interface_name: eth0
+kube_ovn_default_vlan_id: 100
+kube_ovn_vlan_name: product
+
+## pod nic type, support: veth-pair or internal-port
+kube_ovn_pod_nic_type: veth_pair
+
+## Enable load balancer
+kube_ovn_enable_lb: true
+
+## Enable network policy support
+kube_ovn_enable_np: true
+
+## Enable external vpc support
+kube_ovn_enable_external_vpc: true
+
+## Enable checksum
+kube_ovn_encap_checksum: true
+
+## enable ssl
+kube_ovn_enable_ssl: false
+
+## dpdk
+kube_ovn_dpdk_enabled: false
--- /dev/null
+# See roles/network_plugin/kube-router//defaults/main.yml
+
+# Enables Pod Networking -- Advertises and learns the routes to Pods via iBGP
+# kube_router_run_router: true
+
+# Enables Network Policy -- sets up iptables to provide ingress firewall for pods
+# kube_router_run_firewall: true
+
+# Enables Service Proxy -- sets up IPVS for Kubernetes Services
+# see docs/kube-router.md "Caveats" section
+# kube_router_run_service_proxy: false
+
+# Add Cluster IP of the service to the RIB so that it gets advertises to the BGP peers.
+# kube_router_advertise_cluster_ip: false
+
+# Add External IP of service to the RIB so that it gets advertised to the BGP peers.
+# kube_router_advertise_external_ip: false
+
+# Add LoadbBalancer IP of service status as set by the LB provider to the RIB so that it gets advertised to the BGP peers.
+# kube_router_advertise_loadbalancer_ip: false
+
+# Adjust manifest of kube-router daemonset template with DSR needed changes
+# kube_router_enable_dsr: false
+
+# Array of arbitrary extra arguments to kube-router, see
+# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md
+# kube_router_extra_args: []
+
+# ASN number of the cluster, used when communicating with external BGP routers
+# kube_router_cluster_asn: ~
+
+# ASN numbers of the BGP peer to which cluster nodes will advertise cluster ip and node's pod cidr.
+# kube_router_peer_router_asns: ~
+
+# The ip address of the external router to which all nodes will peer and advertise the cluster ip and pod cidr's.
+# kube_router_peer_router_ips: ~
+
+# The remote port of the external BGP to which all nodes will peer. If not set, default BGP port (179) will be used.
+# kube_router_peer_router_ports: ~
+
+# Setups node CNI to allow hairpin mode, requires node reboots, see
+# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode
+# kube_router_support_hairpin_mode: false
+
+# Select DNS Policy ClusterFirstWithHostNet, ClusterFirst, etc.
+# kube_router_dns_policy: ClusterFirstWithHostNet
+
+# Array of annotations for master
+# kube_router_annotations_master: []
+
+# Array of annotations for every node
+# kube_router_annotations_node: []
+
+# Array of common annotations for every node
+# kube_router_annotations_all: []
+
+# Enables scraping kube-router metrics with Prometheus
+# kube_router_enable_metrics: false
+
+# Path to serve Prometheus metrics on
+# kube_router_metrics_path: /metrics
+
+# Prometheus metrics port to use
+# kube_router_metrics_port: 9255
--- /dev/null
+---
+# private interface, on a l2-network
+macvlan_interface: "eth1"
+
+# Enable nat in default gateway network interface
+enable_nat_default_gateway: true
--- /dev/null
+# see roles/network_plugin/weave/defaults/main.yml
+
+# Weave's network password for encryption, if null then no network encryption.
+# weave_password: ~
+
+# If set to 1, disable checking for new Weave Net versions (default is blank,
+# i.e. check is enabled)
+# weave_checkpoint_disable: false
+
+# Soft limit on the number of connections between peers. Defaults to 100.
+# weave_conn_limit: 100
+
+# Weave Net defaults to enabling hairpin on the bridge side of the veth pair
+# for containers attached. If you need to disable hairpin, e.g. your kernel is
+# one of those that can panic if hairpin is enabled, then you can disable it by
+# setting `HAIRPIN_MODE=false`.
+# weave_hairpin_mode: true
+
+# The range of IP addresses used by Weave Net and the subnet they are placed in
+# (CIDR format; default 10.32.0.0/12)
+# weave_ipalloc_range: "{{ kube_pods_subnet }}"
+
+# Set to 0 to disable Network Policy Controller (default is on)
+# weave_expect_npc: "{{ enable_network_policy }}"
+
+# List of addresses of peers in the Kubernetes cluster (default is to fetch the
+# list from the api-server)
+# weave_kube_peers: ~
+
+# Set the initialization mode of the IP Address Manager (defaults to consensus
+# amongst the KUBE_PEERS)
+# weave_ipalloc_init: ~
+
+# Set the IP address used as a gateway from the Weave network to the host
+# network - this is useful if you are configuring the addon as a static pod.
+# weave_expose_ip: ~
+
+# Address and port that the Weave Net daemon will serve Prometheus-style
+# metrics on (defaults to 0.0.0.0:6782)
+# weave_metrics_addr: ~
+
+# Address and port that the Weave Net daemon will serve status requests on
+# (defaults to disabled)
+# weave_status_addr: ~
+
+# Weave Net defaults to 1376 bytes, but you can set a smaller size if your
+# underlying network has a tighter limit, or set a larger size for better
+# performance if your network supports jumbo frames (e.g. 8916)
+# weave_mtu: 1376
+
+# Set to 1 to preserve the client source IP address when accessing Service
+# annotated with `service.spec.externalTrafficPolicy=Local`. The feature works
+# only with Weave IPAM (default).
+# weave_no_masq_local: true
+
+# set to nft to use nftables backend for iptables (default is iptables)
+# weave_iptables_backend: iptables
+
+# Extra variables that passing to launch.sh, useful for enabling seed mode, see
+# https://www.weave.works/docs/net/latest/tasks/ipam/ipam/
+# weave_extra_args: ~
+
+# Extra variables for weave_npc that passing to launch.sh, useful for change log level, ex --log-level=error
+# weave_npc_extra_args: ~
--- /dev/null
+# ## Configure 'ip' variable to bind kubernetes services on a
+# ## different ip than the default iface
+# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
+[all]
+# node1 ansible_host=95.54.0.12 # ip=10.3.0.1 etcd_member_name=etcd1
+# node2 ansible_host=95.54.0.13 # ip=10.3.0.2 etcd_member_name=etcd2
+# node3 ansible_host=95.54.0.14 # ip=10.3.0.3 etcd_member_name=etcd3
+# node4 ansible_host=95.54.0.15 # ip=10.3.0.4 etcd_member_name=etcd4
+# node5 ansible_host=95.54.0.16 # ip=10.3.0.5 etcd_member_name=etcd5
+# node6 ansible_host=95.54.0.17 # ip=10.3.0.6 etcd_member_name=etcd6
+
+# ## configure a bastion host if your nodes are not directly reachable
+# [bastion]
+# bastion ansible_host=x.x.x.x ansible_user=some_user
+
+[kube_control_plane]
+# node1
+# node2
+# node3
+
+[etcd]
+# node1
+# node2
+# node3
+
+[kube_node]
+# node2
+# node3
+# node4
+# node5
+# node6
+
+[calico_rr]
+
+[k8s_cluster:children]
+kube_control_plane
+kube_node
+calico_rr
--- /dev/null
+---
+# This is an inventory compatibility playbook to ensure we keep compatibility with old style group names
+
+- name: Add kube-master nodes to kube_control_plane
+ hosts: kube-master
+ gather_facts: false
+ tags: always
+ tasks:
+ - name: add nodes to kube_control_plane group
+ group_by:
+ key: 'kube_control_plane'
+
+- name: Add kube-node nodes to kube_node
+ hosts: kube-node
+ gather_facts: false
+ tags: always
+ tasks:
+ - name: add nodes to kube_node group
+ group_by:
+ key: 'kube_node'
+
+- name: Add k8s-cluster nodes to k8s_cluster
+ hosts: k8s-cluster
+ gather_facts: false
+ tags: always
+ tasks:
+ - name: add nodes to k8s_cluster group
+ group_by:
+ key: 'k8s_cluster'
+
+- name: Add calico-rr nodes to calico_rr
+ hosts: calico-rr
+ gather_facts: false
+ tags: always
+ tasks:
+ - name: add nodes to calico_rr group
+ group_by:
+ key: 'calico_rr'
+
+- name: Add no-floating nodes to no_floating
+ hosts: no-floating
+ gather_facts: false
+ tags: always
+ tasks:
+ - name: add nodes to no-floating group
+ group_by:
+ key: 'no_floating'
--- /dev/null
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+DOCUMENTATION = """
+---
+module: kube
+short_description: Manage Kubernetes Cluster
+description:
+ - Create, replace, remove, and stop resources within a Kubernetes Cluster
+version_added: "2.0"
+options:
+ name:
+ required: false
+ default: null
+ description:
+ - The name associated with resource
+ filename:
+ required: false
+ default: null
+ description:
+ - The path and filename of the resource(s) definition file(s).
+ - To operate on several files this can accept a comma separated list of files or a list of files.
+ aliases: [ 'files', 'file', 'filenames' ]
+ kubectl:
+ required: false
+ default: null
+ description:
+ - The path to the kubectl bin
+ namespace:
+ required: false
+ default: null
+ description:
+ - The namespace associated with the resource(s)
+ resource:
+ required: false
+ default: null
+ description:
+ - The resource to perform an action on. pods (po), replicationControllers (rc), services (svc)
+ label:
+ required: false
+ default: null
+ description:
+ - The labels used to filter specific resources.
+ server:
+ required: false
+ default: null
+ description:
+ - The url for the API server that commands are executed against.
+ force:
+ required: false
+ default: false
+ description:
+ - A flag to indicate to force delete, replace, or stop.
+ wait:
+ required: false
+ default: false
+ description:
+ - A flag to indicate to wait for resources to be created before continuing to the next step
+ all:
+ required: false
+ default: false
+ description:
+ - A flag to indicate delete all, stop all, or all namespaces when checking exists.
+ log_level:
+ required: false
+ default: 0
+ description:
+ - Indicates the level of verbosity of logging by kubectl.
+ state:
+ required: false
+ choices: ['present', 'absent', 'latest', 'reloaded', 'stopped']
+ default: present
+ description:
+ - present handles checking existence or creating if definition file provided,
+ absent handles deleting resource(s) based on other options,
+ latest handles creating or updating based on existence,
+ reloaded handles updating resource(s) definition using definition file,
+ stopped handles stopping resource(s) based on other options.
+ recursive:
+ required: false
+ default: false
+ description:
+ - Process the directory used in -f, --filename recursively.
+ Useful when you want to manage related manifests organized
+ within the same directory.
+requirements:
+ - kubectl
+author: "Kenny Jones (@kenjones-cisco)"
+"""
+
+EXAMPLES = """
+- name: test nginx is present
+ kube: name=nginx resource=rc state=present
+
+- name: test nginx is stopped
+ kube: name=nginx resource=rc state=stopped
+
+- name: test nginx is absent
+ kube: name=nginx resource=rc state=absent
+
+- name: test nginx is present
+ kube: filename=/tmp/nginx.yml
+
+- name: test nginx and postgresql are present
+ kube: files=/tmp/nginx.yml,/tmp/postgresql.yml
+
+- name: test nginx and postgresql are present
+ kube:
+ files:
+ - /tmp/nginx.yml
+ - /tmp/postgresql.yml
+"""
+
+
+class KubeManager(object):
+
+ def __init__(self, module):
+
+ self.module = module
+
+ self.kubectl = module.params.get('kubectl')
+ if self.kubectl is None:
+ self.kubectl = module.get_bin_path('kubectl', True)
+ self.base_cmd = [self.kubectl]
+
+ if module.params.get('server'):
+ self.base_cmd.append('--server=' + module.params.get('server'))
+
+ if module.params.get('log_level'):
+ self.base_cmd.append('--v=' + str(module.params.get('log_level')))
+
+ if module.params.get('namespace'):
+ self.base_cmd.append('--namespace=' + module.params.get('namespace'))
+
+
+ self.all = module.params.get('all')
+ self.force = module.params.get('force')
+ self.wait = module.params.get('wait')
+ self.name = module.params.get('name')
+ self.filename = [f.strip() for f in module.params.get('filename') or []]
+ self.resource = module.params.get('resource')
+ self.label = module.params.get('label')
+ self.recursive = module.params.get('recursive')
+
+ def _execute(self, cmd):
+ args = self.base_cmd + cmd
+ try:
+ rc, out, err = self.module.run_command(args)
+ if rc != 0:
+ self.module.fail_json(
+ msg='error running kubectl (%s) command (rc=%d), out=\'%s\', err=\'%s\'' % (' '.join(args), rc, out, err))
+ except Exception as exc:
+ self.module.fail_json(
+ msg='error running kubectl (%s) command: %s' % (' '.join(args), str(exc)))
+ return out.splitlines()
+
+ def _execute_nofail(self, cmd):
+ args = self.base_cmd + cmd
+ rc, out, err = self.module.run_command(args)
+ if rc != 0:
+ return None
+ return out.splitlines()
+
+ def create(self, check=True, force=True):
+ if check and self.exists():
+ return []
+
+ cmd = ['apply']
+
+ if force:
+ cmd.append('--force')
+
+ if self.wait:
+ cmd.append('--wait')
+
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+
+ if not self.filename:
+ self.module.fail_json(msg='filename required to create')
+
+ cmd.append('--filename=' + ','.join(self.filename))
+
+ return self._execute(cmd)
+
+ def replace(self, force=True):
+
+ cmd = ['apply']
+
+ if force:
+ cmd.append('--force')
+
+ if self.wait:
+ cmd.append('--wait')
+
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+
+ if not self.filename:
+ self.module.fail_json(msg='filename required to reload')
+
+ cmd.append('--filename=' + ','.join(self.filename))
+
+ return self._execute(cmd)
+
+ def delete(self):
+
+ if not self.force and not self.exists():
+ return []
+
+ cmd = ['delete']
+
+ if self.filename:
+ cmd.append('--filename=' + ','.join(self.filename))
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+ else:
+ if not self.resource:
+ self.module.fail_json(msg='resource required to delete without filename')
+
+ cmd.append(self.resource)
+
+ if self.name:
+ cmd.append(self.name)
+
+ if self.label:
+ cmd.append('--selector=' + self.label)
+
+ if self.all:
+ cmd.append('--all')
+
+ if self.force:
+ cmd.append('--ignore-not-found')
+
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+
+ return self._execute(cmd)
+
+ def exists(self):
+ cmd = ['get']
+
+ if self.filename:
+ cmd.append('--filename=' + ','.join(self.filename))
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+ else:
+ if not self.resource:
+ self.module.fail_json(msg='resource required without filename')
+
+ cmd.append(self.resource)
+
+ if self.name:
+ cmd.append(self.name)
+
+ if self.label:
+ cmd.append('--selector=' + self.label)
+
+ if self.all:
+ cmd.append('--all-namespaces')
+
+ cmd.append('--no-headers')
+
+ result = self._execute_nofail(cmd)
+ if not result:
+ return False
+ return True
+
+ # TODO: This is currently unused, perhaps convert to 'scale' with a replicas param?
+ def stop(self):
+
+ if not self.force and not self.exists():
+ return []
+
+ cmd = ['stop']
+
+ if self.filename:
+ cmd.append('--filename=' + ','.join(self.filename))
+ if self.recursive:
+ cmd.append('--recursive={}'.format(self.recursive))
+ else:
+ if not self.resource:
+ self.module.fail_json(msg='resource required to stop without filename')
+
+ cmd.append(self.resource)
+
+ if self.name:
+ cmd.append(self.name)
+
+ if self.label:
+ cmd.append('--selector=' + self.label)
+
+ if self.all:
+ cmd.append('--all')
+
+ if self.force:
+ cmd.append('--ignore-not-found')
+
+ return self._execute(cmd)
+
+
+def main():
+
+ module = AnsibleModule(
+ argument_spec=dict(
+ name=dict(),
+ filename=dict(type='list', aliases=['files', 'file', 'filenames']),
+ namespace=dict(),
+ resource=dict(),
+ label=dict(),
+ server=dict(),
+ kubectl=dict(),
+ force=dict(default=False, type='bool'),
+ wait=dict(default=False, type='bool'),
+ all=dict(default=False, type='bool'),
+ log_level=dict(default=0, type='int'),
+ state=dict(default='present', choices=['present', 'absent', 'latest', 'reloaded', 'stopped', 'exists']),
+ recursive=dict(default=False, type='bool'),
+ ),
+ mutually_exclusive=[['filename', 'list']]
+ )
+
+ changed = False
+
+ manager = KubeManager(module)
+ state = module.params.get('state')
+ if state == 'present':
+ result = manager.create(check=False)
+
+ elif state == 'absent':
+ result = manager.delete()
+
+ elif state == 'reloaded':
+ result = manager.replace()
+
+ elif state == 'stopped':
+ result = manager.stop()
+
+ elif state == 'latest':
+ result = manager.replace()
+
+ elif state == 'exists':
+ result = manager.exists()
+ module.exit_json(changed=changed,
+ msg='%s' % result)
+
+ else:
+ module.fail_json(msg='Unrecognized state %s.' % state)
+
+ module.exit_json(changed=changed,
+ msg='success: %s' % (' '.join(result))
+ )
+
+
+from ansible.module_utils.basic import * # noqa
+if __name__ == '__main__':
+ main()
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"]}
+
+- hosts: etcd[0]
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: recover_control_plane/etcd, when: "not etcd_kubeadm_enabled|default(false)" }
+
+- hosts: kube_control_plane[0]
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: recover_control_plane/control-plane }
+
+- include: cluster.yml
+
+- hosts: kube_control_plane
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: recover_control_plane/post-recover }
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"] }
+
+- hosts: "{{ node | default('etcd:k8s_cluster:calico_rr') }}"
+ gather_facts: no
+ tasks:
+ - name: Confirm Execution
+ pause:
+ prompt: "Are you sure you want to delete nodes state? Type 'yes' to delete nodes."
+ register: pause_result
+ run_once: True
+ when:
+ - not (skip_confirmation | default(false) | bool)
+
+ - name: Fail if user does not confirm deletion
+ fail:
+ msg: "Delete nodes confirmation failed"
+ when: pause_result.user_input | default('yes') != 'yes'
+
+- name: Gather facts
+ import_playbook: facts.yml
+ when: reset_nodes|default(True)|bool
+
+- hosts: "{{ node | default('kube_node') }}"
+ gather_facts: no
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults, when: reset_nodes|default(True)|bool }
+ - { role: remove-node/pre-remove, tags: pre-remove }
+ - { role: remove-node/remove-etcd-node }
+ - { role: reset, tags: reset, when: reset_nodes|default(True)|bool }
+
+# Currently cannot remove first master or etcd
+- hosts: "{{ node | default('kube_control_plane[1:]:etcd[1:]') }}"
+ gather_facts: no
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults, when: reset_nodes|default(True)|bool }
+ - { role: remove-node/post-remove, tags: post-remove }
--- /dev/null
+ansible==4.10.0
+ansible-core==2.11.11
+cryptography==3.4.8
+jinja2==2.11.3
+netaddr==0.7.19
+pbr==5.4.4
+jmespath==0.9.5
+ruamel.yaml==0.16.10
+ruamel.yaml.clib==0.2.6
+MarkupSafe==1.1.1
--- /dev/null
+ansible==5.7.1
+ansible-core==2.12.5
+cryptography==3.4.8
+jinja2==2.11.3
+netaddr==0.7.19
+pbr==5.4.4
+jmespath==0.9.5
+ruamel.yaml==0.16.10
+ruamel.yaml.clib==0.2.6
+MarkupSafe==1.1.1
--- /dev/null
+ansible==5.7.1
+ansible-core==2.12.5
+cryptography==3.4.8
+jinja2==2.11.3
+netaddr==0.7.19
+pbr==5.4.4
+jmespath==0.9.5
+ruamel.yaml==0.16.10
+ruamel.yaml.clib==0.2.6
+MarkupSafe==1.1.1
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"]}
+
+- name: Gather facts
+ import_playbook: facts.yml
+
+- hosts: etcd:k8s_cluster:calico_rr
+ gather_facts: False
+ vars_prompt:
+ name: "reset_confirmation"
+ prompt: "Are you sure you want to reset cluster state? Type 'yes' to reset your cluster."
+ default: "no"
+ private: no
+
+ pre_tasks:
+ - name: check confirmation
+ fail:
+ msg: "Reset confirmation failed"
+ when: reset_confirmation != "yes"
+
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults}
+ - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_early: true }
+ - { role: reset, tags: reset }
--- /dev/null
+ANSIBLE_HOST_IP=155.98.36.82
+ANSIBLE_USERNAME=osc_int
+ANSIBLE_PASSWORD=osc_int
+PROJECT_ROOT=/kubespray/config
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"] }
+
+- name: Bootstrap any new workers
+ hosts: kube_node
+ strategy: linear
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ gather_facts: false
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bootstrap-os, tags: bootstrap-os }
+
+- name: Gather facts
+ tags: always
+ import_playbook: facts.yml
+
+- name: Generate the etcd certificates beforehand
+ hosts: etcd
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: etcd, tags: etcd, etcd_cluster_setup: false }
+
+- name: Download images to ansible host cache via first kube_control_plane node
+ hosts: kube_control_plane[0]
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults, when: "not skip_downloads and download_run_once and not download_localhost" }
+ - { role: kubernetes/preinstall, tags: preinstall, when: "not skip_downloads and download_run_once and not download_localhost" }
+ - { role: download, tags: download, when: "not skip_downloads and download_run_once and not download_localhost" }
+
+- name: Target only workers to get kubelet installed and checking in on any new nodes(engine)
+ hosts: kube_node
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, tags: preinstall }
+ - { role: container-engine, tags: "container-engine", when: deploy_container_engine }
+ - { role: download, tags: download, when: "not skip_downloads" }
+ - { role: etcd, tags: etcd, etcd_cluster_setup: false, when: "etcd_deployment_type != 'kubeadm'" }
+
+- name: Target only workers to get kubelet installed and checking in on any new nodes(node)
+ hosts: kube_node
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/node, tags: node }
+
+- name: Upload control plane certs and retrieve encryption key
+ hosts: kube_control_plane | first
+ environment: "{{ proxy_disable_env }}"
+ gather_facts: False
+ tags: kubeadm
+ roles:
+ - { role: kubespray-defaults }
+ tasks:
+ - name: Upload control plane certificates
+ command: >-
+ {{ bin_dir }}/kubeadm init phase
+ --config {{ kube_config_dir }}/kubeadm-config.yaml
+ upload-certs
+ --upload-certs
+ environment: "{{ proxy_disable_env }}"
+ register: kubeadm_upload_cert
+ changed_when: false
+ - name: set fact 'kubeadm_certificate_key' for later use
+ set_fact:
+ kubeadm_certificate_key: "{{ kubeadm_upload_cert.stdout_lines[-1] | trim }}"
+ when: kubeadm_certificate_key is not defined
+
+- name: Target only workers to get kubelet installed and checking in on any new nodes(network)
+ hosts: kube_node
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/kubeadm, tags: kubeadm }
+ - { role: kubernetes/node-label, tags: node-label }
+ - { role: network_plugin, tags: network }
+
+- name: Apply resolv.conf changes now that cluster DNS is up
+ hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_late: true }
--- /dev/null
+---
+- hosts: all
+ become: true
+ gather_facts: no
+
+ vars:
+ docker_bin_dir: /usr/bin
+ bin_dir: /usr/local/bin
+ ansible_ssh_pipelining: true
+ etcd_cert_dir: /etc/ssl/etcd/ssl
+ kube_network_plugin: calico
+ archive_dirname: collect-info
+ commands:
+ - name: timedate_info
+ cmd: timedatectl status
+ - name: kernel_info
+ cmd: uname -r
+ - name: docker_info
+ cmd: "{{ docker_bin_dir }}/docker info"
+ - name: ip_info
+ cmd: ip -4 -o a
+ - name: route_info
+ cmd: ip ro
+ - name: proc_info
+ cmd: ps auxf | grep -v ]$
+ - name: systemctl_failed_info
+ cmd: systemctl --state=failed --no-pager
+ - name: k8s_info
+ cmd: "{{ bin_dir }}/kubectl get all --all-namespaces -o wide"
+ - name: errors_info
+ cmd: journalctl -p err --no-pager
+ - name: etcd_info
+ cmd: "{{ bin_dir }}/etcdctl endpoint --cluster health"
+ - name: calico_info
+ cmd: "{{ bin_dir }}/calicoctl node status"
+ when: '{{ kube_network_plugin == "calico" }}'
+ - name: calico_workload_info
+ cmd: "{{ bin_dir }}/calicoctl get workloadEndpoint -o wide"
+ when: '{{ kube_network_plugin == "calico" }}'
+ - name: calico_pool_info
+ cmd: "{{ bin_dir }}/calicoctl get ippool -o wide"
+ when: '{{ kube_network_plugin == "calico" }}'
+ - name: weave_info
+ cmd: weave report
+ when: '{{ kube_network_plugin == "weave" }}'
+ - name: weave_logs
+ cmd: "{{ docker_bin_dir }}/docker logs weave"
+ when: '{{ kube_network_plugin == "weave" }}'
+ - name: kube_describe_all
+ cmd: "{{ bin_dir }}/kubectl describe all --all-namespaces"
+ - name: kube_describe_nodes
+ cmd: "{{ bin_dir }}/kubectl describe nodes"
+ - name: kubelet_logs
+ cmd: journalctl -u kubelet --no-pager
+ - name: coredns_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l k8s-app=coredns -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: apiserver_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: controller_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l component=kube-controller-manager -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: scheduler_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l component=kube-scheduler -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: proxy_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: nginx_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l k8s-app=kube-nginx -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system; done"
+ - name: flannel_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l app=flannel -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system flannel-container; done"
+ when: '{{ kube_network_plugin == "flannel" }}'
+ - name: canal_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l k8s-app=canal-node -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system flannel; done"
+ when: '{{ kube_network_plugin == "canal" }}'
+ - name: calico_policy_logs
+ cmd: "for i in `{{ bin_dir }}/kubectl get pods -n kube-system -l k8s-app=calico-kube-controllers -o jsonpath={.items..metadata.name}`;
+ do {{ bin_dir }}/kubectl logs ${i} -n kube-system ; done"
+ when: '{{ kube_network_plugin in ["canal", "calico"] }}'
+ - name: helm_show_releases_history
+ cmd: "for i in `{{ bin_dir }}/helm list -q`; do {{ bin_dir }}/helm history ${i} --col-width=0; done"
+ when: "{{ helm_enabled|default(true) }}"
+
+ logs:
+ - /var/log/syslog
+ - /var/log/daemon.log
+ - /var/log/kern.log
+ - /var/log/dpkg.log
+ - /var/log/apt/history.log
+ - /var/log/yum.log
+ - /var/log/messages
+ - /var/log/dmesg
+
+ environment:
+ ETCDCTL_API: 3
+ ETCDCTL_CERT: "{{ etcd_cert_dir }}/admin-{{ inventory_hostname }}.pem"
+ ETCDCTL_KEY: "{{ etcd_cert_dir }}/admin-{{ inventory_hostname }}-key.pem"
+ ETCDCTL_CACERT: "{{ etcd_cert_dir }}/ca.pem"
+ ETCDCTL_ENDPOINTS: "{{ etcd_access_addresses }}"
+
+ tasks:
+ - name: set etcd_access_addresses
+ set_fact:
+ etcd_access_addresses: |-
+ {% for item in groups['etcd'] -%}
+ https://{{ item }}:2379{% if not loop.last %},{% endif %}
+ {%- endfor %}
+ when: "'etcd' in groups"
+
+ - name: Storing commands output
+ shell: "{{ item.cmd }} &> {{ item.name }}"
+ failed_when: false
+ with_items: "{{ commands }}"
+ when: item.when | default(True)
+ no_log: True
+
+ - name: Fetch results
+ fetch: src={{ item.name }} dest=/tmp/{{ archive_dirname }}/commands
+ with_items: "{{ commands }}"
+ when: item.when | default(True)
+ failed_when: false
+
+ - name: Fetch logs
+ fetch: src={{ item }} dest=/tmp/{{ archive_dirname }}/logs
+ with_items: "{{ logs }}"
+ failed_when: false
+
+ - name: Pack results and logs
+ archive:
+ path: "/tmp/{{ archive_dirname }}"
+ dest: "{{ dir|default('.') }}/logs.tar.gz"
+ remove: true
+ mode: 0640
+ delegate_to: localhost
+ connection: local
+ become: false
+ run_once: true
+
+ - name: Clean up collected command outputs
+ file: path={{ item.name }} state=absent
+ with_items: "{{ commands }}"
--- /dev/null
+#!/bin/bash
+
+cp -rfp inventory/sample inventory/oransc-cluster
+. sample_env
+declare -a IPS=($ANSIBLE_HOST_IP)
+CONFIG_FILE=inventory/oransc-cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
+ansible-playbook -i inventory/oransc-cluster/hosts.yaml --become --private-key inventory/oransc-cluster/id_rsa cluster.yml
+
+#sshpass -p $ANSIBLE_PASSWORD scp -o StrictHostKeyChecking=no -i inventory/oransc-cluster/id_rsa -q root@$ANSIBLE_HOST_IP:/root/.kube/config ${PROJECT_ROOT}/config
+scp -o StrictHostKeyChecking=no -i inventory/oransc-cluster/id_rsa -q root@$ANSIBLE_HOST_IP:/root/.kube/config ${PROJECT_ROOT}/config
+sed -i "s/127.0.0.1/${ANSIBLE_HOST_IP}/g" "${PROJECT_ROOT}"/config
--- /dev/null
+#!/usr/bin/env python3
+
+# After a new version of Kubernetes has been released,
+# run this script to update roles/download/defaults/main.yml
+# with new hashes.
+
+import hashlib
+import sys
+
+import requests
+from ruamel.yaml import YAML
+
+MAIN_YML = "../roles/download/defaults/main.yml"
+
+def open_main_yaml():
+ yaml = YAML()
+ yaml.explicit_start = True
+ yaml.preserve_quotes = True
+ yaml.width = 4096
+
+ with open(MAIN_YML, "r") as main_yml:
+ data = yaml.load(main_yml)
+
+ return data, yaml
+
+
+def download_hash(versions):
+ architectures = ["arm", "arm64", "amd64", "ppc64le"]
+ downloads = ["kubelet", "kubectl", "kubeadm"]
+
+ data, yaml = open_main_yaml()
+
+ for download in downloads:
+ checksum_name = f"{download}_checksums"
+ for arch in architectures:
+ for version in versions:
+ if not version.startswith("v"):
+ version = f"v{version}"
+ url = f"https://storage.googleapis.com/kubernetes-release/release/{version}/bin/linux/{arch}/{download}"
+ download_file = requests.get(url, allow_redirects=True)
+ download_file.raise_for_status()
+ sha256sum = hashlib.sha256(download_file.content).hexdigest()
+ data[checksum_name][arch][version] = sha256sum
+
+ with open(MAIN_YML, "w") as main_yml:
+ yaml.dump(data, main_yml)
+ print(f"\n\nUpdated {MAIN_YML}\n")
+
+
+def usage():
+ print(f"USAGE:\n {sys.argv[0]} [k8s_version1] [[k8s_version2]....[k8s_versionN]]")
+
+
+def main(argv=None):
+ if not argv:
+ argv = sys.argv[1:]
+ if not argv:
+ usage()
+ return 1
+ download_hash(argv)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
--- /dev/null
+#!/bin/sh
+set -eo pipefail
+
+VERSIONS="$@"
+ARCHITECTURES="arm arm64 amd64 ppc64le"
+DOWNLOADS="kubelet kubectl kubeadm"
+DOWNLOAD_DIR="tmp/kubeadm_hasher"
+
+if [ -z "$VERSIONS" ]; then
+ echo "USAGE: $0 <versions>"
+ exit 1
+fi
+
+mkdir -p ${DOWNLOAD_DIR}
+for download in ${DOWNLOADS}; do
+ echo -e "\n\n${download}_checksums:"
+ for arch in ${ARCHITECTURES}; do
+ echo -e " ${arch}:"
+ for version in ${VERSIONS}; do
+ TARGET="${DOWNLOAD_DIR}/${download}-$version-$arch"
+ if [ ! -f ${TARGET} ]; then
+ curl -L -f -S -s -o ${TARGET} "https://storage.googleapis.com/kubernetes-release/release/${version}/bin/linux/${arch}/${download}"
+ fi
+ echo -e " ${version}: $(sha256sum ${TARGET} | awk '{print $1}')"
+ done
+ done
+done
+echo -e "\n\nAdd these values to roles/download/defaults/main.yml"
--- /dev/null
+#!/bin/sh
+set -eo pipefail
+
+#Generate MD formatted tags from roles and cluster yaml files
+printf "|%25s |%9s\n" "Tag name" "Used for"
+echo "|--------------------------|---------"
+tags=$(grep -r tags: . | perl -ne '/tags:\s\[?(([\w\-_]+,?\s?)+)/ && printf "%s ", "$1"'|\
+ perl -ne 'print join "\n", split /\s|,/' | sort -u)
+for tag in $tags; do
+ match=$(cat docs/ansible.md | perl -ne "/^\|\s+${tag}\s\|\s+((\S+\s?)+)/ && printf \$1")
+ printf "|%25s |%s\n" "${tag}" " ${match}"
+done
--- /dev/null
+openrc
+venv
--- /dev/null
+# gitlab-branch-cleanup
+
+Cleanup old branches in a GitLab project
+
+## Installation
+
+```shell
+pip install -r requirements.txt
+python main.py --help
+```
+
+## Usage
+
+```console
+$ export GITLAB_API_TOKEN=foobar
+$ python main.py kargo-ci/kubernetes-sigs-kubespray
+Deleting branch pr-5220-containerd-systemd from 2020-02-17 ...
+Deleting branch pr-5561-feature/cinder_csi_fixes from 2020-02-17 ...
+Deleting branch pr-5607-add-flatcar from 2020-02-17 ...
+Deleting branch pr-5616-fix-typo from 2020-02-17 ...
+Deleting branch pr-5634-helm_310 from 2020-02-18 ...
+Deleting branch pr-5644-patch-1 from 2020-02-15 ...
+Deleting branch pr-5647-master from 2020-02-17 ...
+```
--- /dev/null
+import gitlab
+import argparse
+import os
+import sys
+from datetime import timedelta, datetime, timezone
+
+
+parser = argparse.ArgumentParser(
+ description='Cleanup old branches in a GitLab project')
+parser.add_argument('--api', default='https://gitlab.com/',
+ help='URL of GitLab API, defaults to gitlab.com')
+parser.add_argument('--age', type=int, default=30,
+ help='Delete branches older than this many days')
+parser.add_argument('--prefix', default='pr-',
+ help='Cleanup only branches with names matching this prefix')
+parser.add_argument('--dry-run', action='store_true',
+ help='Do not delete anything')
+parser.add_argument('project',
+ help='Path of the GitLab project')
+
+args = parser.parse_args()
+limit = datetime.now(timezone.utc) - timedelta(days=args.age)
+
+if os.getenv('GITLAB_API_TOKEN', '') == '':
+ print("Environment variable GITLAB_API_TOKEN is required.")
+ sys.exit(2)
+
+gl = gitlab.Gitlab(args.api, private_token=os.getenv('GITLAB_API_TOKEN'))
+gl.auth()
+
+p = gl.projects.get(args.project)
+for b in p.branches.list(all=True):
+ date = datetime.fromisoformat(b.commit['created_at'])
+ if date < limit and not b.protected and not b.default and b.name.startswith(args.prefix):
+ print("Deleting branch %s from %s ..." %
+ (b.name, date.date().isoformat()))
+ if not args.dry_run:
+ b.delete()
--- /dev/null
+python-gitlab
--- /dev/null
+#!/bin/sh
+
+docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/cache:/srv/gitlab-runner/cache -v /srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:v1.10.0
+
+#
+#/srv/gitlab-runner/config# cat config.toml
+#concurrent = 10
+#check_interval = 1
+
+#[[runners]]
+# name = "2edf3d71fe19"
+# url = "https://gitlab.com"
+# token = "THE TOKEN-CHANGEME"
+# executor = "docker"
+# [runners.docker]
+# tls_verify = false
+# image = "docker:latest"
+# privileged = true
+# disable_cache = false
+# cache_dir = "/srv/gitlab-runner/cache"
+# volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/srv/gitlab-runner/cache:/cache:rw"]
+# [runners.cache]
--- /dev/null
+# openstack-cleanup
+
+Tool to deletes openstack servers older than a specific age (default 4h).
+
+Useful to cleanup orphan servers that are left behind when CI is manually cancelled or fails unexpectedly.
+
+## Installation
+
+```shell
+pip install -r requirements.txt
+python main.py --help
+```
+
+## Usage
+
+```console
+$ python main.py
+This will delete VMs... (ctrl+c to cancel)
+Will delete server example1
+Will delete server example2
+```
--- /dev/null
+#!/usr/bin/env python
+import argparse
+import openstack
+import logging
+import datetime
+import time
+
+DATE_FORMAT = '%Y-%m-%dT%H:%M:%SZ'
+PAUSE_SECONDS = 5
+
+log = logging.getLogger('openstack-cleanup')
+
+parser = argparse.ArgumentParser(description='Cleanup OpenStack resources')
+
+parser.add_argument('-v', '--verbose', action='store_true',
+ help='Increase verbosity')
+parser.add_argument('--hours', type=int, default=4,
+ help='Age (in hours) of VMs to cleanup (default: 4h)')
+parser.add_argument('--dry-run', action='store_true',
+ help='Do not delete anything')
+
+args = parser.parse_args()
+
+oldest_allowed = datetime.datetime.now() - datetime.timedelta(hours=args.hours)
+
+
+def main():
+ if args.dry_run:
+ print('Running in dry-run mode')
+ else:
+ print('This will delete resources... (ctrl+c to cancel)')
+ time.sleep(PAUSE_SECONDS)
+
+ conn = openstack.connect()
+
+ print('Servers...')
+ map_if_old(conn.compute.delete_server,
+ conn.compute.servers())
+
+ print('Security groups...')
+ map_if_old(conn.network.delete_security_group,
+ conn.network.security_groups())
+
+ print('Ports...')
+ try:
+ map_if_old(conn.network.delete_port,
+ conn.network.ports())
+ except openstack.exceptions.ConflictException as ex:
+ # Need to find subnet-id which should be removed from a router
+ for sn in conn.network.subnets():
+ try:
+ fn_if_old(conn.network.delete_subnet, sn)
+ except openstack.exceptions.ConflictException:
+ for r in conn.network.routers():
+ print("Deleting subnet %s from router %s", sn, r)
+ try:
+ conn.network.remove_interface_from_router(
+ r, subnet_id=sn.id)
+ except Exception as ex:
+ print("Failed to delete subnet from router as %s", ex)
+
+ # After removing unnecessary subnet from router, retry to delete ports
+ map_if_old(conn.network.delete_port,
+ conn.network.ports())
+
+ print('Subnets...')
+ map_if_old(conn.network.delete_subnet,
+ conn.network.subnets())
+
+ print('Networks...')
+ for n in conn.network.networks():
+ if not n.is_router_external:
+ fn_if_old(conn.network.delete_network, n)
+
+
+# runs the given fn to all elements of the that are older than allowed
+def map_if_old(fn, items):
+ for item in items:
+ fn_if_old(fn, item)
+
+
+# run the given fn function only if the passed item is older than allowed
+def fn_if_old(fn, item):
+ created_at = datetime.datetime.strptime(item.created_at, DATE_FORMAT)
+ if item.name == "default": # skip default security group
+ return
+ if created_at < oldest_allowed:
+ print('Will delete %(name)s (%(id)s)' % item)
+ if not args.dry_run:
+ fn(item)
+
+
+if __name__ == '__main__':
+ # execute only if run as a script
+ main()
--- /dev/null
+openstacksdk>=0.43.0
--- /dev/null
+#!/bin/bash
+# A naive premoderation script to allow Gitlab CI pipeline on a specific PRs' comment
+# Exits with 0, if the pipeline is good to go
+# Exits with 1, if the user is not allowed to start pipeline
+# Exits with 2, if script is unable to get issue id from CI_BUILD_REF_NAME variable
+# Exits with 3, if missing the magic comment in the pipeline to start the pipeline
+
+CURL_ARGS="-fs --retry 4 --retry-delay 5"
+MAGIC="${MAGIC:-ci check this}"
+exit_code=0
+
+# Get PR number from CI_BUILD_REF_NAME
+issue=$(echo ${CI_BUILD_REF_NAME} | perl -ne '/^pr-(\d+)-\S+$/ && print $1')
+
+if [ "$issue" = "" ]; then
+ echo "Unable to get issue id from: $CI_BUILD_REF_NAME"
+ exit 2
+fi
+
+echo "Fetching labels from PR $issue"
+labels=$(curl ${CURL_ARGS} "https://api.github.com/repos/kubernetes-sigs/kubespray/issues/${issue}?access_token=${GITHUB_TOKEN}" | jq '{labels: .labels}' | jq '.labels[].name' | jq -s '')
+labels_to_patch=$(echo -n $labels | jq '. + ["needs-ci-auth"]' | tr -d "\n")
+
+echo "Checking for '$MAGIC' comment in PR $issue"
+
+# Get the user name from the PR comments with the wanted magic incantation casted
+user=$(curl ${CURL_ARGS} "https://api.github.com/repos/kubernetes-sigs/kubespray/issues/${issue}/comments" | jq -M "map(select(.body | contains (\"$MAGIC\"))) | .[0] .user.login" | tr -d '"')
+
+# Check for the required user group membership to allow (exit 0) or decline (exit >0) the pipeline
+if [ "$user" = "" ] || [ "$user" = "null" ]; then
+ echo "Missing '$MAGIC' comment from one of the OWNERS"
+ exit_code=3
+else
+ echo "Found comment from user: $user"
+
+ curl ${CURL_ARGS} "https://api.github.com/orgs/kubernetes-sigs/members/${user}"
+
+ if [ $? -ne 0 ]; then
+ echo "User does not have permissions to start CI run"
+ exit_code=1
+ else
+ labels_to_patch=$(echo -n $labels | jq '. - ["needs-ci-auth"]' | tr -d "\n")
+ exit_code=0
+ echo "$user has allowed CI to start"
+ fi
+fi
+
+# Patch labels on PR
+curl ${CURL_ARGS} --request PATCH "https://api.github.com/repos/kubernetes-sigs/kubespray/issues/${issue}?access_token=${GITHUB_TOKEN}" -H "Content-Type: application/json" -d "{\"labels\": ${labels_to_patch}}"
+
+exit $exit_code
--- /dev/null
+[metadata]
+name = kubespray
+summary = Ansible modules for installing Kubernetes
+description-file =
+ README.md
+author = Kubespray
+author-email = smainklh@gmail.com
+license = Apache License (2.0)
+home-page = https://github.com/kubernetes-sigs/kubespray
+classifier =
+ License :: OSI Approved :: Apache Software License
+ Development Status :: 4 - Beta
+ Intended Audience :: Developers
+ Intended Audience :: System Administrators
+ Intended Audience :: Information Technology
+ Topic :: Utilities
+
+[global]
+setup-hooks =
+ pbr.hooks.setup_hook
+
+[files]
+data_files =
+ usr/share/kubespray/playbooks/ =
+ cluster.yml
+ upgrade-cluster.yml
+ scale.yml
+ reset.yml
+ remove-node.yml
+ extra_playbooks/upgrade-only-k8s.yml
+ usr/share/kubespray/roles = roles/*
+ usr/share/kubespray/library = library/*
+ usr/share/doc/kubespray/ =
+ LICENSE
+ README.md
+ usr/share/doc/kubespray/inventory/ =
+ inventory/sample/inventory.ini
+ etc/kubespray/ =
+ ansible.cfg
+ etc/kubespray/inventory/sample/group_vars/ =
+ inventory/sample/group_vars/etcd.yml
+ etc/kubespray/inventory/sample/group_vars/all/ =
+ inventory/sample/group_vars/all/all.yml
+ inventory/sample/group_vars/all/azure.yml
+ inventory/sample/group_vars/all/coreos.yml
+ inventory/sample/group_vars/all/docker.yml
+ inventory/sample/group_vars/all/oci.yml
+ inventory/sample/group_vars/all/openstack.yml
+
+[wheel]
+universal = 1
+
+[pbr]
+skip_authors = True
+skip_changelog = True
+
+[bdist_rpm]
+group = "System Environment/Libraries"
+requires =
+ ansible
+ python-jinja2
+ python-netaddr
--- /dev/null
+# Copyright Red Hat, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import setuptools
+
+setuptools.setup(
+ setup_requires=['pbr'],
+ pbr=True)
--- /dev/null
+deploy:
+ ansible-playbook -i hosts.ini -e docker_password=$(docker_password) cluster.yml
--- /dev/null
+# See the OWNERS docs at https://go.k8s.io/owners
+
+approvers:
+ - woopstar
+ - ant31
+reviewers:
+ - woopstar
+ - ant31
--- /dev/null
+---
+- hosts: image-builder
+ gather_facts: false
+ roles:
+ - kubevirt-images
--- /dev/null
+image-builder-1 ansible_ssh_host=xxx.xxx.xxx.xxx
+
+[image-builder]
+image-builder-1
--- /dev/null
+---
+images_dir: /images/base
+
+docker_user: kubespray+buildvmimages
+docker_host: quay.io
+registry: quay.io/kubespray
+
+images:
+ ubuntu-1604:
+ filename: xenial-server-cloudimg-amd64-disk1.img
+ url: https://storage.googleapis.com/kubespray-images/ubuntu/xenial-server-cloudimg-amd64-disk1.img
+ checksum: sha256:c0d099383cd064390b568e20d1c39a9c68ba864764404b70f754a7b1b2f808f7
+ converted: false
+ tag: "latest"
+
+ ubuntu-1804:
+ filename: bionic-server-cloudimg-amd64.img
+ url: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
+ checksum: sha256:c3d0e03f4245ffaabd7647e6dabf346b944a62b9934d0a89f3a04b4236386af2
+ converted: false
+ tag: "latest"
+
+ ubuntu-2004:
+ filename: focal-server-cloudimg-amd64.img
+ url: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
+ checksum: sha256:8faf1f5a27c956ad0c49dac3114a355fbaf1b2d21709e10a18e67213fbb95b81
+ converted: false
+ tag: "latest"
+
+ ubuntu-2204:
+ filename: jammy-server-cloudimg-amd64.img
+ url: https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img
+ checksum: sha256:d3f3f446bf35b2e58b82c10c8fa65525264efe5b0e398238f00ab670f49528ab
+ converted: false
+ tag: "latest"
+
+ fedora-35:
+ filename: Fedora-Cloud-Base-35-1.2.x86_64.qcow2
+ url: https://download.fedoraproject.org/pub/fedora/linux/releases/35/Cloud/x86_64/images/Fedora-Cloud-Base-35-1.2.x86_64.qcow2
+ checksum: sha256:fe84502779b3477284a8d4c86731f642ca10dd3984d2b5eccdf82630a9ca2de6
+ converted: true
+ tag: "latest"
+
+ fedora-36:
+ filename: Fedora-Cloud-Base-36-1.5.x86_64.qcow2
+ url: https://download.fedoraproject.org/pub/fedora/linux/releases/36/Cloud/x86_64/images/Fedora-Cloud-Base-36-1.5.x86_64.qcow2
+ checksum: sha256:ca9e514cc2f4a7a0188e7c68af60eb4e573d2e6850cc65b464697223f46b4605
+ converted: true
+ tag: "latest"
+
+ fedora-coreos:
+ filename: fedora-coreos-32.20200601.3.0-openstack.x86_64.qcow2.xz
+ url: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200601.3.0/x86_64/fedora-coreos-32.20200601.3.0-openstack.x86_64.qcow2.xz
+ checksum: sha256:fe78c348189d745eb5f6f80ff9eb2af67da8e84880d264f4301faaf7c2a72646
+ converted: true
+ tag: "latest"
+
+ centos-7:
+ filename: CentOS-7-x86_64-GenericCloud-2009.qcow2
+ url: http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-2009.qcow2
+ checksum: sha256:e38bab0475cc6d004d2e17015969c659e5a308111851b0e2715e84646035bdd3
+ converted: true
+ tag: "latest"
+
+ centos-8:
+ filename: CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2
+ url: http://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2
+ checksum: sha256:7ec97062618dc0a7ebf211864abf63629da1f325578868579ee70c495bed3ba0
+ converted: true
+ tag: "latest"
+
+ almalinux-8:
+ filename: AlmaLinux-8-GenericCloud-latest.x86_64.qcow2
+ url: https://repo.almalinux.org/almalinux/8.5/cloud/x86_64/images/AlmaLinux-8-GenericCloud-8.5-20211119.x86_64.qcow2
+ checksum: sha256:d629247b12802157be127db53a7fcb484b80fceae9896d750c953a51a8c6688f
+ converted: true
+ tag: "latest"
+
+ rockylinux-8:
+ filename: Rocky-8-GenericCloud-8.6-20220515.x86_64.qcow2
+ url: https://download.rockylinux.org/pub/rocky/8.6/images/Rocky-8-GenericCloud-8.6-20220515.x86_64.qcow2
+ checksum: sha256:77e79f487c70f6bfa5655d8084e02cb8d31900a2c2a22b2334c3401b40a1231c
+ converted: true
+ tag: "latest"
+
+ debian-9:
+ filename: debian-9-openstack-amd64.qcow2
+ url: https://cdimage.debian.org/cdimage/openstack/current-9/debian-9-openstack-amd64.qcow2
+ checksum: sha256:01d9345ba7a6523d214d2eaabe07fe7b4b69b28e63d7a6b322521e99e5768719
+ converted: true
+ tag: "latest"
+
+ debian-10:
+ filename: debian-10-openstack-amd64.qcow2
+ url: https://cdimage.debian.org/cdimage/openstack/current-10/debian-10-openstack-amd64.qcow2
+ checksum: sha512:296ad8345cb49e52464a0cb8bf4365eb0b9e4220c47ebdd73d134d51effc756d5554aee15027fffd038fef4ad5fa984c94208bce60572d58b2ab26f74bb2a5de
+ converted: true
+ tag: "latest"
+
+ debian-11:
+ filename: debian-11-generic-amd64-20210814-734.qcow2
+ url: https://cdimage.debian.org/cdimage/cloud/bullseye/20210814-734/debian-11-generic-amd64-20210814-734.qcow2
+ checksum: sha512:ed680265ce925e3e02336b052bb476883e2d3b023f7b7d39d064d58ba5f1856869f75dca637c27c0303b731d082ff23a7e45ea2e3f9bcb8f3c4ce0c24332885d
+ converted: true
+ tag: "latest"
+
+ oracle-7:
+ filename: oracle-linux-76.qcow2
+ url: https://storage.googleapis.com/born-images/oracle76/oracle-linux-76.qcow2
+ checksum: sha256:f396c03e907fa2a0c94d6807b9f62622f23ee3499df4456ae2a15da381fbdca5
+ converted: true
+ tag: "latest"
+
+ opensuse-leap-15:
+ filename: openSUSE-Leap-15.3.x86_64-1.0.1-NoCloud-Build2.63.qcow2
+ url: https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.3/images/openSUSE-Leap-15.3.x86_64-1.0.1-NoCloud-Build2.63.qcow2
+ checksum: sha256:289248945e2d058551c71c1bdcb31a361cefe7136c7fd88a09b524eedfaf5215
+ converted: true
+ tag: "latest"
--- /dev/null
+---
+
+- name: Create image directory
+ file:
+ state: directory
+ path: "{{ images_dir }}"
+ mode: 0755
+
+- name: Download images files
+ get_url:
+ url: "{{ item.value.url }}"
+ dest: "{{ images_dir }}/{{ item.value.filename }}"
+ checksum: "{{ item.value.checksum }}"
+ loop: "{{ images|dict2items }}"
+
+- name: Unxz compressed images
+ command: unxz --force {{ images_dir }}/{{ item.value.filename }}
+ loop: "{{ images|dict2items }}"
+ when:
+ - item.value.filename.endswith('.xz')
+
+- name: Convert images which is not in qcow2 format
+ command: qemu-img convert -O qcow2 {{ images_dir }}/{{ item.value.filename.rstrip('.xz') }} {{ images_dir }}/{{ item.key }}.qcow2
+ loop: "{{ images|dict2items }}"
+ when:
+ - not (item.value.converted|bool)
+
+- name: Make sure all images are ending with qcow2
+ command: cp {{ images_dir }}/{{ item.value.filename.rstrip('.xz') }} {{ images_dir }}/{{ item.key }}.qcow2
+ loop: "{{ images|dict2items }}"
+ when:
+ - item.value.converted|bool
+
+- name: Resize images # noqa 301
+ command: qemu-img resize {{ images_dir }}/{{ item.key }}.qcow2 +8G
+ loop: "{{ images|dict2items }}"
+
+# STEP 2: Include the images inside a container
+- name: Template default Dockerfile
+ template:
+ src: Dockerfile
+ dest: "{{ images_dir }}/Dockerfile"
+ mode: 0644
+
+- name: Create docker images for each OS # noqa 301
+ command: docker build -t {{ registry }}/vm-{{ item.key }}:{{ item.value.tag }} --build-arg cloud_image="{{ item.key }}.qcow2" {{ images_dir }}
+ loop: "{{ images|dict2items }}"
+
+- name: docker login # noqa 301
+ command: docker login -u="{{ docker_user }}" -p="{{ docker_password }}" "{{ docker_host }}"
+
+- name: docker push image # noqa 301
+ command: docker push {{ registry }}/vm-{{ item.key }}:{{ item.value.tag }}
+ loop: "{{ images|dict2items }}"
+
+- name: docker logout # noqa 301
+ command: docker logout -u="{{ docker_user }}" "{{ docker_host }}"
--- /dev/null
+FROM kubevirt/registry-disk-v1alpha
+
+ARG cloud_image
+MAINTAINER "The Kubespray Project" <kubespray@googlegroups.com>
+
+COPY $cloud_image /disk
--- /dev/null
+# Docker image published at quay.io/kubespray/vagrant
+
+ARG KUBESPRAY_VERSION
+FROM quay.io/kubespray/kubespray:${KUBESPRAY_VERSION}
+
+ENV VAGRANT_VERSION=2.2.19
+ENV VAGRANT_DEFAULT_PROVIDER=libvirt
+ENV VAGRANT_ANSIBLE_TAGS=facts
+
+RUN apt-get update && apt-get install -y wget libvirt-dev openssh-client rsync git
+
+# Install Vagrant
+RUN wget https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_x86_64.deb && \
+ dpkg -i vagrant_${VAGRANT_VERSION}_x86_64.deb && \
+ rm vagrant_${VAGRANT_VERSION}_x86_64.deb && \
+ vagrant plugin install vagrant-libvirt
--- /dev/null
+# vagrant docker image
+
+This image is used for the vagrant CI jobs. It is using the libvirt driver.
+
+## Usage
+
+```console
+$ docker run --net host --rm -it -v /var/run/libvirt/libvirt-sock:/var/run/libvirt/libvirt-sock quay.io/kubespray/vagrant
+$ vagrant up
+Bringing machine 'k8s-1' up with 'libvirt' provider...
+Bringing machine 'k8s-2' up with 'libvirt' provider...
+Bringing machine 'k8s-3' up with 'libvirt' provider...
+[...]
+```
+
+## Cache
+
+You can set `/root/kubespray_cache` as a volume to keep cache between runs.
+
+## Building
+
+```shell
+./build.sh v2.12.5
+```
--- /dev/null
+#!/bin/sh
+set -euo pipefail
+
+if [ "$#" -ne 1 ]; then
+ echo "Usage: $0 tag" >&2
+ exit 1
+fi
+
+VERSION="$1"
+IMG="quay.io/kubespray/vagrant:${VERSION}"
+
+docker build . --build-arg "KUBESPRAY_VERSION=${VERSION}" --tag "$IMG"
+docker push "$IMG"
--- /dev/null
+---
+- name: Check ansible version
+ import_playbook: ansible_version.yml
+
+- name: Ensure compatibility with old groups
+ import_playbook: legacy_groups.yml
+
+- hosts: bastion[0]
+ gather_facts: False
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bastion-ssh-config, tags: ["localhost", "bastion"] }
+
+- hosts: k8s_cluster:etcd:calico_rr
+ strategy: linear
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ gather_facts: false
+ environment: "{{ proxy_disable_env }}"
+ vars:
+ # Need to disable pipelining for bootstrap-os as some systems have requiretty in sudoers set, which makes pipelining
+ # fail. bootstrap-os fixes this on these systems, so in later plays it can be enabled.
+ ansible_ssh_pipelining: false
+ roles:
+ - { role: kubespray-defaults }
+ - { role: bootstrap-os, tags: bootstrap-os}
+
+- name: Gather facts
+ tags: always
+ import_playbook: facts.yml
+
+- name: Download images to ansible host cache via first kube_control_plane node
+ hosts: kube_control_plane[0]
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults, when: "not skip_downloads and download_run_once and not download_localhost"}
+ - { role: kubernetes/preinstall, tags: preinstall, when: "not skip_downloads and download_run_once and not download_localhost" }
+ - { role: download, tags: download, when: "not skip_downloads and download_run_once and not download_localhost" }
+
+- name: Prepare nodes for upgrade
+ hosts: k8s_cluster:etcd:calico_rr
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, tags: preinstall }
+ - { role: download, tags: download, when: "not skip_downloads" }
+
+- name: Upgrade container engine on non-cluster nodes
+ hosts: etcd:calico_rr:!k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ serial: "{{ serial | default('20%') }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: container-engine, tags: "container-engine", when: deploy_container_engine }
+
+- hosts: etcd
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - role: etcd
+ tags: etcd
+ vars:
+ etcd_cluster_setup: true
+ etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
+ when: etcd_deployment_type != "kubeadm"
+
+- hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - role: etcd
+ tags: etcd
+ vars:
+ etcd_cluster_setup: false
+ etcd_events_cluster_setup: false
+ when: etcd_deployment_type != "kubeadm"
+
+- name: Handle upgrades to master components first to maintain backwards compat.
+ gather_facts: False
+ hosts: kube_control_plane
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ serial: 1
+ roles:
+ - { role: kubespray-defaults }
+ - { role: upgrade/pre-upgrade, tags: pre-upgrade }
+ - { role: container-engine, tags: "container-engine", when: deploy_container_engine }
+ - { role: kubernetes/node, tags: node }
+ - { role: kubernetes/control-plane, tags: master, upgrade_cluster_setup: true }
+ - { role: kubernetes/client, tags: client }
+ - { role: kubernetes/node-label, tags: node-label }
+ - { role: kubernetes-apps/cluster_roles, tags: cluster-roles }
+ - { role: kubernetes-apps, tags: csi-driver }
+ - { role: upgrade/post-upgrade, tags: post-upgrade }
+
+- name: Upgrade calico and external cloud provider on all masters, calico-rrs, and nodes
+ hosts: kube_control_plane:calico_rr:kube_node
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ serial: "{{ serial | default('20%') }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes-apps/external_cloud_controller, tags: external-cloud-controller }
+ - { role: network_plugin, tags: network }
+ - { role: kubernetes-apps/network_plugin, tags: network }
+ - { role: kubernetes-apps/policy_controller, tags: policy-controller }
+
+- name: Finally handle worker upgrades, based on given batch size
+ hosts: kube_node:calico_rr:!kube_control_plane
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ serial: "{{ serial | default('20%') }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: upgrade/pre-upgrade, tags: pre-upgrade }
+ - { role: container-engine, tags: "container-engine", when: deploy_container_engine }
+ - { role: kubernetes/node, tags: node }
+ - { role: kubernetes/kubeadm, tags: kubeadm }
+ - { role: kubernetes/node-label, tags: node-label }
+ - { role: upgrade/post-upgrade, tags: post-upgrade }
+
+- hosts: kube_control_plane[0]
+ gather_facts: False
+ any_errors_fatal: true
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: win_nodes/kubernetes_patch, tags: ["master", "win_nodes"] }
+
+- hosts: calico_rr
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: network_plugin/calico/rr, tags: network }
+
+- hosts: kube_control_plane
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes-apps/ingress_controller, tags: ingress-controller }
+ - { role: kubernetes-apps/external_provisioner, tags: external-provisioner }
+ - { role: kubernetes-apps, tags: apps }
+
+- name: Apply resolv.conf changes now that cluster DNS is up
+ hosts: k8s_cluster
+ gather_facts: False
+ any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
+ environment: "{{ proxy_disable_env }}"
+ roles:
+ - { role: kubespray-defaults }
+ - { role: kubernetes/preinstall, when: "dns_mode != 'none' and resolvconf_mode == 'host_resolvconf'", tags: resolvconf, dns_late: true }
Code Review / it / test.git / commitdiff
