1 # Configurable Parameters in Kubespray
3 ## Generic Ansible variables
5 You can view facts gathered by Ansible automatically
6 [here](https://docs.ansible.com/ansible/latest/user_guide/playbooks_vars_facts.html#ansible-facts).
8 Some variables of note include:
10 * *ansible_user*: user to connect to via SSH
11 * *ansible_default_ipv4.address*: IP address Ansible automatically chooses.
12 Generated based on the output from the command ``ip -4 route get 8.8.8.8``
14 ## Common vars that are used in Kubespray
16 * *calico_version* - Specify version of Calico to use
17 * *calico_cni_version* - Specify version of Calico CNI plugin to use
18 * *docker_version* - Specify version of Docker to use (should be quoted
19 string). Must match one of the keys defined for *docker_versioned_pkg*
20 in `roles/container-engine/docker/vars/*.yml`.
21 * *containerd_version* - Specify version of containerd to use when setting `container_manager` to `containerd`
22 * *docker_containerd_version* - Specify which version of containerd to use when setting `container_manager` to `docker`
23 * *etcd_version* - Specify version of ETCD to use
24 * *calico_ipip_mode* - Configures Calico ipip encapsulation - valid values are 'Never', 'Always' and 'CrossSubnet' (default 'Never')
25 * *calico_vxlan_mode* - Configures Calico vxlan encapsulation - valid values are 'Never', 'Always' and 'CrossSubnet' (default 'Always')
26 * *calico_network_backend* - Configures Calico network backend - valid values are 'none', 'bird' and 'vxlan' (default 'vxlan')
27 * *kube_network_plugin* - Sets k8s network plugin (default Calico)
28 * *kube_proxy_mode* - Changes k8s proxy mode to iptables mode
29 * *kube_version* - Specify a given Kubernetes version
30 * *searchdomains* - Array of DNS domains to search when looking up hostnames
31 * *nameservers* - Array of nameservers to use for DNS lookup
32 * *preinstall_selinux_state* - Set selinux state, permitted values are permissive, enforcing and disabled.
34 ## Addressing variables
36 * *ip* - IP to use for binding services (host var)
37 * *access_ip* - IP for other hosts to use to connect to. Often required when
38 deploying from a cloud, such as OpenStack or GCE and you have separate
39 public/floating and private IPs.
40 * *ansible_default_ipv4.address* - Not Kubespray-specific, but it is used if ip
41 and access_ip are undefined
42 * *ip6* - IPv6 address to use for binding services. (host var)
43 If *enable_dual_stack_networks* is set to ``true`` and *ip6* is defined,
44 kubelet's ``--node-ip`` and node's ``InternalIP`` will be the combination of *ip* and *ip6*.
45 * *loadbalancer_apiserver* - If defined, all hosts will connect to this
46 address instead of localhost for kube_control_planes and kube_control_plane[0] for
47 kube_nodes. See more details in the
48 [HA guide](/docs/ha-mode.md).
49 * *loadbalancer_apiserver_localhost* - makes all hosts to connect to
50 the apiserver internally load balanced endpoint. Mutual exclusive to the
51 `loadbalancer_apiserver`. See more details in the
52 [HA guide](/docs/ha-mode.md).
56 Kubernetes needs some parameters in order to get deployed. These are the
57 following default cluster parameters:
59 * *cluster_name* - Name of cluster (default is cluster.local)
61 * *container_manager* - Container Runtime to install in the nodes (default is containerd)
63 * *image_command_tool* - Tool used to pull images (default depends on `container_manager`
64 and is `nerdctl` for `containerd`, `crictl` for `crio`, `docker` for `docker`)
66 * *image_command_tool_on_localhost* - Tool used to pull images on localhost
67 (default is equal to `image_command_tool`)
69 * *dns_domain* - Name of cluster DNS domain (default is cluster.local)
71 * *kube_network_plugin* - Plugin to use for container networking
73 * *kube_service_addresses* - Subnet for cluster IPs (default is
74 10.233.0.0/18). Must not overlap with kube_pods_subnet
76 * *kube_pods_subnet* - Subnet for Pod IPs (default is 10.233.64.0/18). Must not
77 overlap with kube_service_addresses.
79 * *kube_network_node_prefix* - Subnet allocated per-node for pod IPs. Remaining
80 bits in kube_pods_subnet dictates how many kube_nodes can be in cluster. Setting this > 25 will
81 raise an assertion in playbooks if the `kubelet_max_pods` var also isn't adjusted accordingly
82 (assertion not applicable to calico which doesn't use this as a hard limit, see
83 [Calico IP block sizes](https://docs.projectcalico.org/reference/resources/ippool#block-sizes).
85 * *enable_dual_stack_networks* - Setting this to true will provision both IPv4 and IPv6 networking for pods and services.
87 * *kube_service_addresses_ipv6* - Subnet for cluster IPv6 IPs (default is ``fd85:ee78:d8a6:8607::1000/116``). Must not overlap with ``kube_pods_subnet_ipv6``.
89 * *kube_pods_subnet_ipv6* - Subnet for Pod IPv6 IPs (default is ``fd85:ee78:d8a6:8607::1:0000/112``). Must not overlap with ``kube_service_addresses_ipv6``.
91 * *kube_network_node_prefix_ipv6* - Subnet allocated per-node for pod IPv6 IPs. Remaining bits in ``kube_pods_subnet_ipv6`` dictates how many kube_nodes can be in cluster.
93 * *skydns_server* - Cluster IP for DNS (default is 10.233.0.3)
95 * *skydns_server_secondary* - Secondary Cluster IP for CoreDNS used with coredns_dual deployment (default is 10.233.0.4)
97 * *enable_coredns_k8s_external* - If enabled, it configures the [k8s_external plugin](https://coredns.io/plugins/k8s_external/)
98 on the CoreDNS service.
100 * *coredns_k8s_external_zone* - Zone that will be used when CoreDNS k8s_external plugin is enabled
101 (default is k8s_external.local)
103 * *enable_coredns_k8s_endpoint_pod_names* - If enabled, it configures endpoint_pod_names option for kubernetes plugin.
104 on the CoreDNS service.
106 * *cloud_provider* - Enable extra Kubelet option if operating inside GCE or
107 OpenStack (default is unset)
109 * *kube_feature_gates* - A list of key=value pairs that describe feature gates for
110 alpha/experimental Kubernetes features. (defaults is `[]`).
111 Additionally, you can use also the following variables to individually customize your kubernetes components installation (they works exactly like `kube_feature_gates`):
112 * *kube_apiserver_feature_gates*
113 * *kube_controller_feature_gates*
114 * *kube_scheduler_feature_gates*
115 * *kube_proxy_feature_gates*
116 * *kubelet_feature_gates*
118 * *kubeadm_feature_gates* - A list of key=value pairs that describe feature gates for
119 alpha/experimental Kubeadm features. (defaults is `[]`)
121 * *authorization_modes* - A list of [authorization mode](
122 https://kubernetes.io/docs/admin/authorization/#using-flags-for-your-authorization-module)
123 that the cluster should be configured for. Defaults to `['Node', 'RBAC']`
124 (Node and RBAC authorizers).
125 Note: `Node` and `RBAC` are enabled by default. Previously deployed clusters can be
126 converted to RBAC mode. However, your apps which rely on Kubernetes API will
127 require a service account and cluster role bindings. You can override this
128 setting by setting authorization_modes to `[]`.
130 * *kube_apiserver_admission_control_config_file* - Enable configuration for `kube-apiserver` admission plugins.
131 Currently this variable allow you to configure the `EventRateLimit` admission plugin.
133 To configure the **EventRateLimit** plugin you have to define a data structure like this:
136 kube_apiserver_admission_event_rate_limits:
149 * *kube_apiserver_service_account_lookup* - Enable validation service account before validating token. Default `true`.
151 Note, if cloud providers have any use of the ``10.233.0.0/16``, like instances'
152 private addresses, make sure to pick another values for ``kube_service_addresses``
153 and ``kube_pods_subnet``, for example from the ``172.18.0.0/16``.
155 ## Enabling Dual Stack (IPV4 + IPV6) networking
157 If *enable_dual_stack_networks* is set to ``true``, Dual Stack networking will be enabled in the cluster. This will use the default IPv4 and IPv6 subnets specified in the defaults file in the ``kubespray-defaults`` role, unless overridden of course. The default config will give you room for up to 256 nodes with 126 pods per node, and up to 4096 services.
161 By default, hosts are set up with 8.8.8.8 as an upstream DNS server and all
162 other settings from your existing /etc/resolv.conf are lost. Set the following
163 variables to match your requirements.
165 * *upstream_dns_servers* - Array of upstream DNS servers configured on host in
166 addition to Kubespray deployed DNS
167 * *nameservers* - Array of DNS servers configured for use by hosts
168 * *searchdomains* - Array of up to 4 search domains
169 * *dns_etchosts* - Content of hosts file for coredns and nodelocaldns
171 For more information, see [DNS
172 Stack](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/dns-stack.md).
174 ## Other service variables
176 * *docker_options* - Commonly used to set
177 ``--insecure-registry=myregistry.mydomain:5000``
178 * *docker_plugins* - This list can be used to define [Docker plugins](https://docs.docker.com/engine/extend/) to install.
179 * *containerd_default_runtime* - If defined, changes the default Containerd runtime used by the Kubernetes CRI plugin.
180 * *containerd_additional_runtimes* - Sets the additional Containerd runtimes used by the Kubernetes CRI plugin.
181 [Default config](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/container-engine/containerd/defaults/main.yml) can be overriden in inventory vars.
182 * *http_proxy/https_proxy/no_proxy/no_proxy_exclude_workers/additional_no_proxy* - Proxy variables for deploying behind a
183 proxy. Note that no_proxy defaults to all internal cluster IPs and hostnames
184 that correspond to each node.
185 * *kubelet_cgroup_driver* - Allows manual override of the cgroup-driver option for Kubelet.
186 By default autodetection is used to match container manager configuration.
187 `systemd` is the preferred driver for `containerd` though it can have issues with `cgroups v1` and `kata-containers` in which case you may want to change to `cgroupfs`.
188 * *kubelet_rotate_certificates* - Auto rotate the kubelet client certificates by requesting new certificates
189 from the kube-apiserver when the certificate expiration approaches.
190 * *kubelet_rotate_server_certificates* - Auto rotate the kubelet server certificates by requesting new certificates
191 from the kube-apiserver when the certificate expiration approaches.
192 **Note** that server certificates are **not** approved automatically. Approve them manually
193 (`kubectl get csr`, `kubectl certificate approve`) or implement custom approving controller like
194 [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp).
195 * *kubelet_streaming_connection_idle_timeout* - Set the maximum time a streaming connection can be idle before the connection is automatically closed.
196 * *kubelet_make_iptables_util_chains* - If `true`, causes the kubelet ensures a set of `iptables` rules are present on host.
197 * *node_labels* - Labels applied to nodes via kubelet --node-labels parameter.
198 For example, labels can be set in the inventory as variables or more widely in group_vars.
199 *node_labels* can only be defined as a dict:
203 label1_name: label1_value
204 label2_name: label2_value
207 * *node_taints* - Taints applied to nodes via kubelet --register-with-taints parameter.
208 For example, taints can be set in the inventory as variables or more widely in group_vars.
209 *node_taints* has to be defined as a list of strings in format `key=value:effect`, e.g.:
213 - "node.example.com/external=true:NoSchedule"
216 * *podsecuritypolicy_enabled* - When set to `true`, enables the PodSecurityPolicy admission controller and defines two policies `privileged` (applying to all resources in `kube-system` namespace and kubelet) and `restricted` (applying all other namespaces).
217 Addons deployed in kube-system namespaces are handled.
218 * *kubernetes_audit* - When set to `true`, enables Auditing.
219 The auditing parameters can be tuned via the following variables (which default values are shown below):
220 * `audit_log_path`: /var/log/audit/kube-apiserver-audit.log
221 * `audit_log_maxage`: 30
222 * `audit_log_maxbackups`: 1
223 * `audit_log_maxsize`: 100
224 * `audit_policy_file`: "{{ kube_config_dir }}/audit-policy/apiserver-audit-policy.yaml"
226 By default, the `audit_policy_file` contains [default rules](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubernetes/control-plane/templates/apiserver-audit-policy.yaml.j2) that can be overridden with the `audit_policy_custom_rules` variable.
227 * *kubernetes_audit_webhook* - When set to `true`, enables the webhook audit backend.
228 The webhook parameters can be tuned via the following variables (which default values are shown below):
229 * `audit_webhook_config_file`: "{{ kube_config_dir }}/audit-policy/apiserver-audit-webhook-config.yaml"
230 * `audit_webhook_server_url`: `"https://audit.app"`
231 * `audit_webhook_server_extra_args`: {}
232 * `audit_webhook_mode`: batch
233 * `audit_webhook_batch_max_size`: 100
234 * `audit_webhook_batch_max_wait`: 1s
236 ### Custom flags for Kube Components
238 For all kube components, custom flags can be passed in. This allows for edge cases where users need changes to the default deployment that may not be applicable to all deployments.
240 Extra flags for the kubelet can be specified using these variables,
241 in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubelet YAML config file. The `kubelet_node_config_extra_args` apply kubelet settings only to nodes and not control planes. Example:
244 kubelet_config_extra_args:
246 memory.available: "100Mi"
247 evictionSoftGracePeriod:
248 memory.available: "30s"
250 memory.available: "300Mi"
253 The possible vars are:
255 * *kubelet_config_extra_args*
256 * *kubelet_node_config_extra_args*
258 Previously, the same parameters could be passed as flags to kubelet binary with the following vars:
260 * *kubelet_custom_flags*
261 * *kubelet_node_custom_flags*
263 The `kubelet_node_custom_flags` apply kubelet settings only to nodes and not control planes. Example:
266 kubelet_custom_flags:
267 - "--eviction-hard=memory.available<100Mi"
268 - "--eviction-soft-grace-period=memory.available=30s"
269 - "--eviction-soft=memory.available<300Mi"
272 This alternative is deprecated and will remain until the flags are completely removed from kubelet
274 Extra flags for the API server, controller, and scheduler components can be specified using these variables,
275 in the form of dicts of key-value pairs of configuration parameters that will be inserted into the kubeadm YAML config file:
277 * *kube_kubeadm_apiserver_extra_args*
278 * *kube_kubeadm_controller_extra_args*
279 * *kube_kubeadm_scheduler_extra_args*
283 * *helm_version* - Only supports v3.x. Existing v2 installs (with Tiller) will not be modified and need to be removed manually.