1 # Adding/replacing a node
3 Modified from [comments in #3471](https://github.com/kubernetes-sigs/kubespray/issues/3471#issuecomment-530036084)
5 ## Limitation: Removal of first kube_control_plane and etcd-master
7 Currently you can't remove the first node in your kube_control_plane and etcd-master list. If you still want to remove this node you have to:
9 ### 1) Change order of current control planes
11 Modify the order of your control plane list by pushing your first entry to any other position. E.g. if you want to remove `node-1` of the following example:
32 change your inventory to:
53 ## 2) Upgrade the cluster
55 run `upgrade-cluster.yml` or `cluster.yml`. Now you are good to go on with the removal.
57 ## Adding/replacing a worker node
59 This should be the easiest.
61 ### 1) Add new node to the inventory
63 ### 2) Run `scale.yml`
65 You can use `--limit=NODE_NAME` to limit Kubespray to avoid disturbing other nodes in the cluster.
67 Before using `--limit` run playbook `facts.yml` without the limit to refresh facts cache for all nodes.
69 ### 3) Remove an old node with remove-node.yml
71 With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
73 If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars: `-e node=NODE_NAME -e reset_nodes=false -e allow_ungraceful_removal=true`.
74 Use this flag even when you remove other types of nodes like a control plane or etcd nodes.
76 ### 4) Remove the node from the inventory
80 ## Adding/replacing a control plane node
82 ### 1) Run `cluster.yml`
84 Append the new host to the inventory and run `cluster.yml`. You can NOT use `scale.yml` for that.
86 ### 2) Restart kube-system/nginx-proxy
88 In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.
92 docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart
95 ### 3) Remove old control plane nodes
97 With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=NODE_NAME` to the playbook to limit the execution to the node being removed.
98 If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
100 ## Replacing a first control plane node
102 ### 1) Change control plane nodes order in inventory
122 ### 2) Remove old first control plane node from cluster
124 With the old node still in the inventory, run `remove-node.yml`. You need to pass `-e node=node-1` to the playbook to limit the execution to the node being removed.
125 If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
127 ### 3) Edit cluster-info configmap in kube-public namespace
129 `kubectl edit cm -n kube-public cluster-info`
131 Change ip of old kube_control_plane node with ip of live kube_control_plane node (`server` field). Also, update `certificate-authority-data` field if you changed certs.
133 ### 4) Add new control plane node
135 Update inventory (if needed)
137 Run `cluster.yml` with `--limit=kube_control_plane`
139 ## Adding an etcd node
141 You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replace or scale up operation. Either add two new nodes or remove an old one.
143 ### 1) Add the new node running cluster.yml
145 Update the inventory and run `cluster.yml` passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`.
146 If the node you want to add as an etcd node is already a worker or control plane node in your cluster, you have to remove him first using `remove-node.yml`.
148 Run `upgrade-cluster.yml` also passing `--limit=etcd,kube_control_plane -e ignore_assert_errors=yes`. This is necessary to update all etcd configuration in the cluster.
150 At this point, you will have an even number of nodes.
151 Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node.
152 Even so, running applications should continue to be available.
154 If you add multiple etcd nodes with one run, you might want to append `-e etcd_retries=10` to increase the amount of retries between each etcd node join.
155 Otherwise the etcd cluster might still be processing the first join and fail on subsequent nodes. `etcd_retries=10` might work to join 3 new nodes.
157 ### 2) Add the new node to apiserver config
159 In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure the new etcd nodes are present in the apiserver command line parameter `--etcd-servers=...`.
161 ## Removing an etcd node
163 ### 1) Remove an old etcd node
165 With the node still in the inventory, run `remove-node.yml` passing `-e node=NODE_NAME` as the name of the node that should be removed.
166 If the node you want to remove is not online, you should add `reset_nodes=false` and `allow_ungraceful_removal=true` to your extra-vars.
168 ### 2) Make sure only remaining nodes are in your inventory
170 Remove `NODE_NAME` from your inventory file.
172 ### 3) Update kubernetes and network configuration files with the valid list of etcd members
174 Run `cluster.yml` to regenerate the configuration files on all remaining nodes.
176 ### 4) Remove the old etcd node from apiserver config
178 In every control plane node, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`. Make sure only active etcd nodes are still present in the apiserver command line parameter `--etcd-servers=...`.
180 ### 5) Shutdown the old instance