XTesting/kubespray/docs/hardening.md

   1 # Cluster Hardening
   2
   3 If you want to improve the security on your cluster and make it compliant with the [CIS Benchmarks](https://learn.cisecurity.org/benchmarks), here you can find a configuration to harden your **kubernetes** installation.
   4
   5 To apply the hardening configuration, create a file (eg. `hardening.yaml`) and paste the content of the following code snippet into that.
   6
   7 ## Minimum Requirements
   8
   9 The **kubernetes** version should be at least `v1.23.6` to have all the most recent security features (eg. the new `PodSecurity` admission plugin, etc).
  10
  11 **N.B.** Some of these configurations have just been added to **kubespray**, so ensure that you have the latest version to make it works properly. Also, ensure that other configurations doesn't override these.
  12
  13 `hardening.yaml`:
  14
  15 ```yaml
  16 # Hardening
  17 ---
  18
  19 ## kube-apiserver
  20 authorization_modes: ['Node','RBAC']
  21 # AppArmor-based OS
  22 #kube_apiserver_feature_gates: ['AppArmor=true']
  23 kube_apiserver_request_timeout: 120s
  24 kube_apiserver_service_account_lookup: true
  25
  26 # enable kubernetes audit
  27 kubernetes_audit: true
  28 audit_log_path: "/var/log/kube-apiserver-log.json"
  29 audit_log_maxage: 30
  30 audit_log_maxbackups: 10
  31 audit_log_maxsize: 100
  32
  33 tls_min_version: VersionTLS12
  34 tls_cipher_suites:
  35   - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  36   - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  37   - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  38
  39 # enable encryption at rest
  40 kube_encrypt_secret_data: true
  41 kube_encryption_resources: [secrets]
  42 kube_encryption_algorithm: "secretbox"
  43
  44 kube_apiserver_enable_admission_plugins: ['EventRateLimit,AlwaysPullImages,ServiceAccount,NamespaceLifecycle,NodeRestriction,LimitRanger,ResourceQuota,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,PodNodeSelector,PodSecurity']
  45 kube_apiserver_admission_control_config_file: true
  46 # EventRateLimit plugin configuration
  47 kube_apiserver_admission_event_rate_limits:
  48   limit_1:
  49     type: Namespace
  50     qps: 50
  51     burst: 100
  52     cache_size: 2000
  53   limit_2:
  54     type: User
  55     qps: 50
  56     burst: 100
  57 kube_profiling: false
  58
  59 ## kube-controller-manager
  60 kube_controller_manager_bind_address: 127.0.0.1
  61 kube_controller_terminated_pod_gc_threshold: 50
  62 # AppArmor-based OS
  63 #kube_controller_feature_gates: ["RotateKubeletServerCertificate=true","AppArmor=true"]
  64 kube_controller_feature_gates: ["RotateKubeletServerCertificate=true"]
  65
  66 ## kube-scheduler
  67 kube_scheduler_bind_address: 127.0.0.1
  68 kube_kubeadm_scheduler_extra_args:
  69   profiling: false
  70 # AppArmor-based OS
  71 #kube_scheduler_feature_gates: ["AppArmor=true"]
  72
  73 ## etcd
  74 etcd_deployment_type: kubeadm
  75
  76 ## kubelet
  77 kubelet_authorization_mode_webhook: true
  78 kubelet_authentication_token_webhook: true
  79 kube_read_only_port: 0
  80 kubelet_rotate_server_certificates: true
  81 kubelet_protect_kernel_defaults: true
  82 kubelet_event_record_qps: 1
  83 kubelet_rotate_certificates: true
  84 kubelet_streaming_connection_idle_timeout: "5m"
  85 kubelet_make_iptables_util_chains: true
  86 kubelet_feature_gates: ["RotateKubeletServerCertificate=true","SeccompDefault=true"]
  87 kubelet_seccomp_default: true
  88
  89 # additional configurations
  90 kube_owner: root
  91 kube_cert_group: root
  92
  93 # create a default Pod Security Configuration and deny running of insecure pods
  94 # kube_system namespace is exempted by default
  95 kube_pod_security_use_default: true
  96 kube_pod_security_default_enforce: restricted
  97 ```
  98
  99 Let's take a deep look to the resultant **kubernetes** configuration:
 100
 101 * The `anonymous-auth` (on `kube-apiserver`) is set to `true` by default. This is fine, because it is considered safe if you enable `RBAC` for the `authorization-mode`.
 102 * The `enable-admission-plugins` has not the `PodSecurityPolicy` admission plugin. This because it is going to be definitely removed from **kubernetes** `v1.25`. For this reason we decided to set the newest `PodSecurity` (for more details, please take a look here: <https://kubernetes.io/docs/concepts/security/pod-security-admission/>). Then, we set the `EventRateLimit` plugin, providing additional configuration files (that are automatically created under the hood and mounted inside the `kube-apiserver` container) to make it work.
 103 * The `encryption-provider-config` provide encryption at rest. This means that the `kube-apiserver` encrypt data that is going to be stored before they reach `etcd`. So the data is completely unreadable from `etcd` (in case an attacker is able to exploit this).
 104 * The `rotateCertificates` in `KubeletConfiguration` is set to `true` along with `serverTLSBootstrap`. This could be used in alternative to `tlsCertFile` and `tlsPrivateKeyFile` parameters. Additionally it automatically generates certificates by itself, but you need to manually approve them or at least using an operator to do this (for more details, please take a look here: <https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/>).
 105 * If you are installing **kubernetes** in an AppArmor-based OS (eg. Debian/Ubuntu) you can enable the `AppArmor` feature gate uncommenting the lines with the comment `# AppArmor-based OS` on top.
 106
 107 Once you have the file properly filled, you can run the **Ansible** command to start the installation:
 108
 109 ```bash
 110 ansible-playbook -v cluster.yml \
 111         -i inventory.ini \
 112         -b --become-user=root \
 113         --private-key ~/.ssh/id_ecdsa \
 114         -e "@vars.yaml" \
 115         -e "@hardening.yaml"
 116 ```
 117
 118 **N.B.** The `vars.yaml` contains our general cluster information (SANs, load balancer, dns, etc..) and `hardening.yaml` is the file described above.
 119
 120 Once completed the cluster deployment, don't forget to approve the generated certificates (check them with `kubectl get csr`, approve with `kubectl certificate approve <csr_name>`). This action is necessary because the `secureTLSBootstrap` option and `RotateKubeletServerCertificate` feature gate for `kubelet` are enabled (CIS [4.2.11](https://www.tenable.com/audits/items/CIS_Kubernetes_v1.20_v1.0.0_Level_1_Worker.audit:05af3dfbca8e0c3fb3559c6c7de29191), [4.2.12](https://www.tenable.com/audits/items/CIS_Kubernetes_v1.20_v1.0.0_Level_1_Worker.audit:5351c76f8c5bff8f98c29a5200a35435)).