K8S: Unable To Connect To The Server: X509: Certificate Has Expired Or Is Not Yet Valid

cloud

K8S: Unable To Connect To The Server: X509: Certificate Has Expired Or Is Not Yet Valid

Problem

When trying to connect to a kubernetes cluster using kubectl, the following error is returned or something very similar:

1
2
$ k get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-03-31T07:09:11Z is after 2023-03-24T19:43:42Z

Solution

The solution is to regenerate the certificates on the master node.

Regenerate the certificates

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Check the expiration dates

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0331 07:15:07.313285 2799806 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 30, 2024 07:10 UTC   364d            ca                      no
apiserver                  Mar 30, 2024 07:10 UTC   364d            ca                      no
apiserver-etcd-client      Mar 30, 2024 07:10 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Mar 30, 2024 07:10 UTC   364d            ca                      no
controller-manager.conf    Mar 30, 2024 07:10 UTC   364d            ca                      no
etcd-healthcheck-client    Mar 30, 2024 07:10 UTC   364d            etcd-ca                 no
etcd-peer                  Mar 30, 2024 07:10 UTC   364d            etcd-ca                 no
etcd-server                Mar 30, 2024 07:10 UTC   364d            etcd-ca                 no
front-proxy-client         Mar 30, 2024 07:10 UTC   364d            front-proxy-ca          no
scheduler.conf             Mar 30, 2024 07:10 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 21, 2032 19:43 UTC   8y              no
etcd-ca                 Mar 21, 2032 19:43 UTC   8y              no
front-proxy-ca          Mar 21, 2032 19:43 UTC   8y              no

Find the config file

1
2
$ find / -name admin.conf 2>/dev/null
/etc/kubernetes/admin.conf

Check if the new certifcates are valid

1
2
3
4
5
$ sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
NAME      STATUS   ROLES                  AGE    VERSION
master    Ready    control-plane,master   371d   v1.23.5
worker1   Ready    <none>                 371d   v1.23.5
worker2   Ready    <none>                 371d   v1.23.5

Make the config your own

1
$ sudo cp /etc/kubernetes/admin.conf ~/.kube/config

check if the new config works

1
2
3
4
5
$ kubectl get nodes
NAME      STATUS   ROLES                  AGE    VERSION
master    Ready    control-plane,master   371d   v1.23.5
worker1   Ready    <none>                 371d   v1.23.5
worker2   Ready    <none>                 371d   v1.23.5

Conclusion

With a few simple commands, the certificates can be renewed and the cluster can be used again.

Note that I only tested this on my own cluster, so I can’t guarantee that this will work for you or if you have the rights to do this yourself. If you have any questions, please let me know in the comments.

References