Kubernetes HA leader control plane services restarting -
while following kubernetes-the-hard-way , porting local ubuntu 16.04 vms, bringing ha control plane giving me problems. kube-apiserver , kube-controller-manager services elected leader keep failing , restarting control planes on other 2 non-leader masters come fine. behavior follows leader around cluster.
etcd looks happy:
member 7f44a7567a5e995 healthy: got healthy result https://10.1.15.117:2379 member 43d2258c438cbf4e healthy: got healthy result https://10.1.15.116:2379 member a83b22e9f907f471 healthy: got healthy result https://10.1.15.115:2379 cluster healthy
i've verified current leader by:
kubectl endpoints kube-controller-manager --namespace=kube-system -o yaml apiversion: v1 kind: endpoints metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderidentity":"df-dev-kube-test02","leasedurationseconds":15,"acquiretime":"2017-09-13t17:15:06z","renewtime":"2017-09-13t18:10:46z","leadertransitions":7}' creationtimestamp: 2017-09-13t14:59:57z name: kube-controller-manager namespace: kube-system resourceversion: "7814" selflink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 3599a959-9894-11e7-a34f-005056b24128 subsets: []
kube-apiserver logs: pastebin apiserver logs
kube-controller-manager logs: pastebin controller logs
considering various "storage" related errors in logs , since related leader elections, thought perhaps issue might've been etcd related. etcd kept reporting fine though. grins removed --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
from api-server.service , control pane happy.
i'm not sure if encryption breaking things or if automation of encryption hosed it.
Comments
Post a Comment