升级到 1.35

kubelet 1.35 的启动参数有以下改动：

• --container-runtime-endpoint: 已弃用，改到了配置文件中

• --pod-infra-container-image: 不再支持

所以需要修改以下配置（适用于 kubeadm 安装的集群）：

1. 修改 /var/lib/kubelet/kubeadm-flags.env

   -KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.10"
   +KUBELET_KUBEADM_ARGS=""
   

2. 修改 /var/lib/kubelet/config.yaml

    apiVersion: kubelet.config.k8s.io/v1beta1
   +featureGates:
   +  GenericWorkload: true
    authentication:
      anonymous:
        enabled: false
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
   -containerRuntimeEndpoint: ""
   +containerRuntimeEndpoint: "unix:///var/run/containerd/containerd.sock"
    cpuManagerReconcilePeriod: 0s
    crashLoopBackOff: {}
    evictionPressureTransitionPeriod: 0s
   

然后更换各个组件的映像。

1. kube-apiserver, 修改 /etc/kubernetes/manifests/kube-apiserver.yaml:

        - --service-cluster-ip-range=10.96.0.0/12
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
   -    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.34.0
   +    - --feature-gates=GenericWorkload=true
   +    - --runtime-config=scheduling.k8s.io/v1alpha1=true
   +    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.35.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
   

2. kube-scheduler, 修改 /etc/kubernetes/manifests/kube-scheduler.yaml:

        - --bind-address=127.0.0.1
        - --kubeconfig=/etc/kubernetes/scheduler.conf
        - --leader-elect=true
   -    image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.34.0
   +    - --feature-gates=GenericWorkload=true
   +    image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.35.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
   

3. kube-controller-manager, 修改 /etc/kubernetes/manifests/kube-controller-manager.yaml:

        - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
        - --service-cluster-ip-range=10.96.0.0/12
        - --use-service-account-credentials=true
   -    image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.34.0
   +    - --feature-gates=GenericWorkload=true
   +    image: registry.aliyuncs.com/google_containers/kube-controller-manager:v1.35.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
   

4. kube-proxy, 运行命令：

   $ kubectl -n kube-system set image daemonset/kube-proxy kube-proxy=registry.aliyuncs.com/google_containers/kube-proxy:v1.35.0
   daemonset.apps/kube-proxy image updated
   

以上配置同时启用了特性门 GenericWorkload.

重启 kubelet 服务成功以后：

$ kubectl get no
NAME   STATUS   ROLES           AGE    VERSION
las0   Ready    control-plane   242d   v1.35.0
las1   Ready    <none>          242d   v1.35.0
las2   Ready    <none>          242d   v1.35.0
las3   Ready    <none>          238d   v1.35.0

查询 kubelet 特性门是否启用：

$ kubectl get --raw "/api/v1/nodes/las1/proxy/configz" | jq '.kubeletconfig.featureGates'
{
  "GenericWorkload": true
}