controller manager - Install API server - Config them correctly - Start them Installation - etcd - SSH - Install etcd - Config them correctly - Start them Installation kops, kubeup.sh, kube-AWS,... AWS runtime - Upgrade Kubelet Upgrade - master - SSH - Upgrade master components Upgrade - etcd - SSH - Upgrade etcd Upgrade kops AWS, GCP API node1 node2 node3 Rollback ??? AWS, GCP API node1 node2 ootkube How Bootkube Works etcd Kubelet Bootkube API Server Scheduler Controller Manager etcd Kubelet Bootkube API Server Scheduler Controller Manager etcd Kubelet Bootkube API Server

where secrets are used vs managed Encryption at different layers (or turtles) disks file system etcd Recommendation: Use two-layers of encryption, e.g., full-disk & application-layer … then tries also after a suspected incident Kubernetes secrets Kubernetes secrets ● Secrets are stored in etcd ○ base64 encoded ● A pod can access secrets via the filesystem, as an environment variable, or via via Kubernetes API call ● Operations with secrets are audit logged Master kube-apiserver etcd SECRET Kubernetes secrets: 1.7 EncryptionConfig ● Encrypt secrets with a locally managed key ● EncryptionConfig

4 k8s.gcr.io/kube-scheduler:v1.19.4 k8s.gcr.io/kube-proxy:v1.19.4 k8s.gcr.io/pause:3.2 k8s.gcr.io/etcd:3.4.13-0 k8s.gcr.io/coredns:1.7.0 #可以先下载以上7个镜像，传到每台k8s服务器上，再docker load导入；或者使 用内部registry仓库（内部registry镜像仓库里要有以上7个镜像） /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: cof-lee.com:5443/k8s kind: ClusterConfigura�on kubernetesVersion: .19.4 cof-lee.com:5443/k8s/kube-proxy:v1.19.4 cof-lee.com:5443/k8s/pause:3.2 cof-lee.com:5443/k8s/etcd:3.4.13-0 #可见镜像名已由默认的k8s.gcr.io/换成了配置文件里指定的docker镜像源 # kubeadm config images pull --image-repository="cof-lee

的架构特点 中心化架构 所有组件通过 apisever 交互 随着规模增大存储系统成为瓶颈 etcd 存在性能问题 apiserver etcd K8s 各组件 apiserver 元信息存储 etcd etcd 存在的问题 自研元信息存储 调优 etcd 参数 按照对象拆分 etcd 设计新的元信息存储 … 如何解决存储瓶颈？ KubeBrain 1. 大脑 2. version • 单 Key CAS  Watch • Kubernetes list-watch 的底层依赖 K8s 元信息存储的需求 (2) K8s 元信息存储的需求 (3) 所以 etcd 为目前 K8s 唯一支持的存储 KubeBrain 架构 Kine KubeBrain KubeBrain 架构 • 主从架构 • 主负责写和事件分发 • 从负责读 • 底层对接分布式强一致性存储 Range 分区 • 强一致性 • 支持多 key 事务 • 支持 CAS • 支持快照读 • 高性能 存储层 - 数据格式 etcd KubeBrain 能否使用类似的格式？ 1. 否 2. 底层存储引擎全局有序，有写热点那问题 Etcd 以 Revision 为 Key 内存 Btree 索引维护 key 和 revision 的映射关系 存储层 - 数据格式 KubeBrain

“OO” 1.API objects stores in etcd 2.Control loops (Sync Loop) to reconcile API objects Example kubelet SyncLoop kubelet SyncLoop proxy proxy 1 Pod created etcd scheduler api-server Example proxy proxy 2 Object added etcd scheduler api-server Example kubelet SyncLoop kubelet SyncLoop proxy proxy 3.1 New Pod detected 3.2 Bind Pod to a node etcd scheduler api-server Example SyncLoop kubelet SyncLoop proxy proxy 4.1 Detected bind operation 4.2 Start Pod on this machine etcd scheduler api-server Pattern 1: Controller • Control everything by Controller • Level driven

Kube-APIServer Webhook ETCD 数据构建 压测场景 压测环境 压测报告 压测平台 监控&大盘• APIServer & ETCD & Webhook Load balance ETCD Webhook Client Kubelets APIServer Http2 -> http1.1 Upgrade Etcd client v3.3.15 周期性重建连接 周期性重建连接 slb slb 直连 设置maxSurge• 客户端和服务端的同步机制 List & Watch优化 ETCD Cache Pod A V1 Pod A V2 Pod A V3 Reflector APIServer Watch Cache List & Watch Informer Reflector Store List & Watch• 网络抖动造成informer重新List APIServer Client List / Get ETCD rv=nil Page read Filter by condition• APIServer 缓存一致性读 • 索引支持动态新增 • Cache Ready Cache Read & Index APIServer Client list/get @t0 ETCD rv=nil 1. Get rv@t0 Cache

Stored in etcd • distributed Key-Value data store • How about their security? • Default K8s setup • etcd contents not encrypted (only base64 encoded) • > K8s 1.7+ • at-rest encryption for etcd (local Background: K8s Secrets • Encryption Keys stored on API Server • Secrets encrypted prior to storage in etcd • Secrets decrypted on API Server prior to use • Encryption keys stored in a remote KMS • Use Reduce / minimize remote KMS interactions w/o compromising security • Address security threats • etcd compromise • Host (KMS plugin) compromise Ø leak DEKs Ø leak KEKs [1] KubeCon NA 2019: "TEE-based

Pods that must be running Worker Node Worker Node Worker Node Kubernetes Master Node (Master & etcd nodes) API K K K App_Y.yaml ContainerImage1 Replicas: 1 ContainerImage2 Replicas: 2 https://youtu VM VM Kubernetes Trend Worker Node The Kubernetes Master Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-scheduler • Key/Value Store • Leader based clustering capacity • Affinity/Anti-Affinity Capable The Kubernetes Worker Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-Scheduler Worker Node CRI-containerd Kubeproxy

haproxy将流量转发至 apiserver 每个控制平面节点创建一个本 地etcd成员，该etcd成员仅与 kube-apiserver该节点通信 kubernetes cluster HA etcd cluster load balancer-vip apiserver controller-manager scheduler etcd master haproxy keeplived apiserver apiserver controller-manager scheduler etcd master haproxy keeplived apiserver controller-manager scheduler etcd master haproxy keeplived kube-proxy kubelet docker flannel kube-proxy kubelet

和令牌控制器。负责维护集群的状态，比如故障检测、自动扩展、滚动更新等。 Scheduler调度器（kube-scheduler）：负责资源调度（Pod调度）的进程，相当于“调度室”。按照预定的调度策略 将Pod调度到相应的机器上 etcd：集群的数据存储，他存储着集群中所有的资源对象。数据存储采用的是键值对存储。保存了整个集群的状态。 11 www.h3c.com Confidential 秘密 11 11 K8s基本概念和术语介绍（Node） com Confidential 秘密 13 13 K8s基本概念和术语介绍（Pod） Pod： pod分两种：普通pod和静态pod（static pod） 普通pod：一旦被创建，会被放到etcd中存储，随后被k8s master调度到某个具体的node上并进行绑定，随后该pod被 对应的node上的kubelet进程实例化成一组相关的docker容器并启动起来。默认情况下，当pod中的某个容器停止时， 停止时， K8s会自动检测到这个问题并重新启动这个pod（重启pod里面的所有容器），如果pod所在的node宕机，则会将这个 node上的所有pod重新调度到其他节点上。 静态pod：不存储在etcd中，而是存放在某个具体的node上的一个具体文件中，并只在此node上启动运行。 每个pod可以设置限额，目前可以设置CPU和内存，cpu的单位为core的数量， 是一个绝对值而不是相对值。k8