VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Michael Gasch KubeCon North America December 13, 2018 2 Open Source Community Relations options, for both control plane and worker nodes. 2 levels of scheduling and resource management are active. Currently no automatic scheduling integration occurs, that is, Kubernetes is not aware of the distribution running on the vSphere stack. Agenda 4 Kubernetes default scheduling How it works Utilizing Zones to improve scheduling Using vSphere tags to define regions and zones – add cloud provider

VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Hui Luo VMware Cloud Native Applications Business Unit November 12, 2018 2 Open options, for both control plane and worker nodes. 2 levels of scheduling and resource management are active. ​Currently no automatic scheduling integration occurs, that is, Kubernetes is not aware of the distribution running on the vSphere stack. Agenda 4 ​Kubernetes default scheduling ​How it works ​Utilizing Zones to improve scheduling ​Using vSphere tags to define regions and zones – add cloud provider

optimizations • plan translation alternatives • Runtime optimizations • load management, scheduling, state management • Optimization semantics, correctness, profitability Topics covered in this There may exist several ways to execute a computation • query plans, e.g. order of operators • scheduling and placement decisions • different algorithms, e.g. hash-based vs. broadcast join • What does data structures • sorting vs hashing • indexing, pre-fetching • minimize disk access • scheduling Objectives • optimize resource utilization or minimize resources • decrease latency, increase

缺少作业概念、缺少完善的生命周期的管理 • 缺少任务依赖、作业依赖支持 调度策略局限 • 不支持Gang-scheduling、Fair-share scheduling • 不支持多场景的Resource reservation，backfill • 不支持CPU/IO topology based scheduling 领域框架支持不足 • 1:1的operator部署运维复杂 • 不同框架对作业管理、并行计算等要求不通 金融投资公司，业务场景主要为策略研究开发、AI 训练与推理、 大数据ETL和离线批处理任务 客户诉求： • 要求调度系统提供公平机制，满足公司内多团队资源共享，保 证各自业务的SLA • 要求系统提供Gang-scheduling解决基本死锁问题 • 要求调度系统统一支持AI、大数据、Batch Job 解决方案： • Volcano 统一支持AI、数据ETL和离线Batch job • Volcano提供的 job2 T3 小作业 job3 开始运行 提交小作业 job4 场景：大作业与小作业共存时，存在饿死问题 Small job3 Small job4 丰富的调度算法 • Gang-Scheduling • Job priority • Job queue • Job order • Preemption • backfill • Job Fair-share • Namespace

the form "-Dx=y". # (Default: none). # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT Official Online Document: Dynamic Resource Allocation [https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation] 2. Spark Official Online Document: Dynamic Resource Allocation small partitions or tasks. Spark tasks will have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead. [2] From Databricks Blog Combining small partitions saves resources

the form "-Dx=y". # (Default: none). # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT Official Online Document: Dynamic Resource Allocation [https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation] 2. Spark Official Online Document: Dynamic Resource Allocation small partitions or tasks. Spark tasks will have worse I/O throughput and tend to suffer more from scheduling overhead and task setup overhead. [2] From Databricks Blog Combining small partitions saves resources

batch system (Volcano) development • IBM spectrum computing - Cluster resource and workload scheduling platform development l Gaps for Spark • Agenda l Why Spark on Kubernetes l Volcano solution … Ø Scheduler p Job preemption p Fair-share scheduling p Queue scheduling p Resource reservation p Binpack p Task topology p Zone aware scheduling p … Volcano: A Kubernetes native batch system pod executor pod executor pod executor pod apiVersion：v1 kind: Pod metadata: annotations: scheduling.k8s.io/group-name: job-1574739729783- podgroup volcano.sh/task-spec: spark-driver createTimestamp:

in the form "-Dx=Y". # (Default: none) # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn- site/CapacityScheduler.html], of resource scheduling management services, such as YARN and K8s. At application layer, we’d be better to acquire and Contributors Of Resource Waste The time to wait for the resource to be allocated, such as the scheduling delay, the start/stop cost. A longer time-to-live(TTL) for allocated resources can significantly

in the form "-Dx=Y". # (Default: none) # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn- site/CapacityScheduler.html], of resource scheduling management services, such as YARN and K8s. At application layer, we’d be better to acquire and Contributors Of Resource Waste The time to wait for the resource to be allocated, such as the scheduling delay, the start/stop cost. A longer time-to-live(TTL) for allocated resources can significantly

in the form "-Dx=Y". # (Default: none) # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn- site/CapacityScheduler.html], of resource scheduling management services, such as YARN and K8s. At application layer, we’d be better to acquire and Contributors Of Resource Waste The time to wait for the resource to be allocated, such as the scheduling delay, the start/stop cost. A longer time-to-live(TTL) for allocated resources can significantly