����������� �������� ����������� �������� Retail spark search adds MR flink Sigma Fuxi Kubernetes YARN Node ���������� �� ��������� ������� ��� ����������� Co-located�40% Seperated �10% 30% ������ Kubernetes Focus on long running service. Driving current state towards desired state with control loops YARN Focus on scheduling jobs ���������� ���� Kubernetes Container centric – bottom up. Container is is the primitive. Other primitives such as replicaset, deployment are built around containers. YARN Application centric: top down. Scheduling sequence: Queue -> user -> application -> container request

TensorFlow on Yarn 李远策 2017年4月17日 内容大纲 Ø TensorFlow使用现状及痛点� Ø TensorFlow on Yarn设计� Ø TensorFlow on Yarn技术细节揭秘� Ø 深度学习平台演进及SparkFlow介绍� 背景 坐标：360-系统部-⼤数据团队� 专业：Yarn、Spark、MR、HDFS …� 集群资源的管理（目前支持CPU、内存，需要扩展GPU 资源管理）� • 作业的统⼀管理、状态跟踪� • 资源组（Schedule Pool）的划分� • 作业进程的资源隔离� Yarn能解决什么问题：� TensorFlow on Yarn设计 • 同时支持单机和分布式TensorFlow程序� • 支持GPU资源管理和调度� • 不再需要⼿动配置CluserSpec信息，仅需要设置work 和ps的数量� TensorFlow on Yarn设计 • 支持GPU亲和性调度（提⾼通信效率）� • Web的⽅式查看作业的运⾏状况和作业日志� • 在线查看Tensorboard� • HistoryServer支持查看结束作业的日志和状态信息� • 控制已有的TensorFlow作业的迁移成本（最多改三⾏ 代码）� 扩展目标：� TensorFlow on Yarn设计 tensorflow-submit

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High engine deploy mode, The local mode, the engine operates on the same n YARN mode, the engine runs within the Appl container of YARN. kyuubi.engine.hive.event.loggers JSON A comma-separated list of engine timeout. kyuubi.engine.yarn.app.name The YARN app name when the engine deploy kyuubi.engine.yarn.cores 1 kyuubi engine container core number when th YARN. kyuubi.engine.yarn.java.options

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High local jars to be shipped with the job to the cluster. For example, SQL UDF jars. Only effective in yarn application mode. strin g 1.8.0 kyuubi.engine.flin k.extra.classpath The extra classpath effective in yarn session mode. strin g 1.6.0 Key Default Meaning Type Since kyuubi.engine.flin k.java.options The extra Java options for the Flink SQL engine. Only effective in yarn session

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High local jars to be shipped with the job to the cluster. For example, SQL UDF jars. Only effective in yarn application mode. strin g 1.8.0 kyuubi.engine.flin k.extra.classpath The extra classpath effective in yarn session mode. strin g 1.6.0 Key Default Meaning Type Since kyuubi.engine.flin k.java.options The extra Java options for the Flink SQL engine. Only effective in yarn session

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High local jars to be shipped with the job to the cluster. For example, SQL UDF jars. Only effective in yarn application mode. strin g 1.8.0 kyuubi.engine.flin k.extra.classpath The extra classpath effective in yarn session mode. strin g 1.6.0 Key Default Meaning Type Since kyuubi.engine.flin k.java.options The extra Java options for the Flink SQL engine. Only effective in yarn session

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High initialize.sql. kyuubi.engine.flink.java.options The extra Java options for the Flink SQL eng yarn session mode. kyuubi.engine.flink.memory 1g The heap memory for the Flink SQL engine. session mode session if its corresponding user is configured timeout. kyuubi.engine.yarn.submit.timeout PT30S The engine submit timeout for YARN applica Event Key Default Meaning Type Since kyuubi.event.async.pool

schedule distributed tasks at runtime. They can process data on single-node machines or clusters, such as YARN and Kubernetes. Besides, the Kyuubi server also supports running on bare metal or in a docker. High User Defaults Engines Deploy Kyuubi engines on Yarn Deploy Kyuubi Spark Engine on Yarn Deploy Kyuubi Flink Engine on Yarn Deploy Kyuubi Hive Engine on Yarn Deploy Kyuubi engines on Kubernetes Requirements visit different Hive metastore server instance. Similarly, this works for other services like HDFS, YARN too. Limitation: As most Hive configurations are final and unmodifiable in Spark at runtime, this

oop-yarn/hadoop-yarn- site/FairScheduler.html#Queue_Access_Control_Lists], from cluster managers, e.g. Apache Hadoop YARN [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html] Run Anywhere Kyuubi can submit Spark applications to all supported cluster managers, including YARN, Mesos, Kubernetes, Standalone, and local. The SPA policy also make it possible for you to launch you can run Spark/Flink/Trino SQL engines created by the Kyuubi on any cluster manager, including YARN, Kubernetes, Mesos, e.t.c… Or, you can manipulate data from different data sources with the Spark

oop-yarn/hadoop-yarn- site/FairScheduler.html#Queue_Access_Control_Lists], from cluster managers, e.g. Apache Hadoop YARN [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html] Run Anywhere Kyuubi can submit Spark applications to all supported cluster managers, including YARN, Mesos, Kubernetes, Standalone, and local. The SPA policy also make it possible for you to launch you can run Spark/Flink/Trino SQL engines created by the Kyuubi on any cluster manager, including YARN, Kubernetes, Mesos, e.t.c… Or, you can manipulate data from different data sources with the Spark