Kubernetes VM Solutions for Multi-Tenant Applications Guangxu Li, Senior Software Engineer, ZTE li.guangxu@zte.com.cn Container and VM Ecosystem Kubernetes Docker Swarm Marathon Nomad Container Container OpenStack Others Why We Run VM on Kubernetes? • Traditional Applications • No linux based Applications • Functions provided by host kernel are not satisfied • OpenStack is too complex • Unified

Traits/Scopes Scale: - 10,000 nodes/cluster - 100,000 apps/cluster - 1,000,000 containers/cluster Applications: - 100,000 deploys/day - 500~1000 replicas/app Function Job Container Virtual Machine Gateway

busi- ness, the focus has been to address this challenge with the development of Smart- Edge, a decentralized approach that goes beyond data collection to data process- ing and analytics, delivering insight

Developer • Hello World! • Containerize Applications • Micro Services Applications • Scalable Kubernetes Applications • Scalable Infrastructure for Applications Application Operating System Physical the App From the OS 2 technologies with 2 different objectives Allows you to run multiple applications on the same OS Allows you to run multiple OS on the same hardware Application Operating System Logs Messages VMware vRealize Log Insight Log analytics, aggregation, and search Virtual Applications vRealize Ops, Log Insight For Comprehensive Visibility 32 UI and API Backend Advanced Analytics

com/GoogleContainerTools/jib Build containers faster with Jib A container image builder for Java applications Our Team Cloud Tools for Java Appu Goundan @coollog @loosebazooka Qingyang “Q” Chen github Kubernetes Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes automated context such as a CI/CD pipeline to leverage the same workflow and tooling when moving applications to production. github.com/GoogleContainerTools/skaffold official website code Development Process

$11B Mobile Our fleet 15 3 US Data Centers POPs 200K+ Managed Vms 4K 100K Managed BMs Applications 4.5PB Managed Storage All of us know that... It’s not easy to manage fleet and infrastructure Replica Config LoadBalancer Rolling Update Quota Docker Image Volume RBAC Kubernetes models applications How about Kubernetes itself? How about the fleet Kubernetes running on? Onboard •Rack •NetworkDevice

StatefulSet/DaemonSet • StatefulSet is the workload API object used to manage stateful applications. Manages the deployment and scaling of a set of pods, and provides guarantees about the ordering a logical set of Pods and a policy by which to access them. • Service provides a way for applications to communicate with each other on K8s platform • In OpenWhisk, we deploy all the dependencies

OS CPU/Disk/Memory Kubernetes Docker Applications 20 2017 Lenovo Internal. All rights reserved. 监控与日志 • 日志平台：多种采集手段并行 OS Kubernetes Docker Applications K8S DaemonSet logspout K8S DaemonSet

Balancing loads Rolling updates Monitoring resources Accessing and ingesting logs Debugging applications Providing authentication and authorization 这提供了PaaS的简单性，并具有IaaS的灵活性，并促进了跨基础架构提供商的可移植性。 Kubernetes是⼀个怎样的平台？ Kubernetes： 不限制⽀持的应⽤类型。不规定应⽤框架（例如 Wildfly ），不限制⽀持的语⾔运⾏时（例如Java，Python， Ruby），不局限于 12-factor applications ，也不区分应⽤程序和服务 。 Kubernetes旨在⽀持各种各样的⼯作负 载，包括⽆状态、有状态以及数据处理⼯作负载。 如果应⽤程序可在容器中运⾏，那么它应该能够很好地在 Kubernetes上运⾏。 与所有其他Kubernetes配置⼀样，Deployment需要 apiVersion 、 kind 和 metadata 等字段。有关使⽤配置⽂件的 ⼀般信息，请参阅 deploying applications ，配置容器以及 using kubectl to manage resources ⽂档。 Deployment还需要⼀个 .spec section 。 Pod Template

secret? Credentials, configurations, API keys, and other small bits of information needed by applications at build or run time Why protect secrets? ● Attractive target ○ Controls access or use of sensitive Encryption Rotation Isolation Node authorizer K8s audit logging In etcd, not in applications aescbc, aesgcm, or secretbox Additional KMS logs aescbc KEK only, depending