Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems Outline ▶ Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing • Record-at-a-Time vs. declarative APIs 6 / 79 Spark Streaming RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream) 7 / 79 Spark Streaming ▶ Run a streaming computation as a series of very small, deterministic batch jobs. • Chops

Spark 简介以及与 Hadoop 的对比 1 Spark 简介 1.1 Spark 概述 Spark 是 UC Berkeley AMP lab 所开源的类 Hadoop MapReduce 的通用的并行计算框 架，Spark 基于 map reduce 算法实现的分布式计算，拥有 Hadoop MapReduce 所具有的 优点；但不同于 MapReduce 的是 Job 中间输出和结果可以保存在内存中，从而不再需要读 写 HDFS，因此 Spark 能更好地适用于数据挖掘与机器学习等需要迭代的 map reduce 的算 法。 1.2 Spark 核心概念 1.2.1 弹性分布数据集（RDD） RDD 是 Spark 的最基本抽象,是对分布式内存的抽象使用，实现了以操作本地集合的方式 来操作分布式数据集的抽象实现。RDD 是 Spark 最核心的东西，它表示已被分区，不可变的 的操作不是马上执行，Spark 在遇 到 Transformations 操作时只会记录需要这样的操作，并不会去执行，需要等到有 Actions 操作的时候才会真正启动计算过程进行计算。 2. 操作(Actions) (如：count, collect, save 等)，Actions 操作会返回结果或把 RDD 数据写 到存储系统中。Actions 是触发 Spark 启动计算的动因。

MathWorks, Inc. MATLAB与Spark/Hadoop相集成：实现大 数据的处理和价值挖 马文辉 2 内容 ▪ 大数据及其带来的挑战 ▪ MATLAB大数据处理 ➢ tall数组 ➢ 并行与分布式计算 ▪ MATLAB与Spark/Hadoop集成 ➢ MATLAB访问HDFS(Hadoop分布式文件系统） ➢ 在Spark/Hadoop集群上运行MATLAB代码 ▪ MapReduce (MDCS/PCT) ▪ MATLAB API for Spark API ▪ Tall Arrays ▪ 计算 ▪ Desktop (Multicore, GPU) ▪ Clusters ▪ Cloud Computing (MDCS on EC2) ▪ Hadoop ▪ Spark ▪ 内存与数据访问 ▪ 64-bit processors ▪ Memory Parallel Computing Toolbox) ▪ MATLAB集群之上的分布式计算 (MDCS, MATLAB Distributed Computing Server) 9 MATLAB与Spark/Hadoop集成 MDCS 10 Hadoop Hadoop是跨计算机集群的分布式大数据处理平台，由两部分组成： • YARN (Yet Another Resource Negotiator)

Apache OFBiz® The Apache OFBiz Project Version Trunk Table of Contents 1. System requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License License Apache 2.0 Apache 2.0 [badge] If you are reading this file in AsciiDoc format you may want to see it at HTML or PDF format Welcome to Apache OFBiz! A powerful top level Apache software project reading section. 3 3. Security If you find a security issue, please report it to: security @ ofbiz.apache.org. Once proper mitigations to the security issues are complete the OFBiz team will disclose this

解读Apache 解读Apache 演讲者 Craig Russell Justin Mclean 姜宁 本演讲包含了 Bertrand Delacretaz, Roman Shaposhnik 以及其 它ASF贡献者的⼯作 Craig Russell介绍 Craig Russell介绍 软件架构师 Object Data Management Group Java Data Objects (MySQL) Apache Committer - from 2005 Apache Member - from 2007 Apache Secretary - from 2010-2019 Apache 孵化器管理委员成员 Apache 董事会主席 Apache 软件基⾦会: Apache 软件基⾦会: 世界上最⼤的开源基⾦会 Apache基⾦会的使命 Apache基⾦会的使命 Apache软基⾦⾦会 Apache软基⾦⾦会 (ASF) 是⼀个根据美国法案 501(c)(3)注册公益性组织。 ASF的使命 是通过向加⼊ASF的志同道合的软件项⽬社区提供服务，为公众提供开源软件。 ASF提供了⼀个独⽴于任何公司影响⼒的中⽴空间，保证其中的 项⽬可以在商业友好Apache许可证2.0下繁荣发展， 为公众利益 创建开源软件。 公开成⽴于1999年的公益组织 公开成⽴于1999年的公益组织 使命:

Apache Explained Presented by Roman Shaposhnik Craig Russell Justin Mclean Including original work of Bertrand Delacretaz and other amazing ASF contributors ➔ Engineer and Entrepreneur ◆ Founder Pivotal Inc. ◆ Ex- Sun microsystems, Huawei, Yahoo!, Cloudera ➔ Apache Committer - from 2010 ➔ Apache Member - from 2014 ➔ Apache Board Member and VP of Legal Affairs - from 2018 ➔ Member, Incubator Management Committee, Bigdata Who is Roman Shaposhnik? Apache Software Foundation: The World’s Largest Open Source Foundation The ASF’s Mission The Apache Software Foundation (ASF) is a US 501(c)(3) charitable

Apache Explained Apache Explained Presented by Craig Russell Justin Mclean Willem Jiang Including original work of Bertrand Delacretaz, Roman Shaposhnik and other amazing ASF contributors Who is Craig Jones (MySQL) Apache Committer - from 2005 Apache Member - from 2007 Apache Secretary - 2010 - 2019 Member, Incubator Project Management Committee Chairman, Apache Board of Directors Apache Software Foundation: Foundation: Apache Software Foundation: The World’s Largest Open The World’s Largest Open Source Foundation Source Foundation The ASF’s Mission The ASF’s Mission The Apache Software Foundation (ASF)

Apache OFBiz® The Apache OFBiz Project Version Trunk Table of Contents 1. System requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License License Apache 2.0 Apache 2.0 [badge] If you are reading this file in AsciiDoc format you may want to see it at HTML or PDF format Welcome to Apache OFBiz! A powerful top level Apache software project reading section. 3 3. Security If you find a security issue, please report it to: security @ ofbiz.apache.org. Once proper mitigations to the security issues are complete the OFBiz team will disclose this

使⽤Apache SkyWalking APM 监控 Apache ServiceComb 吴晟 Sheng Wu Huawei DevCloud http://skywalking.io Twitter @AsfSkyWalking 个⼈介绍 GitHub： https://github.com/wu-sheng Personal Homepage： https://wu-sheng https://wu-sheng.github.io/me/ Apache SkyWalking creator, PPMC member and Committer 2018 Microsoft MVP OpenTracing OTSC & OTAIB member W3C Trace Context specification member 2017 GSoC(Google Summer of GitHub: https://github.com/apache/incubator-skywalking 安装：https://github.com/apache/incubator- skywalking/blob/master/docs/cn/Quick-start-CN.md ⽀持的插件列表： https://github.com/apache/ incubator-skywalking/

GSoC 2020 Apache Proposal Apache RocketMQ Scaler for KEDA Application Name : Hien Nguyen University : Haaga-Helia University of Applied Sciences - Bachelor of Information Technology - (Location: Test, DevOps, Distributed system, Cloud(AWS, Azure) , Golang, Maven, Docker, Kubernetes GSoC - Apache RocketMQ Scaler for KEDA proposal Context KEDA allows for fine-grained autoscaling (including MySQL, RocketMQ,etc; multiple workloads type(jobs,deployments,trigger) - KEDA does not support Apache RocketMQ now. So we need to create PR in KEDA repo for new support for RocketMQ - KEDA has event-driven