���@PingCAP About Me ● �������@PingCAP ● ����������� BigData Infra Team Lead ● ���������������SQL on Hadoop ��� �������� ● ����������������������������� ● ����������� Data Sink ������������������ �� SQL Server Events BI Tools Web Console ����� ● �� NoSQL ○ ����������������� ○ ������������� ● �� Hadoop ○ ����������� RDBMS �� NoSQL ○ �� Update��������� ● ���������� ○ ���������� ○ ������ ● ������������ Query ○ ������������ → ��� Join �� ○ SQL ������� → Data Science / Machine Learning ○ ������� → ��� Hadoop ������ ● ����� TiDB ����� TiSpark ● TiSpark ���� TiDB ����� Apache Spark ��� ● �� Apache Spark

做出决策，故需要将分散在各个系统的 数据汇聚在同⼀个系统并进⾏⼆次加⼯处理⽣成 T+0 或 T+1 的报表。传统常⻅的解决⽅案是采⽤ ETL + Hadoop 来完成，但 Hadoop 体系太复杂，运维、存储成本太⾼⽆法满⾜⽤⼾ 的需求。与 Hadoop 相⽐，TiDB 就简单得多，业务通过 ETL ⼯具或者 TiDB 的同步⼯具将数据同步到 TiDB，在 TiDB 中可通过 SQL 直接⽣成报表

OLTP specialized database or NoSQL ○ For historical data, use Hadoop / analytical database ● Offload data via the ETL process into your Hadoop cluster or analytical database ○ Per hour or even per day

Spanner 2013 F1 BigTable Map Reduce GFS Google 十年前基于内部分布式 处理框架发表的 三篇论文奠定了大数据分析 处理基石；开源社区 以此为基础打造了Hadoop F1 Data Flow Spanner Colossus Google 内部新一代分布式处理框架，于12/13年发表 相关论文，奠定下一代分布式 NewSQL的理论和工程 实践基石。PingCAP以此为基础打造了TiDB 我们的数据库能解决什么问题 - 1 ● 无限线性水平扩展（Scale Out） 无论多大的数据量，都可以轻松通过增加节点来解决，写入和读取时延固定（毫 秒级别），无需分库分表或者搭建复杂的 Hadoop 集群，完整的 MySQL 兼容接 口轻松处理高并发实时写入、实时查询和分析，极大的简化程序设计、应用维护 ，轻松应对大数据存储问题。 ● 高并发、高吞吐、完整的跨行事务支持、强一致性 通过简单

Spark。 对于 Standalone 模式且无需 Hadoop 支持，请选择 Spark 2.1.x 且带有 Hadoop 依赖的 Pre-build with Apache Hadoop 2.x 任意版本。如您有需要配合使用的 Hadoop 集群，请选择对应的 Hadoop 版本号。 您也可以选择从源代码自行构建以配合官方 Hadoop 2.6 之前的版本。请注意目前 TiSpark 仅支持 +-------------+ 2. | Count (1) | 3. +-------------+ 4. | 600000000 | 5. +-------------+ Q. 是独立部署还是和现有 Spark／Hadoop 集群共用资源？ A. 您可以利用现有 Spark 集群无需单独部署，但是如果现有集群繁忙，TiSpark 将无法达到理想速度。 Q. 是否可以和 TiKV 混合部署？ A. 如果 TiDB 以及 Data Federation（桥接其他数据源，最好能和社区同步，这个接进来可以比较好扩展 Usecase，如果 再做一个 InputFormat 适配就可以接 Hive 和 Presto 这些 Hadoop 上的数仓） On-Premise 版本集成部署 (K8s based) On-Premise 版本 Dashboard UI 集群备份和恢复工具（结合物理备份） 数据迁移工具（Wormhole

predicate push down via server side Filters ● MVCC What did they say ? “Nothing is hotter than SQL-on-Hadoop, and now SQL-on- HBase is fast approaching equal hotness status” Form HBaseCon 2015 We

traditional solution is to use ETL and Hadoop, but the Hadoop system is complicated, with high operations and maintenance cost and storage cost. Compared with Hadoop, TiDB is much simpler. You can replicate mode without Hadoop support, use Spark 2.4.x and any version of the pre-build binaries with Hadoop dependencies, for example spark-2.4.8-bin-hadoop2 �→ .7.tgz. If you need to use the Hadoop cluster, choose choose the corresponding Hadoop version. You can also choose to build from the source code to match the previous version of the official Hadoop 2.x. Example: wget https://archive.apache.org/dist/spark/spark-2

traditional solution is to use ETL and Hadoop, but the Hadoop system is complicated, with high operations and maintenance cost and storage cost. Compared with Hadoop, TiDB is much simpler. You can replicate mode without Hadoop support, use Spark 2.3.x and any version of Pre-build with Apache Hadoop 2.x with Hadoop dependencies. If you need to use the Hadoop cluster, choose the corresponding Hadoop version. You You can also choose to build from the source code to match the previous version of the official Hadoop 2.x. Suppose you already have a Spark binaries, and the current PATH is SPARKPATH, you can copy the

traditional solution is to use ETL and Hadoop, but the Hadoop system is complicated, with high operations and maintenance cost and storage cost. Compared with Hadoop, TiDB is much simpler. You can replicate mode without Hadoop support, use Spark 2.3.x and any version of Pre-build with Apache Hadoop 2.x with Hadoop dependencies. If you need to use the Hadoop cluster, choose the corresponding Hadoop version. You You can also choose to build from the source code to match the previous version of the official Hadoop 2.x. Suppose you already have a Spark binaries, and the current PATH is SPARKPATH, you can copy the

小红书是年轻人的生活方式平台, 用户可以通过短视频,图文等形式记录生活点滴, 分享生活方式。截至到 2019 年 10 月, 小红书月活跃用户数已经过亿, 并持续快速增长。 业务挑战 在数据报表场景, 原先采用 Hadoop 数仓对数据做预聚合, 然后放到 MySQL 里面做查询, 随着业务增长， 报表形式更加多样化, MySQL 的扩展性成为瓶颈。多节点 MySQL 的分库分表方案复杂度高, 运维非常困 难。在反欺诈分析场景