Sync Clickhouse with MySQL/MongoDB Company: Xiaoxin Tech. Industry: Education Team: Big Data Leader: wangchao@xiaoheiban.cn About 100 billion data this year till now 30 million users We use Clickhouse Possible Solutions 2. MySQL Engine Not suitable for big tables Not suitable for MongoDB Possible Solutions 3. Reinit whole table every day…… Possible Solutions 4. CollapsingMergeTree ● Merge sharding tables ● SolidKey Typical Provider Config { Type: mysql, // mysql, mongodb, redis Listen: binlog, // binlog, kafka DataSource: user:pass@tcp(example.com:3306)/user

Sync Clickhouse with MySQL/MongoDB Company: Xiaoxin Tech. Industry: Education Team: Big Data Leader: wangchao@xiaoheiban.cn About 100 billion data this year till now 30 million users We use Clickhouse Possible Solutions 2. MySQL Engine Not suitable for big tables Not suitable for MongoDB Possible Solutions 3. Reinit whole table every day…… Possible Solutions 4. CollapsingMergeTree Listen ● Merge sharding tables ● SolidKey Typical Provider Config { Type: mysql, // mysql, mongodb, redis Listen: binlog, // binlog, kafka DataSource: user:pass@tcp(example.com:3306)/user, Table:

Go Language MongoDB Mgo/Beego If we had a system that could do the following, we would have something: hHp://docs.mongodb.org/manual/core/da org/manual/core/data-­‐modeling-­‐introduc*on/ WHY MONGODB - DYNAMIC FEED MANAGEMENT Data is stored in “Collec*ons” as individual documents. Rela*onships can be created by data. MongoDB’s schemaless database provides great flexibility. hHp://docs.mongodb.org/manual/core/data-­‐modeling-­‐introduc*on/ WHY MONGODB - DYNAMIC FEED MANAGEMENT

Members 35 Chapter 4 - Connection 37 About Connection 37 General Settings 38 RDBMS 38 MongoDB 40 Redis 41 Advanced Settings 42 Databases / Attached Databases Settings 45 SSL Settings SQLite 73 Databases 73 Tables 74 Views 74 Other Objects 75 Maintain Objects 75 MongoDB 76 Databases 76 Collections 76 3 Views 77 Functions 77 Indexes 78 MapReduce Records 90 Manipulate Raw Data 90 Format Data View 91 View Field Comments 92 MongoDB 93 MongoDB Data Viewer 93 Use Navigation Bar 93 Grid View 94 Tree View 100 JSON View 102

Members 34 Chapter 4 - Connection 36 About Connection 36 General Settings 37 RDBMS 37 MongoDB 39 Redis 40 Advanced Settings 41 Databases Settings 44 SSL Settings 45 SSH Settings SQLite 72 Databases 72 Tables 72 Views 73 Other Objects 73 Maintain Objects 73 MongoDB 74 Databases 74 Collections 74 3 Views 75 Functions 75 Indexes 76 MapReduce Records 88 Manipulate Raw Data 89 Format Data View 89 View Field Comments 90 MongoDB 90 MongoDB Data Viewer 90 Use Navigation Bar 91 Grid View 92 Tree View 97 JSON View 99

Members 34 Chapter 4 - Connection 36 About Connection 36 General Settings 37 RDBMS 37 MongoDB 39 Redis 40 Advanced Settings 41 Databases / Attached Databases Settings 44 SSL Settings SQLite 72 Databases 72 Tables 73 Views 73 Other Objects 74 Maintain Objects 74 MongoDB 74 Databases 74 Collections 75 3 Views 75 Functions 75 Indexes 76 MapReduce Records 88 Manipulate Raw Data 89 Format Data View 89 View Field Comments 91 MongoDB 91 MongoDB Data Viewer 91 Use Navigation Bar 92 Grid View 93 Tree View 98 JSON View 100

Working with NoSQL Technologies Redis Connecting to Redis MongoDB Connecting to a MongoDB Database MongoTemplate Spring Data MongoDB Repositories Embedded Mongo Neo4j Connecting to a Neo4j Database Auto-configured Data JPA Tests Auto-configured JDBC Tests Auto-configured jOOQ Tests Auto-configured Data MongoDB Tests Auto-configured Data Neo4j Tests Auto-configured Data Redis Tests Auto-configured Data LDAP spring-boot-starter- data-mongodb Starter for using MongoDB document-oriented database and Spring Data MongoDB Pom spring-boot-starter- data-mongodb-reactive Starter for using MongoDB document-oriented database

Working with NoSQL technologies Redis Connecting to Redis MongoDB Connecting to a MongoDB database MongoTemplate Spring Data MongoDB repositories Embedded Mongo Neo4j Connecting to a Neo4j database Auto-configured Data JPA tests Auto-configured JDBC tests Auto-configured jOOQ tests Auto-configured Data MongoDB tests Auto-configured Data Neo4j tests Auto-configured Data Redis tests Auto-configured Data LDAP spring-boot-starter- data-mongodb Starter for using MongoDB document-oriented database and Spring Data MongoDB Pom spring-boot-starter- data-mongodb-reactive Starter for using MongoDB document-oriented database

Working with NoSQL technologies Redis Connecting to Redis MongoDB Connecting to a MongoDB database MongoTemplate Spring Data MongoDB repositories Embedded Mongo Neo4j Connecting to a Neo4j database Auto-configured Data JPA tests Auto-configured JDBC tests Auto-configured jOOQ tests Auto-configured Data MongoDB tests Auto-configured Data Neo4j tests Auto-configured Data Redis tests Auto-configured Data LDAP spring-boot-starter- data-mongodb Starter for using MongoDB document-oriented database and Spring Data MongoDB Pom spring-boot-starter- data-mongodb-reactive Starter for using MongoDB document-oriented database

Working with NoSQL Technologies Redis Connecting to Redis MongoDB Connecting to a MongoDB Database MongoTemplate Spring Data MongoDB Repositories Embedded Mongo Neo4j Connecting to a Neo4j Database Auto-configured Data JPA Tests Auto-configured JDBC Tests Auto-configured jOOQ Tests Auto-configured Data MongoDB Tests Auto-configured Data Neo4j Tests Auto-configured Data Redis Tests Auto-configured Data LDAP spring-boot-starter- data-mongodb Starter for using MongoDB document-oriented database and Spring Data MongoDB Pom spring-boot-starter- data-mongodb-reactive Starter for using MongoDB document-oriented database