FLINK_HOME instead of the JAR packages under PyFlink installation package. You can refer to PyFlink documentation for more details. 1.3.1.2 O2: Java gateway process exited before sending its port number The following command: ls -lh /Users/duanchen/miniconda3/lib/python3.7/site-packages/pyflink The structure would be as following: total 144 -rw-r--r-- 1 duanchen staff 1.3K Oct 19 16:01 README.txt -rw-r--r-- connector JAR in PyFlink jobs, you can refer to the dependency management page of official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An

FLINK_HOME instead of the JAR packages under PyFlink installation package. You can refer to PyFlink documentation for more details. 1.3.1.2 O2: Java gateway process exited before sending its port number The following command: ls -lh /Users/duanchen/miniconda3/lib/python3.7/site-packages/pyflink The structure would be as following: total 144 -rw-r--r-- 1 duanchen staff 1.3K Oct 19 16:01 README.txt -rw-r--r-- connector JAR in PyFlink jobs, you can refer to the dependency management page of official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An

University 2020 Filtering streams 21 ??? Vasiliki Kalavri | Boston University 2020 22 What data structure would you use to: • Filter out all emails that are sent from a suspected spam address? • Filter upstream backup? The membership problem ??? Vasiliki Kalavri | Boston University 2020 22 What data structure would you use to: • Filter out all emails that are sent from a suspected spam address? • Filter Kalavri | Boston University 2020 23 • Introduced by Burton Bloom in 1970. • A probabilistic data structure for representing a (possibly growing) dataset of elements that supports: • adding an element

Filter ??? Vasiliki Kalavri | Boston University 2020 20 • A space-efficient probabilistic data structure that can be used to estimate frequencies and heavy hitters in data streams • It was introduced = 10−6 The recommended number of counters is . m = 2.71828 10−6 ≈ 2,718,280 The sketch data structure requires a counter array of size 5 * 2,718,280. Space requirements ??? Vasiliki Kalavri | Boston = 10−6 The recommended number of counters is . m = 2.71828 10−6 ≈ 2,718,280 The sketch data structure requires a counter array of size 5 * 2,718,280. Considering 32-bit counters, the count-min sketch

message is processed only once, by a single consumer • Event retrieval is not defined by content / structure but its order • FIFO, priority producer consumer queue 6 Message brokers Message broker: multiple consumers can retrieve the same message - many-to-many communication - message content / structure matters for delivery 8 MB architecture advantages • Multiple producers/consumers as concurrent

2020 Graph streams Graph streams model interactions as events that update an underlying graph structure 5 Edge events: A purchase, a movie rating, a like on an online post, a bitcoin transaction Boston University 2020 Streaming Connected Components • State: a disjoint set (union-find) data structure for the components • it stores a set of elements partitioned in disjoint subsets • Single-pass

University 2020 Stream denotation An abstract interpretation of the stream as a mathematical structure, e.g. a sequence of (finite) relation states over a common schema R: [r1(R), r2(R), ..., ],

file:///home/user/wordcount_out 19 Flink commands Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://flink.apache.org/ • Community • https://flink.apache.org/community.html#mailing-lists losing any records committed to the log. Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://kafka.apache.org/ • Community • https://kafka.apache.org/contact • Conference

additional dependencies on the classpath. Please check out the Flink system resource metrics documentation9 for additional guidance and details. System resource monitoring in Flink can be very helpful metrics and monitoring capabilities of Apache Flink. I highly recommend going over Flink’s metrics documentation10 for a full reference of Flink’s metrics system.

pre-processing, deployment, and testing): (40%) • code deliverables must be accompanied by documentation 8 Vasiliki Kalavri | Boston University 2020 Schedule 9 vasia.github.io/dspa20/ lectures