Stream Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 4/09: Flow control and load shedding ??? Vasiliki Kalavri | Boston University 2020 Keeping up with the producers queue: what if the queue grows larger than available memory? • block the producer (back-pressure, flow control) 2 ??? Vasiliki Kalavri | Boston University 2020 Load management approaches 3 ! Load shedder applications with strict latency constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers excess data for later processing, once input rates stabilize.

CurveBS I/O processing flow Before introducing IO processing flow, we first describe the overall architecture, data organization and topology structure of CURVE. CurveBS uses the central interface, such as read/write/aioread/aiowrite in data plane and open/create/rename/extend, etc in control plane. 2. Depending on libCurve, the application must be restarted every time the library is the NEBDServer, which affects services in few seconds. CurveBS IO processing flow Overall process The flow of user's read and write request to curve: 1. The user initiates a request (fd=1

Pepabo, Inc. 2019.07.25 GopherCon 2019 Optimization for number of goroutines using feedback control Principal engineer Yusuke MIYAKE @monochromegane Pepabo R&D Institute, GMO Pepabo, Inc. https://blog Issues to solve for the realization �27 1. Selection of performance metrics 2. Finding how to control rapidly and continuously Performance metrics Performance metrics �29 • Independent resource type number of goroutines based on the metrics • Continuously • Rapidly • Accurately • Feedback control meets these conditions. • Tracking a given set-point using errors from the set-point. • Applying

vec[index] = 0; // valid vec.clear(); vec[index] = 2; // invalid }Goals – Control Flow Validate based of control flow for: if/else, for loops, etc... void foo() { my_vector vec = { /* .. 1} });Control Flow Record and validate control flow enum class control_flow { if_, else_if, for_, return_, /*...*/ }; if (cond) { M::add<control_flow::if_>; } M::add<control_flow::end_scope>;Macro space */ }));Control Flow - Parsing Parse string of code into a control flow enum value constexpr control_flow parse_control(std::string_view str) { if (str == "{") return control_flow::start_scope;

Code Packages and Package Imports §11. Expressions, Statements and Simple Statements §12. Basic Control Flows §13. Goroutines, Deferred Function Calls and Panic/Recover Go Type System §14. Go Type programming languages support several kinds of conditional and loop control flows, we can think of them as special operations. The syntax of these control flows is close to human language so that the code written variables (both as 0). 17. var a, b int 18. // A for-loop control flow. 19. for i := 0; i < n; i++ { 20. // An if-else control flow. 21. if rand.Intn(MaxRnd) < MaxRnd/2 { 22. a

Code Packages and Package Imports §11. Expressions, Statements and Simple Statements §12. Basic Control Flows §13. Goroutines, Deferred Function Calls and Panic/Recover Go Type System §14. Go Type System languages support several kinds of conditional and loop control flows, we can think of them as special operations. The syntax of these control flows is close to human language so that the code written (both as 0). 17 | var a, b int 18 | // A for-loop control flow. 19 | for i := 0; i < n; i++ { 20 | // An if-else control flow. 21 | if rand.Intn(MaxRnd) < MaxRnd/2 { 22 |

Code Packages and Package Imports §11. Expressions, Statements and Simple Statements §12. Basic Control Flows §13. Goroutines, Deferred Function Calls and Panic/Recover Go Type System §14. Go Type System languages support several kinds of conditional and loop control flows, we can think of them as special operations. The syntax of these control flows is close to human language so that the code written (both as 0). 17| var a, b int 18| // A for-loop control flow. 19| for i := 0; i < n; i++ { 20| // An if-else control flow. 21| if rand.Intn(MaxRnd) < MaxRnd/2 { 22|

object is destroyed Stack-based (automatic) data objects • Is symmetry to define state based on control-flow (static lexical scoping) • Edge cases managed by the C++ Standard (Guaranteed!) “The compiler object is destroyed Stack-based (automatic) data objects • Is symmetry to define state based on control-flow (static lexical scoping) • Edge cases managed by the C++ Standard (Guaranteed!) “The compiler taketh away” Heap-based (dynamic) data objects • Is symmetry to define state independent of control-flow (static lexical scoping) • Edge cases managed by the developer “The developer giveth, and the

Connections Detecting Slow Consumers Avoiding Network Isolation Resource Manager Configuration Flow Control Guarantees of sends and commits Message Redelivery and Undelivered Messages Message Expiry making each file fit on a disk cylinder. Modern disk topologies are complex and we are not in control over which cylinder(s) the file is mapped onto so this is not an exact science. But by minimising the remote address. This is typically used in the Application Server or when running Embedded to control which address is used for outbound connections. If the local-address is not set then the connector