GPU Tile 1 Tile 0 Xe LinkProject Goals - Offer high-level, standard C++ distributed data structures - Support distributed algorithms - Achieve high performance for both multi-GPU, NUMA, and multi-node reduce(par_unseq, z, 0, std::plus()); }Outline - Background (Ranges, Parallelism, Distributed Data Structures) - Distributed Ranges (Concepts) - Implementation (Algorithms and views) - Complex sparse matrices) - Lessons learnedOutline - Background (Ranges, Parallelism, Distributed Data Structures) - Distributed Ranges (Concepts) - Implementation (Algorithms and views) - Complex

program for a supercomputer? Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer supercomputer? Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer? Introduce Introduce PGAS Model, RDMA Building Remote Pointer Types Building Distributed Data Structures Extending to GPUsThis Talk Background: how do we write a program for a supercomputer? Introduce PGAS Model

file-system caching as the previous level and adds support for distributed cachingCaching Level 4 - Bazel Supports file-system and distributed caching as with the previous level, but based on the digest cache and/or run targets on a shared distributed systemDistributed/remote builds The ability of a build system to cache and/or run targets on a shared distributed system * also known as the thing that that pays my salary :)Distributed builds Level 1 - Make, Ninja, Meson No distributed caching or execution out of the box, everything runs locallyDistributed builds Level 2 - Gradle Has support for

periods The sum of a sufficient number of independent, identically distributed random variables is approximately normally distributed. STABILIZER repeatedly randomizes layoutSTABILIZER generates a new periods The sum of a sufficient number of independent, identically distributed random variables is approximately normally distributed. STABILIZER repeatedly randomizes layoutSTABILIZER generates a new identically distributed random variables is approximately normally distributed. STABILIZER repeatedly randomizes layoutCentral Limit Theorem execution times are normally distributed The sum of a

type erasurekey challenges we’ve faced building distributed systems with good performance and correctnesskey challenges we’ve faced building distributed systems with good performance and correctnesschallenges “messages” to other actors is non-blocking (no waiting!) ● same programming model whether local or distributed“actors” in C++ many actor libraries are based on low-level message-passing “send/receive” struct will often need to be shared and/or moved between execution resources (i.e., cores) but for distributed/network services, the data is often only shared with other machines and bouncing the data unnecessarily

any remote execution API server supported by Bazel ○ Single Host Server/Proxy: BuildBox-CASD ○ Distributed Server: BuildGrid© 2024 Bloomberg Finance L.P. All rights reserved. How RECC Works? ● Computes across developer builds ● Used in all phases of the Software Development Lifecycle (SDLC) ● Distributed remote compilation and linking ● Builds on Small VMs with recc; outperforms non-recc builds on FARM BuildBox-CASD (Local Caching Proxy) BuildBox-Fuse (FUSE filesystem) + Distributed Cache + Distributed Build:© 2024 Bloomberg Finance L.P. All rights reserved. Discover More! Install via

4c: e7f4 b 0x38 Cost of 4 bytes per function call 187Distributed vs Centralized Error Handling 188Distributed Error Handling ● A form of error handling where the work & code for propagating propagating error information up the stack to a handler is distributed across all functions that participate in error handling. ● If a function wants to call a function that can emit an error, it is the of code size compared to distributed error handling. 191Point 1: Branching isn't FREE! 192Point 1: Branching isn't free! Consider the following function using any distributed type error mechanism… uint32_t

That’s why I have a clock on my wall that runs counter-clockwise. Grace HopperA distributed system is knowableA distributed system is one in which the failure of a computer you didn’t even know existed doubt and uncertainty!twitter.com/KevlinHenney/status/1351956942877552646 It is a feature of a distributed system that it may not be in a consistent state, but it is a bug for a client to contradict

UC Berkeley Data structures and algorithms for parallel programs. Working on C++ library of distributed data structures. Please hire me! About Us 2 Scott, Principal Engineer at CMU SEI Graph/ML/AI distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for parallel systems. If you enjoy parallel programming: “PGAS in C++: A Portable Abstraction for Distributed Data Structures” Tuesday, 4:45 PM MDT Location: D) Valley 1 Virtual: Wednesday, 12:30 PM MDT

…C++ parallel algorithms and tasks specifically • Message Passing Interface (MPI): 3 decades of distributed-memory parallel programming • MPI teaches design patterns to detect and handle recoverable errorsWhat true; } What do we do?!?Message Passing Interface • C and Fortran interface – For writing distributed- memory parallel code • Standard for 3 decades – Unified divergent interfaces circa 1991