Variance If p-value ≤ 5% we reject the null hypothesis p-value = 26.4% -O3 -O2 vs one in four experiments will show an effect that does not exist!Analysis of Variance If p-value ≤ 5% we reject the know causes this effect?� � � � � � Performance Experiments � � � � If we could magically speed up … �� � � � � � Performance Experiments � � � � If we could magically speed up � � � Performance Experiments � � � � If we could magically speed up … � More speedup in … � leads to a larger program speedup.� � � � � � Performance Experiments � � � � If we

desired outcomes. 3. Repeat iii. Intuit’s rampant innovation culture – went from 7 experiments/year to 165 experiments during the 3 month US tax season in 2010 with website conversion rates up 50% b the Analysis and Experimentation group at Microsoft: “evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving INTO OUR RELEASE i. A/B testing requires fast CD to support ii. Use feature toggles to control experiments, cohort creation, etc. iii. Use telemetry to measure outcomes iv. Etsy open-sourced their experimentation

THE OPTIMIZER SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Yes (at least in my experiments) A: Yes (at least in my experiments) 56HOW DID WE DO? HOW DID WE DO? No macros? No manual control �ow? Declarative SEE SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Again yes (at least in my experiments) A: Again yes (at least in my experiments) 64A CONCLUSION FROM THIS A CONCLUSION FROM THIS EXPERIMENT? EXPERIMENT

algorithm This is just error propagation (the bane of all freshman physics labs) 24Qualities of experiments 1. Precision 2. Reproducibility 3. Accuracy 25Precision Precision maximizes information content the code in question 26Reproducibility (handling nondeterministic results) Most scientific experiments have measurement error or noise. Interference from the environment: noisy signals Inherent in

meant we could not share saved replays between Xbox and PC.” - Shawn Hargreaves, MSDN Blog “Experiments are usually reproducible only on the same machine with the same system library and the same compiler

unprotected garbage that has accumulated Optimizing latency (work in progress) • Throughput experiments show that the average latency is quite low. • What about the tail latency? Is there anything

chapter—by quickly deploying functionality, learning, and adjusting plans.  We can conduct experiments that test the assumptions in our business cases.  We can reduce the overhead of our governance

his co-authors suggest that instead of using detailed planning to manage risk, we instead use experiments.  When we encounter a risk, we should think of something we can do that will help us gain information

[13] Persistent Programming in Real Life 2019 (PIRL 2019), Persistent Memory Evaluation and Experiments (https://www.youtube.com/watch?v=M_kCL1OZjko). Retrieved 3/22/2021. [14] C. Click, “A lock-free

Demand-driven implementation ▪ Collection of the generalizable part of our vectorization Platform for experiments and discussions of future developments ▪ Single header include provided 17 Olaf Krzikalla, DLR