Variance If p-value ≤ 5% we reject the null hypothesis p-value = 26.4% -O3 -O2 vs one in four experiments will show an effect that does not exist!Analysis of Variance If p-value ≤ 5% we reject the know causes this effect?� � � � � � Performance Experiments � � � � If we could magically speed up … �� � � � � � Performance Experiments � � � � If we could magically speed up � � � Performance Experiments � � � � If we could magically speed up … � More speedup in … � leads to a larger program speedup.� � � � � � Performance Experiments � � � � If we

Knowledge Graph oriented News Data Experiments – Experimental setting A Conceptual Knowledge Graph oriented News Data Experiments – Evaluation measures Experiments – Performance analysis of our extraction (including topic 1 to topic 5) from two datasets A Conceptual Knowledge Graph oriented News Data Experiments – Performance analysis of our extraction approach Table 1: Evaluation of precision, recall, and F-score on five independent document topics (including topic 6 to topic 10) from two datasets Experiments – Quality analysis of the conceptual knowledge graph A Conceptual Knowledge Graph oriented News

THE OPTIMIZER SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Yes (at least in my experiments) A: Yes (at least in my experiments) 56HOW DID WE DO? HOW DID WE DO? No macros? No manual control �ow? Declarative SEE SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Again yes (at least in my experiments) A: Again yes (at least in my experiments) 64A CONCLUSION FROM THIS A CONCLUSION FROM THIS EXPERIMENT? EXPERIMENT

2014 : une webapp mobile pour LACIS • Plein d’exemples – html5 rocks!5 => web.dev6 – Chrome Experiments7 – MDN8 – html5 demos9 – plus de demos ?10 1https://web.archive.org/web/20140320042652/http://slides html5rocks.com 2https://github.com/html5rocks/slides.html5rocks.com 3https://www.soundstep.com/blog/experiments/jsdetection/ 4https://auduno.github.io/clmtrackr/examples/facesubstitution.html 5https://www.html5rocks

Hypothesis around Steady State Behavior • Vary Real-world Events • Run Experiments in Production • Automate Experiments to Run Continuously • Minimize Blast Radius http://principlesofchaos.org

algorithm This is just error propagation (the bane of all freshman physics labs) 24Qualities of experiments 1. Precision 2. Reproducibility 3. Accuracy 25Precision Precision maximizes information content the code in question 26Reproducibility (handling nondeterministic results) Most scientific experiments have measurement error or noise. Interference from the environment: noisy signals Inherent in

which records a start value, end value, and the change (start - end). We would like to find all experiments where the change was equal to a certain amount (Experiment.objects. filter(change__abs=27)), or Experiment.objects.filter(change__abs=27) will gener- ate the following SQL: SELECT ... WHERE ABS("experiments"."change") = 27 By using Transform instead of Lookup it means we are able to chain further lookups Experiment.objects.filter(change__abs__lt=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") < 27 Note that in case there is no other lookup specified, Django interprets change__abs=27

which records a start value, end value, and the change (start - end). We would like to find all experiments where the change was equal to a certain amount (Experiment.objects.filter(change__abs=27)), or Experiment.objects.filter(change__abs=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") = 27 By using Transform instead of Lookup it means we are able to chain further lookups Experiment.objects.filter(change__abs__lt=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") < 27 Note that in case there is no other lookup specified, Django interprets change__abs=27

which records a start value, end value, and the change (start - end). We would like to find all experiments where the change was equal to a certain amount (Experiment.objects.filter(change__abs=27)), or Experiment.objects.filter(change__abs=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") = 27 By using Transform instead of Lookup it means we are able to chain further lookups Experiment.objects.filter(change__abs__lt=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") < 27 Note that in case there is no other lookup specified, Django interprets change__abs=27

which records a start value, end value, and the change (start - end). We would like to find all experiments where the change was equal to a certain amount (Experiment.objects. filter(change__abs=27)), or Experiment.objects.filter(change__abs=27) will gener- ate the following SQL: SELECT ... WHERE ABS("experiments"."change") = 27 By using Transform instead of Lookup it means we are able to chain further lookups Experiment.objects.filter(change__abs__lt=27) will generate the following SQL: SELECT ... WHERE ABS("experiments"."change") < 27 Note that in case there is no other lookup specified, Django interprets change__abs=27