1 SCIENCE PASSION TECHNOLOGY Introduction to Scientific Writing 03 Experiments & Reproducibility Matthias Boehm Graz University of Technology, Austria Institute of Interactive Systems and Data Science Data Management Last update: Nov 11, 2021 2 706.015 Introduction to Scientific Writing – 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Announcements/Org 015 Introduction to Scientific Writing – 03 Experiments & Reproducibility Matthias Boehm, Graz University of Technology, WS 2021/22 Agenda  Experiments and Result Presentation  Reproducibility and

Papers [Oct 28, 6.15pm, optional]  02 Scientific Reading and Writing [Nov 04, 6pm, optional]  03 Experiments, Reproducibility, and Projects [Nov 11, 6pm, optional] ...  04 Project Presentations [Jan 13, Prototypes and Experiments  Worst Mistake: Schrödinger's Results  Postpone implementation and experiments till last before the deadline  No feedback, no reaction time (experiments require many iterations) Popper: falsifiability of scientific results  Continuous Experiments  Run experiments during survey / prototype building  Systematic experiments  observations and ideas for improvements  Don’t be afraid

overall idea clearly communicated and does it make sense?  Are there missing pieces, missing experiments, missing related work? Scientific Reading  Read out loud  Use PDF-to-Speech 11 706.015 Accept if no time to review  The Goldilocks Method (examples, proofs, theoretical analysis, experiments)  If you can’t say something nasty … (ignore good parts, focus on weaknesses)  Silent but deadly references are omitted”  Proposed Method  To simple, impractical, or well-known; correctness?  Experiments  Datasets synthetic/real, not all aspects evaluated, too small datasets  Conclusions  Disagree

Variance If p-value ≤ 5% we reject the null hypothesis p-value = 26.4% -O3 -O2 vs one in four experiments will show an effect that does not exist!Analysis of Variance If p-value ≤ 5% we reject the know causes this effect?� � � � � � Performance Experiments � � � � If we could magically speed up … �� � � � � � Performance Experiments � � � � If we could magically speed up � � � Performance Experiments � � � � If we could magically speed up … � More speedup in … � leads to a larger program speedup.� � � � � � Performance Experiments � � � � If we

Knowledge Graph oriented News Data Experiments – Experimental setting A Conceptual Knowledge Graph oriented News Data Experiments – Evaluation measures Experiments – Performance analysis of our extraction (including topic 1 to topic 5) from two datasets A Conceptual Knowledge Graph oriented News Data Experiments – Performance analysis of our extraction approach Table 1: Evaluation of precision, recall, and F-score on five independent document topics (including topic 6 to topic 10) from two datasets Experiments – Quality analysis of the conceptual knowledge graph A Conceptual Knowledge Graph oriented News

desired outcomes. 3. Repeat iii. Intuit’s rampant innovation culture – went from 7 experiments/year to 165 experiments during the 3 month US tax season in 2010 with website conversion rates up 50% b the Analysis and Experimentation group at Microsoft: “evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving INTO OUR RELEASE i. A/B testing requires fast CD to support ii. Use feature toggles to control experiments, cohort creation, etc. iii. Use telemetry to measure outcomes iv. Etsy open-sourced their experimentation

THE OPTIMIZER SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Yes (at least in my experiments) A: Yes (at least in my experiments) 56HOW DID WE DO? HOW DID WE DO? No macros? No manual control �ow? Declarative SEE SEE THROUGH ALL THIS? THROUGH ALL THIS? A: Again yes (at least in my experiments) A: Again yes (at least in my experiments) 64A CONCLUSION FROM THIS A CONCLUSION FROM THIS EXPERIMENT? EXPERIMENT

exclude the compute spent in all the intermediate steps in getting to the final model such as experiments with architectures, hyper-parameter tuning, and model performance debugging. However, since the label_smoothing parameter in the CategoricalCrossentropy loss function, which you can easily set in your experiments. Yet another way of improving generalization is to allow the model to learn concepts in the order order of their difficulty. Curriculum learning shows us how. Curriculum Learning We know from experiments and machine learning theory that increasing the size of the dataset typically helps improve quality

is also optimized based on an improved version of FlashAttention-2 (Dao, 2023). We conduct all experiments on a cluster equipped with NVIDIA H800 GPUs. Each node in the H800 cluster contains 8 GPUs connected ({?1, ?2, · · · , ??}) s??({?1, ?2, · · · , ??}) . (34) Training Strategy. In our preliminary experiments, we find that the RL training on reasoning data, such as code and math prompts, exhibits unique with DeepSeek-V2 Chat (SFT) and train them with either a point-wise or a pair-wise loss. In our experiments, we observe that the RL training can fully tap into and activate the potential of our model, enabling

and transfer learning • Automate repeatable ML experiments with containers • Deploy ML components to Kubernetes with Kubeflow • Scale and test ML experiments with Helm • Manage training jobs and pipelines Tensorflow Serving • Seldon Demo: Run TensorFlow Training with Kubeflow Demo: Scale and Test Experiments in Parallel using Kubernetes, TFJob, and Helm • Spin up pods for each variation of hyperparameters