Have you ever wondered what those pesky folks at large tech companies are doing changing the UI of their apps and websites one hundred times a day? They’re probably running an experiment on you; yes you! Experiments are one of the most common tools used by—usually larger, but more than ever, smaller—tech companies to understand user behaviour, increase revenue and drive better experiences for customers. Most commonly in the form of A/B tests, they usually involve making a small modification to the experience of a web application and seeing whether that results in the increase of some “metric”, whether that be users, purchases or something else.

However, besides just pushing whack new experiences to production, there is a lot of planning, consideration and statistics that goes into a successful experiment. The aim of this talk is to explain how we do experiments in the growth team of a reasonably large-sized company, Atlassian, and the importance of seemingly small decisions in experimentation. We'll go over all three main phases of an experiment; thinking, implementing and measuring, all with the help of a worked example.