The need to deliver code changes to production systems to satisfy new requirements has fueled the adoption of an agile software development practice called onlineexperimentation. Online experimentation provides insight into the value delivered by new application versions as they are exposed to users.
To solve the online experimentation problem for web and mobile applications, practitioners use A/B tests or more advanced methods such as multi-armed bandit algorithms. These approaches entail comparing and assessing application versions online to determine the best version based on business requirements such as user-engagement. However, existing techniques and their formulations do not capture the unique complexities in cloud systems.
When assessing the outcomes of releases of microservices or machine learning (ML) models in the cloud, practitioners must simultaneously consider application performance as well as business metrics. This difference arises because a cloud application’s behavior is inherently volatile due to an increased likelihood of performance bugs or variability, which can degrade desired business results. For example, Amazon reported that every 100ms of latency costs them 1% in sales. As a result of these complexities, the deployment of cloud applications is more art than science when contrasted with the approaches adopted in the web and mobile domains. However, practitioners lack rigorous solutions for code releases, making it difficult to automatically learn and optimize for both business metrics and application performance with statistical guarantees.
This tutorial aims to provide a new perspective to rethink online experimentation in the cloud era. The tutorial will study the field of online experimentation and popular existing approaches, address their shortcomings in the cloud, and discuss key challenges and requirements for real-world solutions. Participants will get a chance to craft and run an
online experiment on an open-source system designed for online experimentation of microservices and ML models deployed on the cloud.
Schedule
|
Wed 8:05 a.m. - 8:10 a.m.
|
Introduction
(
Talk
)
|
Mert TOSLALI 🔗 |
|
Wed 8:10 a.m. - 8:25 a.m.
|
Background on the software evolution
(
Talk
)
Background on code delivery cycles. |
Mert TOSLALI 🔗 |
|
Wed 8:25 a.m. - 8:30 a.m.
|
Q/A
(
Q/A
)
|
🔗 |
|
Wed 8:30 a.m. - 8:45 a.m.
|
Online experimentation in web/mobile
(
Talk
)
This session will review online experimentation in web/mobile domains. |
Mert TOSLALI 🔗 |
|
Wed 8:45 a.m. - 8:55 a.m.
|
Break & Q/A
(
Break & Q/A
)
|
🔗 |
|
Wed 8:55 a.m. - 9:10 a.m.
|
Online experimentation in the cloud
(
Talk
)
Introduction of cloud online experimentation. |
Mert TOSLALI 🔗 |
|
Wed 9:10 a.m. - 9:15 a.m.
|
Q/A
(
Q/A
)
|
🔗 |
|
Wed 9:15 a.m. - 9:25 a.m.
|
Principles of a trustworthy cloud solution
(
Talk
)
Review of fundamental properties for a principled trustworthy solution to automate online experimentation in the cloud |
Mert TOSLALI 🔗 |
|
Wed 9:25 a.m. - 9:40 a.m.
|
A prospective solution for cloud
(
Talk
)
Prospective solution that builds on the principles described. |
Mert TOSLALI · Srinivasan Parthasarathy 🔗 |
|
Wed 9:40 a.m. - 9:50 a.m.
|
Q/A & Break
(
Q/A & Break
)
|
🔗 |
|
Wed 9:50 a.m. - 10:15 a.m.
|
Hands-on activity
(
Demo
)
Demo exercise using iter8; releasing a new version of a software on Kubernetes via iter8. |
Mert TOSLALI 🔗 |