Engineering for a science-centric experimentation platform

Nikos Diamantopoulos, Jeffrey Wong, David Issa Mattos, Ilias Gerostathopoulos, Matthew Wardrop, Tobias Mao, Colin McFarland

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

20 Downloads (Pure)


Netflix is an internet entertainment service that routinely employs experimentation to guide strategy around product innovations. As Netflix grew, it had the opportunity to explore increasingly specialized improvements to its service, which generated demand for deeper analyses supported by richer metrics and powered by more diverse statistical methodologies. To facilitate this, and more fully harness the skill sets of both engineering and data science, Netflix engineers created a science-centric experimentation platform that leverages the expertise of scientists from a wide range of backgrounds working on data science tasks by allowing them to make direct code contributions in the languages used by them (Python and R). Moreover, the same code that runs in production is able to be run locally, making it straightforward to explore and graduate both metrics and causal inference methodologies directly into production services. In this paper, we provide two main contributions. Firstly, we report on the architecture of this platform, with a special emphasis on its novel aspects: how it supports science-centric end-to-end workflows without compromising engineering requirements. Secondly, we describe its approach to causal inference, which leverages the potential outcomes conceptual framework to provide a unified abstarction layer for arbitrary statistical models and methodologies.

Original languageEnglish
Title of host publicationICSE-SEIP '20
Subtitle of host publicationProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice
PublisherIEEE Computer Society
Number of pages10
ISBN (Electronic)9781450371230
Publication statusPublished - Jun 2020
Event42nd ACM/IEEE International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2020 - Virtual, Online, Korea, Republic of
Duration: 27 Jun 202019 Jul 2020

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257


Conference42nd ACM/IEEE International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2020
Country/TerritoryKorea, Republic of
CityVirtual, Online


This work was partially supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and by the Software Center. This work was partially sponsored by the German Ministry of Education and Research (grant no 01Is16043A).

FundersFunder number
Autonomous Systems and Software Program
Wallenberg Artificial Intelligence
Bundesministerium für Bildung und Forschung01Is16043A
Knut och Alice Wallenbergs Stiftelse


    • A/B testing
    • Causal inference
    • Experimentation
    • Science-centric
    • Software architecture


    Dive into the research topics of 'Engineering for a science-centric experimentation platform'. Together they form a unique fingerprint.

    Cite this