The state of the art in measurement-based experiments on the mobile web

Context: Nowadays the majority of all worldwide Web traffic comes from mobile devices, as we tend to primarily rely on the browsers installed on our smartphones and tablets (e.g., Chrome for Android, Safari for iOS) for accessing online services. A market of such a large scale leads to an extremely fierce competition, where it is of paramount importance that the developed mobile Web apps are of high quality, e.g., in terms of performance, energy consumption, security, usability. In order to objectively assess the quality of mobile Web apps, practitioners and researchers are conducting experiments based on the measurement of run-time metrics such as battery discharge, CPU and memory usage, number and type of network requests, etc. Objective: The objective of this work is to identify, classify, and evaluate the state of the art of conducting measurement-based experiments on the mobile Web. Specifically, we focus on (i) which metrics are employed during experimentation, how they are measured, and how they are analyzed; (ii) the platforms chosen to run the experiments; (iii) what subjects are used; (iv) the used tools and environments under which the experiments are run. Method: We apply the systematic mapping methodology. Starting from a search process that identified 786 potentially relevant studies, we selected a set of 33 primary studies following a rigorous selection procedure. We defined and applied a classification framework to them to extract data and gather relevant insights. Results: This work contributes with (i) a classification framework for measurement-based experiments on the mobile Web; (ii) a systematic map of current research on the topic; (iii) a discussion of emergent findings and challenges, and resulting implications for future research. Conclusion: This study provides a rigorous and replicable map of the state of the art of conducting measurement-based experiments on the mobile Web. Its results can benefit researchers and practitioners by presenting common techniques, empirical practices, and tools to properly conduct measurement-based experiments on the mobile Web.