V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

Bogdan Ghit; Mihai Capota; Tim Hegeman; Jan Hidders; Dick H.J. Epema; Alexandru Iosup

doi:10.1109/CCGrid.2014.97

V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

Bogdan Ghit, Mihai Capota, Tim Hegeman, Jan Hidders, Dick H.J. Epema, Alexandru Iosup

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In this paper we present the scaling of BTWorld, our MapReduce-based approach to observing and analyzing the global BitTorrent network which we have been monitoring for the past 4 years. BTWorld currently provides a comprehensive and complex set of queries implemented in Pig Latin, with data dependencies between them, which translate to several MapReduce jobs that have a heavy-tailed distribution with respect to both execution time and input size characteristics. Processing BitTorrent data in excess of 1 TB with our BTWorld workflow required an in-depth analysis of the entire software stack and the design of a complete optimization cycle. We analyze our system from both theoretical and experimental perspectives and we show how we attained a 15 times larger scale of data processing than our previous results. © 2014 IEEE.

Original language	English
Title of host publication	Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014
Publisher	ACM, IEEE Computer Society
Pages	927-932
Number of pages	6
ISBN (Print)	9781479927838
DOIs	https://doi.org/10.1109/CCGrid.2014.97
Publication status	Published - 2014
Externally published	Yes
Event	14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014 - Chicago, United States Duration: 26 May 2014 → 29 May 2014

Conference

Conference	14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014
Country/Territory	United States
City	Chicago
Period	26/05/14 → 29/05/14

Access to Document

10.1109/CCGrid.2014.97

http://dx.doi.org/10.1109/CCGrid.2014.97

Persistent URL (handle)

Persistent link

Cite this

Ghit, B., Capota, M., Hegeman, T., Hidders, J., Epema, D. H. J., & Iosup, A. (2014). V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows. In Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014 (pp. 927-932). Article 6846548 ACM, IEEE Computer Society. https://doi.org/10.1109/CCGrid.2014.97

@inproceedings{2e671d245c23466ab54d9e63af49f273,

title = "V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows",

abstract = "In this paper we present the scaling of BTWorld, our MapReduce-based approach to observing and analyzing the global BitTorrent network which we have been monitoring for the past 4 years. BTWorld currently provides a comprehensive and complex set of queries implemented in Pig Latin, with data dependencies between them, which translate to several MapReduce jobs that have a heavy-tailed distribution with respect to both execution time and input size characteristics. Processing BitTorrent data in excess of 1 TB with our BTWorld workflow required an in-depth analysis of the entire software stack and the design of a complete optimization cycle. We analyze our system from both theoretical and experimental perspectives and we show how we attained a 15 times larger scale of data processing than our previous results. {\textcopyright} 2014 IEEE.",

author = "Bogdan Ghit and Mihai Capota and Tim Hegeman and Jan Hidders and Epema, {Dick H.J.} and Alexandru Iosup",

year = "2014",

doi = "10.1109/CCGrid.2014.97",

language = "English",

isbn = "9781479927838",

pages = "927--932",

booktitle = "Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014",

publisher = "ACM, IEEE Computer Society",

note = "14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014 ; Conference date: 26-05-2014 Through 29-05-2014",

}

Ghit, B, Capota, M, Hegeman, T, Hidders, J, Epema, DHJ & Iosup, A 2014, V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows. in Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014., 6846548, ACM, IEEE Computer Society, pp. 927-932, 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, Illinois, United States, 26/05/14. https://doi.org/10.1109/CCGrid.2014.97

V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows. / Ghit, Bogdan; Capota, Mihai; Hegeman, Tim et al.
Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014. ACM, IEEE Computer Society, 2014. p. 927-932 6846548.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

AU - Ghit, Bogdan

AU - Capota, Mihai

AU - Hegeman, Tim

AU - Hidders, Jan

AU - Epema, Dick H.J.

AU - Iosup, Alexandru

PY - 2014

Y1 - 2014

N2 - In this paper we present the scaling of BTWorld, our MapReduce-based approach to observing and analyzing the global BitTorrent network which we have been monitoring for the past 4 years. BTWorld currently provides a comprehensive and complex set of queries implemented in Pig Latin, with data dependencies between them, which translate to several MapReduce jobs that have a heavy-tailed distribution with respect to both execution time and input size characteristics. Processing BitTorrent data in excess of 1 TB with our BTWorld workflow required an in-depth analysis of the entire software stack and the design of a complete optimization cycle. We analyze our system from both theoretical and experimental perspectives and we show how we attained a 15 times larger scale of data processing than our previous results. © 2014 IEEE.

AB - In this paper we present the scaling of BTWorld, our MapReduce-based approach to observing and analyzing the global BitTorrent network which we have been monitoring for the past 4 years. BTWorld currently provides a comprehensive and complex set of queries implemented in Pig Latin, with data dependencies between them, which translate to several MapReduce jobs that have a heavy-tailed distribution with respect to both execution time and input size characteristics. Processing BitTorrent data in excess of 1 TB with our BTWorld workflow required an in-depth analysis of the entire software stack and the design of a complete optimization cycle. We analyze our system from both theoretical and experimental perspectives and we show how we attained a 15 times larger scale of data processing than our previous results. © 2014 IEEE.

UR - http://www.scopus.com/inward/record.url?scp=84904542631&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84904542631&partnerID=8YFLogxK

U2 - 10.1109/CCGrid.2014.97

DO - 10.1109/CCGrid.2014.97

M3 - Conference contribution

SN - 9781479927838

SP - 927

EP - 932

BT - Proceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014, Chicago, IL, USA, May 26-29, 2014

PB - ACM, IEEE Computer Society

T2 - 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014

Y2 - 26 May 2014 through 29 May 2014

ER -

V for Vicissitude: The Challenge of Scaling Complex Big Data Workflows

Abstract

Conference

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this