Abstract
Linux storage stack offers a variety of storage I/O stacks and APIs such as POSIX I/O, asynchronous I/O (libaio), high-performance asynchronous I/O (emerging io_uring) or SPDK, the last of which completely bypasses the kernel. Despite their availability, there has not been a systematic study of their performance and overheads. In order to aid our understanding, in this work we systematically characterize performance, scalability and microarchitectural properties of popular Linux I/O APIs on high-performance storage hardware (Intel Optane SSDs). Our characterization reveals that: (1) at low I/O loads, all APIs perform competitively with each other, with polling helping the performance by 1.7X, but consuming 2.3X CPU instructions; (2) at high-loads and scale, io_uring is more than an order of magnitude slower than SPDK; (3) at high-loads and scale, the benchmarking tool (fio) itself becomes a bottleneck; (4) state-of-practice Linux block I/O schedulers (BFQ, mq-deadline, and Kyber) introduce significant (up to 50%) overheads, and their use of global locks hinder their scalability. All artifacts from this work are available at https://github.com/atlarge-research/Performance-Characterization-Storage-Stacks.
| Original language | English |
|---|---|
| Title of host publication | CHEOPS 2023 |
| Subtitle of host publication | Proceedings of the 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems |
| Publisher | Association for Computing Machinery, Inc |
| Pages | 35-45 |
| Number of pages | 11 |
| ISBN (Electronic) | 9798400700811 |
| DOIs | |
| Publication status | Published - May 2023 |
| Event | 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, CHEOPS 2023 - In Conjunction with EuroSys 2023 - Rome, Italy Duration: 8 May 2023 → … |
Conference
| Conference | 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, CHEOPS 2023 - In Conjunction with EuroSys 2023 |
|---|---|
| Country/Territory | Italy |
| City | Rome |
| Period | 8/05/23 → … |
Bibliographical note
Funding Information:This work is funded by The Dutch Research Council (NWO) grant number OCENW.KLEIN.561. The authors would like to thank Jesse Donkervliet, Sacheendra Talluri, Krijn Doekemeijer, and Nick Tehrany for their help with the paper.
Publisher Copyright:
© 2023 Owner/Author(s).
Funding
This work is funded by The Dutch Research Council (NWO) grant number OCENW.KLEIN.561. The authors would like to thank Jesse Donkervliet, Sacheendra Talluri, Krijn Doekemeijer, and Nick Tehrany for their help with the paper.
Keywords
- efficiency
- io_uring
- linux storage stack
- measurements
- SPDK