TY - GEN
T1 - A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama
AU - Cursaru, Vlad Andrei
AU - Duits, Laura
AU - Milligan, Joel
AU - Ural, Damla
AU - Sanchez, Berta Rodriguez
AU - Stoico, Vincenzo
AU - Malavolta, Ivano
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
PY - 2024
Y1 - 2024
N2 - Context. Large Language Models (LLMs) are now crucial for developers to increase productivity and reduce software development time and cost. Code Llama, an LLM from Meta, is one of the most recent LLM tools. However, currently there is no objective assessment of the energy efficiency of the source code generated by Code Llama. Goal. In this paper, we present an empirical study that assesses the energy efficiency of the source code generated by Code Llama with respect to human-written source code. Method. We design an experiment involving three human-written programming problems implemented in C++, JavaScript, and Python. We ask Code Llama to generate the code of the problems using different prompts and temperatures, which sets the predictability of the output of an LLM. Therefore, we execute both implementations and profile their energy efficiency. Results. Our study shows that the energy efficiency of the code generated by Code Llama varies according to the chosen programming language and code characteristics. Human implementations tend to be more energy efficient overall, with generated JavaScript code outperforming its human counterpart. In addition, explicitly asking Code Llama to generate energy-efficient code results in an equal or worse energy efficiency, and using different temperatures does not seem to affect the energy efficiency of generated code. Conclusions. According to our results, code generated using Code Llama does not guarantee energy efficiency, even when prompted to do so. Therefore, software developers should evaluate the energy efficiency of generated code before integrating it into the software system under development.
AB - Context. Large Language Models (LLMs) are now crucial for developers to increase productivity and reduce software development time and cost. Code Llama, an LLM from Meta, is one of the most recent LLM tools. However, currently there is no objective assessment of the energy efficiency of the source code generated by Code Llama. Goal. In this paper, we present an empirical study that assesses the energy efficiency of the source code generated by Code Llama with respect to human-written source code. Method. We design an experiment involving three human-written programming problems implemented in C++, JavaScript, and Python. We ask Code Llama to generate the code of the problems using different prompts and temperatures, which sets the predictability of the output of an LLM. Therefore, we execute both implementations and profile their energy efficiency. Results. Our study shows that the energy efficiency of the code generated by Code Llama varies according to the chosen programming language and code characteristics. Human implementations tend to be more energy efficient overall, with generated JavaScript code outperforming its human counterpart. In addition, explicitly asking Code Llama to generate energy-efficient code results in an equal or worse energy efficiency, and using different temperatures does not seem to affect the energy efficiency of generated code. Conclusions. According to our results, code generated using Code Llama does not guarantee energy efficiency, even when prompted to do so. Therefore, software developers should evaluate the energy efficiency of generated code before integrating it into the software system under development.
KW - Code Llama
KW - Energy Consumption
KW - Green Software
KW - Large Language Model
UR - http://www.scopus.com/inward/record.url?scp=85204638478&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85204638478&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-70245-7_12
DO - 10.1007/978-3-031-70245-7_12
M3 - Conference contribution
AN - SCOPUS:85204638478
SN - 9783031702440
T3 - Communications in Computer and Information Science
SP - 161
EP - 176
BT - Quality of Information and Communications Technology
A2 - Bertolino, Antonia
A2 - Pascoal Faria, João
A2 - Lago, Patricia
A2 - Semini, Laura
PB - Springer Science and Business Media Deutschland GmbH
T2 - 17th International Conference on the Quality of Information and Communications Technology, QUATIC 2024
Y2 - 11 September 2024 through 13 September 2024
ER -