APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities

Quang Cuong Bui*, Ranindya Paramitha, Duc Ly Vu, Fabio Massacci, Riccardo Scandariato

*Corresponding author for this work

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Security vulnerability fixes could be a promising research avenue for Automated Program Repair (APR) techniques. In recent years, APR tools have been thoroughly developed for fixing generic bugs. However, the area is still relatively unexplored when it comes to fixing security bugs or vulnerabilities. In this paper, we evaluate nine state-of-the-art APR tools and one vulnerability-specific repair tool. In particular, we investigate their ability to generate patches for 79 real-world Java vulnerabilities in the Vul4J dataset, as well as the level of trustworthiness of these patches. We evaluate the tools with respect to their ability to generate security patches that are (i) testable, (ii) having the positive effect of closing the vulnerability, and (iii) not having side effects from a functional point of view. Our results show that the evaluated APR tools were able to generate testable patches for around 20% of the considered vulnerabilities. On average, nearly 73% of the testable patches indeed eliminate the vulnerabilities, but only 44% of them could actually fix security bugs while maintaining the functionalities. To understand the root cause of this phenomenon, we conduct a detailed comparative study of the general bug fix patterns in Defect4J and the vulnerability fix patterns in ExtraVul (which we extend from Vul4J). Our investigation shows that, although security patches are short in terms of lines of code, they contain unique characteristics in their fix patterns compared to general bugs. For example, many security fixes require adding method calls. These method calls contain specific input validation-related keywords, such as encode, normalize, and trim. In this regard, our study suggests that additional repair patterns should be implemented for existing APR tools to fix more types of security vulnerabilities.

Original languageEnglish
Article number18
Number of pages40
JournalEmpirical Software Engineering
Volume29
Early online date6 Dec 2023
DOIs
Publication statusPublished - 2024

Bibliographical note

Funding Information:
Open Access funding enabled and organized by Projekt DEAL. This work is partly funded by EU grants No. 952647 (AssureMOSS) and No. 101120393 (Sec4AI4Sec).

Publisher Copyright:
© 2023, The Author(s).

Funding

Open Access funding enabled and organized by Projekt DEAL. This work is partly funded by EU grants No. 952647 (AssureMOSS) and No. 101120393 (Sec4AI4Sec).

FundersFunder number
H2020 LEIT Information and Communication Technologies
HORIZON EUROPE Civil security for society101120393
HORIZON EUROPE Civil security for society
European Commission952647
European Commission

    Keywords

    • Automated program repair
    • Empirical experiments
    • Java
    • Vulnerability

    Fingerprint

    Dive into the research topics of 'APR4Vul: an empirical study of automatic program repair techniques on real-world Java vulnerabilities'. Together they form a unique fingerprint.

    Cite this