New experimental design to capture bias using LLM to validate security threats

Winnie Bahati Mbaka

doi:10.1145/3661167.3661222

New experimental design to capture bias using LLM to validate security threats

Winnie Bahati Mbaka^*

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

38 Downloads (Pure)

Abstract

The usage of Large Language Models is already well understood in software engineering and security and privacy. Yet, little is known about the effectiveness of LLMs in threat validation or the possibility of biased output when assessing security threats for correctness. To mitigate this research gap, we present a pilot study investigating the effectiveness of chatGPT in the validation of security threats. One main observation made from the results was that chatGPT assessed bogus threats as realistic regardless of the assumptions provided which negated the feasibility of certain threats occurring.

Original language	English
Title of host publication	EASE '24
Subtitle of host publication	Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering
Publisher	Association for Computing Machinery
Pages	458-459
Number of pages	2
ISBN (Electronic)	9798400717017
DOIs	https://doi.org/10.1145/3661167.3661222
Publication status	Published - 2024
Event	28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024 - Salerno, Italy Duration: 18 Jun 2024 → 21 Jun 2024

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024
Country/Territory	Italy
City	Salerno
Period	18/06/24 → 21/06/24

Bibliographical note

Publisher Copyright:
© 2024 Owner/Author.

Keywords

ChatGPT
Large Language Models
Security Threat Validation

Access to Document

10.1145/3661167.3661222

New experimental design to capture bias using LLM to validate security threatsFinal published version, 436 KBLicence: Article 25fa Dutch Copyright Act

Persistent URL (handle)

Persistent link

Cite this

@inproceedings{06e3b19cdf214c93a064aa835b3b5d65,

title = "New experimental design to capture bias using LLM to validate security threats",

abstract = "The usage of Large Language Models is already well understood in software engineering and security and privacy. Yet, little is known about the effectiveness of LLMs in threat validation or the possibility of biased output when assessing security threats for correctness. To mitigate this research gap, we present a pilot study investigating the effectiveness of chatGPT in the validation of security threats. One main observation made from the results was that chatGPT assessed bogus threats as realistic regardless of the assumptions provided which negated the feasibility of certain threats occurring.",

keywords = "ChatGPT, Large Language Models, Security Threat Validation",

author = "Mbaka, \{Winnie Bahati\}",

note = "Publisher Copyright: {\textcopyright} 2024 Owner/Author.; 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024 ; Conference date: 18-06-2024 Through 21-06-2024",

year = "2024",

doi = "10.1145/3661167.3661222",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "458--459",

booktitle = "EASE '24",

}

Mbaka, WB 2024, New experimental design to capture bias using LLM to validate security threats. in EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 458-459, 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024, Salerno, Italy, 18/06/24. https://doi.org/10.1145/3661167.3661222

New experimental design to capture bias using LLM to validate security threats. / Mbaka, Winnie Bahati.
EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. Association for Computing Machinery, 2024. p. 458-459 (ACM International Conference Proceeding Series).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - New experimental design to capture bias using LLM to validate security threats

AU - Mbaka, Winnie Bahati

PY - 2024

Y1 - 2024

N2 - The usage of Large Language Models is already well understood in software engineering and security and privacy. Yet, little is known about the effectiveness of LLMs in threat validation or the possibility of biased output when assessing security threats for correctness. To mitigate this research gap, we present a pilot study investigating the effectiveness of chatGPT in the validation of security threats. One main observation made from the results was that chatGPT assessed bogus threats as realistic regardless of the assumptions provided which negated the feasibility of certain threats occurring.

AB - The usage of Large Language Models is already well understood in software engineering and security and privacy. Yet, little is known about the effectiveness of LLMs in threat validation or the possibility of biased output when assessing security threats for correctness. To mitigate this research gap, we present a pilot study investigating the effectiveness of chatGPT in the validation of security threats. One main observation made from the results was that chatGPT assessed bogus threats as realistic regardless of the assumptions provided which negated the feasibility of certain threats occurring.

KW - ChatGPT

KW - Large Language Models

KW - Security Threat Validation

UR - https://www.scopus.com/pages/publications/85197391481

UR - https://www.scopus.com/inward/citedby.url?scp=85197391481&partnerID=8YFLogxK

U2 - 10.1145/3661167.3661222

DO - 10.1145/3661167.3661222

M3 - Conference contribution

AN - SCOPUS:85197391481

T3 - ACM International Conference Proceeding Series

SP - 458

EP - 459

BT - EASE '24

PB - Association for Computing Machinery

T2 - 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024

Y2 - 18 June 2024 through 21 June 2024

ER -

New experimental design to capture bias using LLM to validate security threats

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this