Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment

Siqi Zhang; Minjie Cai; Lianying Zhao; Xavier De Carné De Carnavalet; Fabio Massacci; Mengyuan Zhang

doi:10.1145/3772318.3790409

Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment

Siqi Zhang
, Minjie Cai
, Lianying Zhao
, Xavier De Carné De Carnavalet
, Fabio Massacci
, Mengyuan Zhang^*

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Quantitative vulnerability assessment is central to security management, guiding how risks are prioritized and mitigated. Yet, severity scoring relies on human judgment and is therefore subject to differences in experience, interpretation, and diligence; prior work has even shown expert disagreement. We examine an NLP-based assistive tool that visualizes keyword cues during assessment. In a controlled survey of 389 participants recruited via Amazon MTurk and Prolific, we statistically analyze how participant skills/demographics, vulnerability characteristics, and tool support affect outcomes. Results show the tool does not consistently improve assessment accuracy across expertise levels, but can help for specific vulnerability types (e.g., CWE-787) and CVSS metrics (AC, PR, Scope), and can increase user confidence. Beyond immediate performance, the tool can support training for manual assessment tasks that are hard to automate, as learning effects yield significant improvements on subsequent tasks. This work informs the design of cybersecurity decision-support tools and motivates future research on security training and human-centered security.

Original language	English
Title of host publication	CHI 2026
Subtitle of host publication	Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
Editors	Nuria Oliver, David A. Shamma, Heloisa Candello, Pablo Cesar, Pedro Lopes, Alessandro Bozzon, Thomas Kosch, Vera Liao, Xiaojuan Ma, Valentino Artizzu, Fiona Draxler, Gustavo Lopez, Anke V. Reinschluessel, Xin Tong, Phoebe O. Toups Dugas
Publisher	Association for Computing Machinery
Pages	1-24
Number of pages	24
ISBN (Electronic)	9798400722783
DOIs	https://doi.org/10.1145/3772318.3790409
Publication status	Published - 13 Apr 2026
Event	2026 CHI Conference on Human Factors in Computing Systems, CHI 2026 - Barcelona, Spain Duration: 13 Apr 2026 → 17 Apr 2026

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2026 CHI Conference on Human Factors in Computing Systems, CHI 2026
Country/Territory	Spain
City	Barcelona
Period	13/04/26 → 17/04/26

Bibliographical note

Publisher Copyright:
© 2026 Copyright held by the owner/author(s).

Funding

The work was partly supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) under grant n. NWA.1215.18.006 (Theseus), the European Union (EU) under Horizon Europe grant n. 101120393 (Sec4AI4Sec), and by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) under grant n. KIC1.VE01.20.004 (HEWSTI), the Dutch sectorplan, NSERC Discovery Grant n. RGPIN-2020-04734, NSERC Alliance Grant n. ALLRP 558365-20, and by the Network Institute, Vrije Universiteit Amsterdam through the Research Visits program. We thank anonymous reviewers for their helpful feedback. We thank Siu Hong (Thomas) Tam for conducting the pilot study

Funders	Funder number
European Commission
HEWSTI
Natural Sciences and Engineering Research Council of Canada	RGPIN-2020-04734, ALLRP 558365-20
HORIZON EUROPE Framework Programme	KIC1.VE01.20.004, 101120393
Nederlandse Organisatie voor Wetenschappelijk Onderzoek	NWA.1215.18.006

Keywords

CVE
CVSS
Human-computer Interaction
NLP
User Study

Access to Document

10.1145/3772318.3790409Licence: CC BY

https://hdl.handle.net/2066/331387

Persistent URL (handle)

Persistent link

Cite this

Zhang, S., Cai, M., Zhao, L., De Carné De Carnavalet, X., Massacci, F., & Zhang, M. (2026). Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment. In N. Oliver, D. A. Shamma, H. Candello, P. Cesar, P. Lopes, A. Bozzon, T. Kosch, V. Liao, X. Ma, V. Artizzu, F. Draxler, G. Lopez, A. V. Reinschluessel, X. Tong, & P. O. Toups Dugas (Eds.), CHI 2026: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (pp. 1-24). Article 1219 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3772318.3790409

Zhang, Siqi ; Cai, Minjie ; Zhao, Lianying et al. / Tool-Assisted CVSS Vulnerability Scoring : A Controlled Quantitative Study of Human Assessment. CHI 2026: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. editor / Nuria Oliver ; David A. Shamma ; Heloisa Candello ; Pablo Cesar ; Pedro Lopes ; Alessandro Bozzon ; Thomas Kosch ; Vera Liao ; Xiaojuan Ma ; Valentino Artizzu ; Fiona Draxler ; Gustavo Lopez ; Anke V. Reinschluessel ; Xin Tong ; Phoebe O. Toups Dugas. Association for Computing Machinery, 2026. pp. 1-24 (Conference on Human Factors in Computing Systems - Proceedings).

@inproceedings{abd83cafa719487e98df0c9cdb137863,

title = "Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment",

abstract = "Quantitative vulnerability assessment is central to security management, guiding how risks are prioritized and mitigated. Yet, severity scoring relies on human judgment and is therefore subject to differences in experience, interpretation, and diligence; prior work has even shown expert disagreement. We examine an NLP-based assistive tool that visualizes keyword cues during assessment. In a controlled survey of 389 participants recruited via Amazon MTurk and Prolific, we statistically analyze how participant skills/demographics, vulnerability characteristics, and tool support affect outcomes. Results show the tool does not consistently improve assessment accuracy across expertise levels, but can help for specific vulnerability types (e.g., CWE-787) and CVSS metrics (AC, PR, Scope), and can increase user confidence. Beyond immediate performance, the tool can support training for manual assessment tasks that are hard to automate, as learning effects yield significant improvements on subsequent tasks. This work informs the design of cybersecurity decision-support tools and motivates future research on security training and human-centered security.",

keywords = "CVE, CVSS, Human-computer Interaction, NLP, User Study",

author = "Siqi Zhang and Minjie Cai and Lianying Zhao and \{De Carn{\'e} De Carnavalet\}, Xavier and Fabio Massacci and Mengyuan Zhang",

note = "Publisher Copyright: {\textcopyright} 2026 Copyright held by the owner/author(s).; 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026 ; Conference date: 13-04-2026 Through 17-04-2026",

year = "2026",

month = apr,

day = "13",

doi = "10.1145/3772318.3790409",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

pages = "1--24",

editor = "Nuria Oliver and Shamma, \{David A.\} and Heloisa Candello and Pablo Cesar and Pedro Lopes and Alessandro Bozzon and Thomas Kosch and Vera Liao and Xiaojuan Ma and Valentino Artizzu and Fiona Draxler and Gustavo Lopez and Reinschluessel, \{Anke V.\} and Xin Tong and \{Toups Dugas\}, \{Phoebe O.\}",

booktitle = "CHI 2026",

}

Zhang, S, Cai, M, Zhao, L, De Carné De Carnavalet, X, Massacci, F & Zhang, M 2026, Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment. in N Oliver, DA Shamma, H Candello, P Cesar, P Lopes, A Bozzon, T Kosch, V Liao, X Ma, V Artizzu, F Draxler, G Lopez, AV Reinschluessel, X Tong & PO Toups Dugas (eds), CHI 2026: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems., 1219, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, pp. 1-24, 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026, Barcelona, Spain, 13/04/26. https://doi.org/10.1145/3772318.3790409

Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment. / Zhang, Siqi; Cai, Minjie; Zhao, Lianying et al.
CHI 2026: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. ed. / Nuria Oliver; David A. Shamma; Heloisa Candello; Pablo Cesar; Pedro Lopes; Alessandro Bozzon; Thomas Kosch; Vera Liao; Xiaojuan Ma; Valentino Artizzu; Fiona Draxler; Gustavo Lopez; Anke V. Reinschluessel; Xin Tong; Phoebe O. Toups Dugas. Association for Computing Machinery, 2026. p. 1-24 1219 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Tool-Assisted CVSS Vulnerability Scoring

T2 - 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026

AU - Zhang, Siqi

AU - Cai, Minjie

AU - Zhao, Lianying

AU - De Carné De Carnavalet, Xavier

AU - Massacci, Fabio

AU - Zhang, Mengyuan

PY - 2026/4/13

Y1 - 2026/4/13

N2 - Quantitative vulnerability assessment is central to security management, guiding how risks are prioritized and mitigated. Yet, severity scoring relies on human judgment and is therefore subject to differences in experience, interpretation, and diligence; prior work has even shown expert disagreement. We examine an NLP-based assistive tool that visualizes keyword cues during assessment. In a controlled survey of 389 participants recruited via Amazon MTurk and Prolific, we statistically analyze how participant skills/demographics, vulnerability characteristics, and tool support affect outcomes. Results show the tool does not consistently improve assessment accuracy across expertise levels, but can help for specific vulnerability types (e.g., CWE-787) and CVSS metrics (AC, PR, Scope), and can increase user confidence. Beyond immediate performance, the tool can support training for manual assessment tasks that are hard to automate, as learning effects yield significant improvements on subsequent tasks. This work informs the design of cybersecurity decision-support tools and motivates future research on security training and human-centered security.

AB - Quantitative vulnerability assessment is central to security management, guiding how risks are prioritized and mitigated. Yet, severity scoring relies on human judgment and is therefore subject to differences in experience, interpretation, and diligence; prior work has even shown expert disagreement. We examine an NLP-based assistive tool that visualizes keyword cues during assessment. In a controlled survey of 389 participants recruited via Amazon MTurk and Prolific, we statistically analyze how participant skills/demographics, vulnerability characteristics, and tool support affect outcomes. Results show the tool does not consistently improve assessment accuracy across expertise levels, but can help for specific vulnerability types (e.g., CWE-787) and CVSS metrics (AC, PR, Scope), and can increase user confidence. Beyond immediate performance, the tool can support training for manual assessment tasks that are hard to automate, as learning effects yield significant improvements on subsequent tasks. This work informs the design of cybersecurity decision-support tools and motivates future research on security training and human-centered security.

KW - CVE

KW - CVSS

KW - Human-computer Interaction

KW - NLP

KW - User Study

UR - https://www.scopus.com/pages/publications/105038771740

UR - https://www.scopus.com/pages/publications/105038771740#tab=citedBy

U2 - 10.1145/3772318.3790409

DO - 10.1145/3772318.3790409

M3 - Conference contribution

AN - SCOPUS:105038771740

T3 - Conference on Human Factors in Computing Systems - Proceedings

SP - 1

EP - 24

BT - CHI 2026

A2 - Oliver, Nuria

A2 - Shamma, David A.

A2 - Candello, Heloisa

A2 - Cesar, Pablo

A2 - Lopes, Pedro

A2 - Bozzon, Alessandro

A2 - Kosch, Thomas

A2 - Liao, Vera

A2 - Ma, Xiaojuan

A2 - Artizzu, Valentino

A2 - Draxler, Fiona

A2 - Lopez, Gustavo

A2 - Reinschluessel, Anke V.

A2 - Tong, Xin

A2 - Toups Dugas, Phoebe O.

PB - Association for Computing Machinery

Y2 - 13 April 2026 through 17 April 2026

ER -

Zhang S, Cai M, Zhao L, De Carné De Carnavalet X, Massacci F , Zhang M. Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment. In Oliver N, Shamma DA, Candello H, Cesar P, Lopes P, Bozzon A, Kosch T, Liao V, Ma X, Artizzu V, Draxler F, Lopez G, Reinschluessel AV, Tong X, Toups Dugas PO, editors, CHI 2026: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2026. p. 1-24. 1219. (Conference on Human Factors in Computing Systems - Proceedings). Epub 2026 Apr 13. doi: 10.1145/3772318.3790409

Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this