Capturing the ineffable: Collecting, analysing, and automating web document quality assessments

Davide Ceolin*, Julia Noordegraaf, Lora Aroyo

*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review


Automatic estimation of the quality of Web documents is a challenging task, especially because the definition of quality heavily depends on the individuals who define it, on the context where it applies, and on the nature of the tasks at hand. Our long-term goal is to allow automatic assessment of Web document quality tailored to specific user requirements and context. This process relies on the possibility to identify document characteristics that indicate their quality. In this paper, we investigate these characteristics as follows: (1) we define features of Web documents that may be indicators of quality; (2) we design a procedure for automatically extracting those features; (3) develop a Web application to present these results to niche users to check the relevance of these features as quality indicators and collect quality assessments; (4) we analyse user’s qualitative assessment of Web documents to refine our definition of the features that determine quality, and establish their relevant weight in the overall quality, i.e., in the summarizing score users attribute to a document, determining whether it meets their standards or not. Hence, our contribution is threefold: a Web application for nichesourcing quality assessments; a curated dataset ofWeb document assessments; and a thorough analysis of the quality assessments collected by means of two case studies involving experts (journalists and media scholars). The dataset obtained is limited in size but highly valuable because of the quality of the experts that provided it. Our analyses show that: (1) it is possible to automate the process of Web document quality estimation to a level of high accuracy; (2) document features shown in isolation are poorly informative to users; and (3) related to the tasks we propose (i.e., choosing Web documents to use as a source for writing an article on the vaccination debate), the most important quality dimensions are accuracy, trustworthiness, and precision.

Original languageEnglish
Title of host publicationKnowledge Engineering and Knowledge Management - 20th International Conference, EKAW 2016, Proceedings
Number of pages15
Volume10024 LNAI
ISBN (Print)9783319490038
Publication statusPublished - 2016
Event20th International Conference on Knowledge Engineering and Knowledge Management, EKAW 2016 - Bologna, Italy
Duration: 19 Nov 201623 Nov 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10024 LNAI
ISSN (Print)03029743
ISSN (Electronic)16113349


Conference20th International Conference on Knowledge Engineering and Knowledge Management, EKAW 2016


Dive into the research topics of 'Capturing the ineffable: Collecting, analysing, and automating web document quality assessments'. Together they form a unique fingerprint.

Cite this