A quantitative analysis of the robustness of knowledge-based systems through degradation studies

The overall aim of this paper is to provide a general setting for quantitative quality measures of knowledge-based system behaviour that is widely applicable to many knowledge-based systems. We propose a general approach that we call degradation studies: an analysis of how system output changes as a function of degrading system input, such as incomplete or incorrect data or knowledge. To show the feasibility of our approach, we have applied it in a case study. We have taken a large and realistic vegetation-classification system, and have analysed its behaviour under various varieties of incomplete and incorrect input. This case study shows that degradation studies can reveal interesting and surprising properties of the system under study.