The Effect of Decision Trees on Rater Accuracy in Assessment of Parental Behavior

Research output: Contribution to ConferencePaperAcademic


Parental behavior plays an important role in the development of children. When working with at-risk families, professionals need to be able to assess behavior of parents with respect to their children. Information about parental behavior helps to be specific about the need of families, and, if necessary, to refer families to appropriate interventions. In current practice, social workers seldom use reliable and valid instruments to characterize parental behavior. Instruments should not only be reliable and valid, but also suitable for the constraints that
professional practice place on time needed for training to become reliable and time needed to conduct observations and assign observer rating scores. Observation scale instructions need to be as efficient as possible while supporting reliable assessment. To gain insight in the type of instructions, visual or written, that promote reliability and efficiency of assessment, this study was performed. The study tested whether decision trees with visual components leading to scale anchors may be an efficient and reliable alternative to text scales. Our hypothesis was that in the condition where raters used a decision tree, interrater reliability and rater accuracy will be higher on average, and the
user experience will be more positive than in the condition where raters used a traditional written scale.
Material and Methods
Participants in the research were social workers working at Youth Protection Amsterdam. The research was conducted with two groups of participants (N= 217) randomly assigned to two research conditions, decision tree-based anchors and text-based anchors. Each participant in both groups observed six target videos and then rated parental sensitivity according to either the decision tree-anchor scale or the text-anchor scale. Also, they filled out a questionnaire about their user experience.
Analyses showed that interrater reliability was good to excellent (.74 and .78) for both conditions and did not differ between conditions. Also, user experience was equally positive in both conditions. However, the text-anchor scale provided for a higher rater accuracy then the decision tree-anchor scale. In both conditions,
the mean absolute difference with the expert rating was higher when sensitivity was low.
The decision tree-anchor scale performed similarly to slightly worse than the original text anchor-scale. The finding that reliability was similar was promising in light of the potential for developing visual aids for scale instructions for an observational instrument suitable for use by practitioners. The text anchor-scale is a welldeveloped instrument, while the decision tree anchor-scale is the first adaptation to provide for an instrument social workers are able to use in daily practice.
Original languageEnglish
Number of pages1
Publication statusPublished - 2 May 2018
EventEUSARF: All Children, All Families - Promoting Excellence in Child Welfare Research, Policy and Practice - Porto, Portugal
Duration: 2 Oct 20185 Oct 2018




Dive into the research topics of 'The Effect of Decision Trees on Rater Accuracy in Assessment of Parental Behavior'. Together they form a unique fingerprint.

Cite this