Abstract
Objectives
Parental behavior plays an important role in the development of children. When working with at-risk families, professionals need to be able to assess behavior of parents with respect to their children. Information about parental behavior helps to be specific about the need of families, and, if necessary, to refer families to appropriate interventions. In current practice, social workers seldom use reliable and valid instruments to characterize parental behavior. Instruments should not only be reliable and valid, but also suitable for the constraints that
professional practice place on time needed for training to become reliable and time needed to conduct observations and assign observer rating scores. Observation scale instructions need to be as efficient as possible while supporting reliable assessment. To gain insight in the type of instructions, visual or written, that promote reliability and efficiency of assessment, this study was performed. The study tested whether decision trees with visual components leading to scale anchors may be an efficient and reliable alternative to text scales. Our hypothesis was that in the condition where raters used a decision tree, interrater reliability and rater accuracy will be higher on average, and the
user experience will be more positive than in the condition where raters used a traditional written scale.
Material and Methods
Participants in the research were social workers working at Youth Protection Amsterdam. The research was conducted with two groups of participants (N= 217) randomly assigned to two research conditions, decision tree-based anchors and text-based anchors. Each participant in both groups observed six target videos and then rated parental sensitivity according to either the decision tree-anchor scale or the text-anchor scale. Also, they filled out a questionnaire about their user experience.
Results
Analyses showed that interrater reliability was good to excellent (.74 and .78) for both conditions and did not differ between conditions. Also, user experience was equally positive in both conditions. However, the text-anchor scale provided for a higher rater accuracy then the decision tree-anchor scale. In both conditions,
the mean absolute difference with the expert rating was higher when sensitivity was low.
Conclusions
The decision tree-anchor scale performed similarly to slightly worse than the original text anchor-scale. The finding that reliability was similar was promising in light of the potential for developing visual aids for scale instructions for an observational instrument suitable for use by practitioners. The text anchor-scale is a welldeveloped instrument, while the decision tree anchor-scale is the first adaptation to provide for an instrument social workers are able to use in daily practice.
Parental behavior plays an important role in the development of children. When working with at-risk families, professionals need to be able to assess behavior of parents with respect to their children. Information about parental behavior helps to be specific about the need of families, and, if necessary, to refer families to appropriate interventions. In current practice, social workers seldom use reliable and valid instruments to characterize parental behavior. Instruments should not only be reliable and valid, but also suitable for the constraints that
professional practice place on time needed for training to become reliable and time needed to conduct observations and assign observer rating scores. Observation scale instructions need to be as efficient as possible while supporting reliable assessment. To gain insight in the type of instructions, visual or written, that promote reliability and efficiency of assessment, this study was performed. The study tested whether decision trees with visual components leading to scale anchors may be an efficient and reliable alternative to text scales. Our hypothesis was that in the condition where raters used a decision tree, interrater reliability and rater accuracy will be higher on average, and the
user experience will be more positive than in the condition where raters used a traditional written scale.
Material and Methods
Participants in the research were social workers working at Youth Protection Amsterdam. The research was conducted with two groups of participants (N= 217) randomly assigned to two research conditions, decision tree-based anchors and text-based anchors. Each participant in both groups observed six target videos and then rated parental sensitivity according to either the decision tree-anchor scale or the text-anchor scale. Also, they filled out a questionnaire about their user experience.
Results
Analyses showed that interrater reliability was good to excellent (.74 and .78) for both conditions and did not differ between conditions. Also, user experience was equally positive in both conditions. However, the text-anchor scale provided for a higher rater accuracy then the decision tree-anchor scale. In both conditions,
the mean absolute difference with the expert rating was higher when sensitivity was low.
Conclusions
The decision tree-anchor scale performed similarly to slightly worse than the original text anchor-scale. The finding that reliability was similar was promising in light of the potential for developing visual aids for scale instructions for an observational instrument suitable for use by practitioners. The text anchor-scale is a welldeveloped instrument, while the decision tree anchor-scale is the first adaptation to provide for an instrument social workers are able to use in daily practice.
| Original language | English |
|---|---|
| Pages | 98-98 |
| Number of pages | 1 |
| Publication status | Published - 2 May 2018 |
| Event | EUSARF: All Children, All Families - Promoting Excellence in Child Welfare Research, Policy and Practice - Porto, Portugal Duration: 2 Oct 2018 → 5 Oct 2018 |
Conference
| Conference | EUSARF |
|---|---|
| Country/Territory | Portugal |
| City | Porto |
| Period | 2/10/18 → 5/10/18 |
