Abstract
In the present study, aspects of the measurement of writing are disentangled in order to investigate the validity of inferences made on the basis of writing performance and to describe implications for the assessment of writing. To include genre as a facet in the measurement, we obtained writing scores of 12 texts in four different genres for each participating student. Results indicate that across raters, tasks and genres, only 10% of the variance in writing scores is related to individual writing skill. In order to draw conclusions about writing proficiency, students should therefore write at least three different texts in each of four genres rated by at least two raters. Moreover, when writing scores are obtained through highly similar tasks, generalization across genres is not warranted. Inferences based on text quality scores should, in this case, be limited to genre-specific writing. These findings replicate the large task variance in writing assessment as consistently found in earlier research and emphasize the effect of genre on the generalizability of writing scores. This research has important implications for writing research and writing education, in which writing proficiency is quite often assessed by only one task rated by one rater.
| Original language | English |
|---|---|
| Pages (from-to) | 83-100 |
| Number of pages | 18 |
| Journal | Language Testing |
| Volume | 32 |
| Issue number | 1 |
| DOIs | |
| Publication status | Published - 1 Jan 2015 |
Funding
Table A1. Writing tasks that are used in this study: three topics in four different genres, categorized according to purpose of writing and audience specification. Purpose Audience Specified reader Unspecified reader Argumentative writing Persuasive letters for a fictional companyTask 1: Collection of toys at a supermarketTask 2: Collection of stamps at a petrol station for earning musical ticketsTask 3: Collection of points on wraps of chocolate bars to earn a music CD Argumentative essays to prepare oneself for a class discussion Task 1: Pros and cons of a candy prohibition for children Task 2: Pros and cons of a smoking ban Task 3: Pros and cons of telling tales about somebody Narrative writing Adventure stories for readers of a school newspaper Personal stories Task 1: Adventure on a sports field Task 1: Personal experience about being frightened by something Task 2: Adventure about a forest fire Task 2: Personal experience about being caught for something Task 3: Adventure about poison Task 3: Personal experience about being home alone Funding This research has been supported by the Netherlands Organization for Scientific Research (NWO), grant 411-11-859 to Huub van den Bergh. We are grateful to Theo Pullens for providing the data.
Keywords
- Generalizability theory
- genre effect
- rater effect
- task effect
- writing assessment
