Steven Dow, Markus Krause, Alvin Yuan, Sophie Vennix, and 2 more February 1, 2016
Expert feedback is valuable but hard to obtain for many de- signers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 facto- rial design. Student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated just as valuable as expert feedback. A follow-up analy- sis on writing style showed that student designers found feed- back most helpful when it was emotionally positive and spe- cific, and that providing a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other fa- vorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.
Download PDF151 Views
94 Downloads
Back to Files
CrowdCrit_CSCW_2016.pdf.txt
document
- 66.16 KB