Designing crowdsourcing tasks with consensus constraints

The notion of consensus is frequently employed in crowdsourcing applications/systems as a solution for quality assessment of task results, either based on an explicit or implicit notion of agreement among workers. In this paper, we describe the task-design activities in our crowdsourcing systems called Argo, where consensus/agreement parameters and constraints are specified and actually enforced. The configuration of system-level parameters is discussed and experience of task-design and execution on a real case-study of web-resource labeling is presented.