Combining crowd consensus and user trustworthiness for managing collective tasks

In this paper, we present the LiquidCrowd methodological approach based on consensus and trustworthiness techniques for managing the execution of collective tasks. By collective task, we refer to a task in which the result depends on the consensus (i.e., degree of agreement) reached by the group of workers involved in the task execution. The more workers agree on a certain task answer, the more this answer is valid/accepted as a committed result for the task. In the paper, we provide crowdsourcing techniques for managing collective tasks based on the notion of supermajority to verify an expected degree of consensus/agreement required for considering a task as successfully completed. Moreover, we define trustworthiness techniques to measure the worker reliability/expertise in collective task execution in order to tailor the composition of a group by involving workers that are more able to express a sharable answer where needed. A case study of a collective task related to web-resource labeling is illustrated and evaluation results of the LiquidCrowd methodological approach are finally discussed.