Paper Reading #6: TurKit

TurKit: Human Computation Algorithms on Mechanical Turk


Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller


Greg Little is a PhD student at MIT working in the User Interface Design group.
Lydia Chilton is a graduate student at the University of Washington and attended MIT prior to that.
Max Goldman is a graduate student at MIT focusing on user interfaces and software development.
Rob Miller is an associate professor at MIT and leads the User Interface Design Group there.


This paper was presented at the UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.


Summary


In this paper, researchers propose a way to improve upon the Mechanical Turk application used for human computation by providing a toolkit that provides a framework upon which to build more complex algorithms. The hypothesis is that TurKit will make human computation more effective than it currently is by enabling complex algorithms.


The first item presented by the researchers is that TurKit Script which extends javascript's ability to communicate with MTurk. TurKit Script adds support for the functions waitForHIT, prompt, vote, and sort as well as fork and join to support concurrency. 


The next thing discussed is crash-and-rerun programming which TurKit Script is based on. Crash-and-rerun programming involves recording all successfully executed lines of code, crashing, and rerunning all recorded lines. TurKit Script had to adjust this to fit their model due to some of the functions being called costing actual money so they developed a once function that tells the script to look for an output to this line before running it to see if it had already been run. 


TurKit was written in Java and used Rhino and E4X for additional support. Implementation of the once function took the most work because a persistent data store had to be created to hold the values after the line is executed so that on the next run it can be retrieved without making unnecessary calls.


The researchers provide examples of where TurKit has been used to great affect and shows what cannot currently be done in standard MTurk. These include iterative writing, text recognition, and decision theory experiments. All of these examples take advantage of the crash-and-rerun nature of TurKit and allow users to build off of other users' answers.


The paper concludes by noting the many uses of TurKit that have already been put into practice as well as note new ways it can be used like for experimental replication or just exploring new algorithms using the crash-and-rerun framework.


Discussion


I think the researchers did not necessarily prove their hypothesis with this paper alone however, I feel that they did still convince me of the success of the project by the examples of people already using TurKit. They explored a new territory in human computation by supporting iterative improvement allowing users to build off of other users to get maximum feedback.


I think this paper was interesting because of the crash-and-rerun programming that they decided to use in TurKit. This approach supports the claim that the simplest solution is usually good enough although not optimal. Recording line executions that succeed is very smart but also very costly especially when trying to scale so there is a very niche audience that I think this is aimed and they better be willing to make some sacrifice. But as mentioned above, people are willing to make these sacrifices and are already using it so it sounds like a success.