Paper Reading #10: Soylent: A Word Processor with a Crowd Inside

Comments
Stephen Morrow
Alyssa Nabors

Reference
Soylent: A Word Processor with a Crowd Inside
Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich
User Interface Software and Technology

Summary
Bernstein et al. talk about the Soylent add-on to Microsoft Word, which adds features to the program by means of crowdsourcing tasks. This add-on was implemented using Visual Studio Tools for Office, Windows Presentation Foundation, and TurKit Mechanical Turk toolkit. While Microsoft's spell checker has eliminated numerous errors when composing a document, it still generates a lot of false positives in spelling and misses a lot of grammatic errors. Thus Soylent adds the human component to the revising part of a document due to the fact that people often seek help from other people in complex cognition tasks. Soylent uses mechanical turks (people willing to perform tasks for money) to accomplish those tasks that AI is yet not able to perform.
The work of Bernstein et al. describes three features of Soylent: Text shortening, crowdproof, and a human macro interface where users can submit arbitrary word processing tasks. Their technique differs from traditional crowdsourcing in that they introduce a three step model into their implementations as opposed to the typical find and fix. The problem with the typical find and fix is that it increases the outliers in the type of Turks that perform work. The lazy Turk is the one that performs the minimum amount of work in order to get credit, whereas the eager beaver Turk is the one that generates too much feedback so it produces more work to the user. The authors were able to generate a more uniform type of worker by implementing the find-fix-verify approach.
In the find part of the request processing the users identify the problems with the provided section of the document. Then, a different group of mechanical Turks generate the possible solutions to the problem; and a third group verifies the solution and submits it back to the user. Typical response times ranged from a few minutes to hours, however the authors clearly specified that this timing would be reduced with the increased popularity of the system.
As a result of implementing this system, text shortening was able to reduce the text to 85% in one iteration without losing the meaning of the content. Crowdproof was able to catch 33 of the 49 errors introduced in the testing; successfully fixing 29 of the 33. The human macro part of the interface was equally successful in transmitting the intention from the user to the mechanical Turks.

Discussion
This was a somewhat interesting paper since it enables users to have their work peer reviewed. The authors touched on the implications of ownership and confidentiality, which I thought were very important subjects to be evaluated in the paper. Also, it would be interesting to see how much response times increase as Soylent becomes more popular