Paper Reading #21: Human model evaluation in interactive supervised learning

Human Model Evaluation in Interactive Supervised Learning


Authors - Rebecca Fiebrink, Perry R. Cook, and Daniel Truema


Authors Bios - Rebecca Fiebrink is an assistant professor at Princeton University in the department of computer science and music and has a PhD from Princeton.
Perry R. Cook is a professor emeritus at Princeton University and has a PhD from Stanford.
Daniel Truema teaches various music courses at Princeton University and is an accomplished composer and performer on the fiddle and laptop.


Venue - This paper was presented at the CHI '11 Proceedings of the 2011 annual conference on Human factors in computing systems.


Summary


Hypothesis - In this paper, researchers explain that machine learning can be a powerful tool in processing large amounts of data and generating output on actionable items but the manner in which these systems learns if often static providing little to no feedback to the user performing the training. The researchers propose a system that allows trainers to supervise machine learning and provide valuable information regarding why certain output is generated and suggesting ways to fix problems. The hypothesis is that a system such as this can give users more of the information they want and help them build better machine learning applications.


Content - The researchers develop a generic tool, called the Wekinator, that implements basic elements of supervised learning in a machine learning environment to recognize physical gestures and label them as a certain input. They chose this application because gesture modeling is one of the most common uses of machine learning and music naturally is gesture driven at times like recognizing a certain gesture as a certain pitch.


Methods - 3 studies were conducted in evaluating the Wekinator:
A) The first study consisted of 7 composers working to refine the Wekinator to control new instruments that existed on the computer only and responded to gesture input. The participants trained the system once a week and made suggestions that were acted upon in between sessions.
B) The second study focused on the supervised learning aspect of the system and observed 21 students in their use of the Wekinator to produce new instruments controlled by certain gestures (one continuously controlled adjusting to changes in gesture in real-time). The students' actions were recorded by the software and they filled out questionnaires.
C) The final study consisted of a professional cellist working with the researchers to classify several gestures that capture many properties of a bow as it is used to play the cello. After the system captures this data, it should be able to capture the notes being played and add them to a composition on a computer.


Results - The studies found that most of the participants chose to train the system by editing the training data. In studies B and C, the researchers found that the participants used cross validation on occasion to observe how accurate their systems were performing and evaluating new ones at the same time. In contrast direct evaluation was used more frequently in all 3 studies and allowed for quick validation that a system was performing as expected. Subjective evaluation allowed the cellist in study C to fix mistakes made in training not caught in cross validation. Many participants noted that they got better in providing training data as the study went on indicating success in a primary goal of the system. At the conclusion of all the studies, most participants praised the Wekinator as working extremely well in accomplishing the tasks they had wanted.


Conclusion - The researchers conclude by saying that supervised learning has a place in machine learning but users must evaluate what qualities they are looking for beforehand as some things are not yet handled in a particularly successful fashion like accuracy and should not be relied on.


Discussion


I think the researchers achieve their goal of proving that supervised learning benefits the users in building a better model. This is interesting because machine learning is still in a young stage of life and studies like this make adoption easier and more practical than ever before.