Paper Reading #20: The aligned rank transform for nonparametric factorial analyses using only anova procedures

The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures


Authors - Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins


Authors Bios - Jacob O. Wobbrock is an Associate Professor at the University of Washington and has a PhD from  Carnegie Mellon University.
Leah Findlater will be an Assistant Professor at the University of Maryland next year, has taught at the University of Washington, and has a PhD from the University of British Columbia.
Darren Gergle is an Associate Professor at Northwestern University and has a PhD from Carnegie Mellon University.
James J. Higgins is a Professor of Statistics at Kansas State University and has a PhD from the University of Missouri-Columbia.


Venue -  This paper was presented at the CHI '11 Proceedings of the 2011 annual conference on Human factors in computing systems.


Summary


Hypothesis - In this paper, researchers explain that common procedures used in analyzing non-parametric data tend to be error prone and propose a new system for analyzing data. This new procedure, called the Aligned Rank Transform (ART), allows researchers in the field of HCI to accurately analyze non-parametric data without fear of error and has easy to use tools to go with it. The hypothesis is that ART is a better way to analyze data than any other method currently in use in HCI and can be very useful in real world applications.


Content - ART corrects for requirements of ANOVA statistics that may not be true in HCI research such as normality. This is done by computing residuals, computing estimated effects for interactions, computing aligned responses, assigning average ranks, performing an ANOVA on this new data, and checking for correctness by checking sums.


Methods - The ARTool and ARWeb products make using ARTs easy and accessible. The ARTool parses long-format data and produces rankings based on this data. ARTool makes only 2  assumptions for use: The first column of data is an identifier and the last is a numeric response. Everything in between is assumed to be factors for the response. ARWeb performs exactly the same as ARTool just in a web setting that is more accessible to everyone with an internet connection. 


Results - The ARTool was used in a real-world case in evaluating results of a satisfaction survey given to users of a test interface. The results were analyzed using a standard ANOVA first and then an ART to see if any observable interaction concurred in the study. The ANOVA did not detect anything and noted their was no significant interaction but the ART found significant interaction and questioned the findings found earlier which agrees with the researchers initial perception that interaction was obviously present. Other examples were also tested with the ART and were well received as rules could be broken that traditional AVOVAs required.


Conclusion - The researchers conclude by stating that ART is useful when analyzing non-parametric data and has been useful in several cases that the researchers had worked on.


Discussion


I think the researchers achieve their goal of providing a better way to provide to analyze data for HCI research. This paper is interesting because it changes something that is already established to work with HCI.