Paper Reading #5: A Framework for Robust and Flexible Handling of Inputs with Uncertainty

A Framework for Robust and Flexible Handling of Inputs with Uncertainty


Julia Schwarz, Scott Hudson, Jennifer Mankoff, Andrew D. Wilson


Julia Schwarz is a PhD student at Carnegie Mellon University studying human-computer interaction.
Scott Hudson is a professor at Carnegie Mellon University and earned a PhD from the University of Colorado.
Jennifer Mankoff is an associate professor at Carnegie Mellon University and has a PhD from the Georgia Institute of Technology.
Andrew D. Wilson is a researcher at Microsoft Research and has a PhD from MIT Media Laboratory.


This paper was presented at the UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.


Summary


In this paper, researchers describe how using simple GUI concepts are not good enough when dealing with touch input especially when attempting to predict what action is actually being performed. The hypothesis proposed is that a framework can be made, that flexibly handles input and treats it as another form of instruction, to improve accuracy among communicating what a user is trying to do and what action is actually being performed.


The framework developed by the researchers is primarily concerned with event dispatch, what item is an event being sent to and what action is being requested by the even, as it relates to uncertain input. Event dispatching is decided by a mediator which can execute the most probable action, take no action, or ask for more clarification from the user. 


A study of 6 example uses is presented next to validate that the framework works as desired. The examples show the simple application of the researchers' toolkit to common ambiguous inputs that touch interfaces are presented with such as uncertain item selection and uncertain action requests.


The researchers conclude that their toolkit can be easily applied to touch interfaces and will help as the industry continues to add support for mobile devices that often have small screens and small elements on screen.


Discussion


I think the researchers achieve their goal of supplying a better way to handle uncertain touch gestures but they fail to present us with any data on the subject such as how much of a problem this really is nor do they conduct a study to see what users would think of such improvements.


This paper is interesting because it attempts to provide a solution to a problem that has only recently become an issue due to the rise in mobile platforms. I think future work in this field is questionable due to the fact that interface design can fix many of these problems within individual apps. This research may have more impact in a more rounded environment such as a desktop OS that commonly has many applications running at a time within close proximity to each other but touch input is not a significant part of that market yet.

Paper Reading #4: Gestalt

Gestalt: Integrated Support for Implementation and Analysis in Machine Learning


Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko, James Landay


Kayur Patel is a PhD student at the University of Washington and focuses on research related to machine learning.
Naomi Bancroft is a recent graduate of the University of Washington and currently works for Google.
Steven M. Drucker is a Principal Researcher at Microsoft Research and also acts as an affiliate professor at the University of Washington.
James A. Fogarty is an assistant professor at the University of Washington.
Andrew J. Ko is an assistant professor at the University of Washington and has a PhD from Carnegie Mellon University.
James Landay is a professor at the University of Washington specializing in human-computer interaction and previously worked at Intel.


This paper was presented at the UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.


Summary


In this paper, the researchers discuss how machine learning can be used to improve the software space and create a tool that makes the task of implementing said methods into new software. Machine learning makes many things easier when programming but requires a different set of skill to do so (debugging vs. analyzing) leading to its slow adoption in industry. The hypothesis proposed is that if machine learning were made easier to adapt to problems, then programmers will use said technology to solve problems in better ways than ever.


Gestalt is the product being introduced in this paper as an IDE that allows developers to easily create machine learning applications using movie review analysis and pen-based gesture analysis as examples. The classification pipeline commonly found in machine learning programs was used as a basis for the design of the IDE. This pipeline consists of the information gathering, parsing, attribute generation, training, and testing steps.


Gestalt is designed with traditional IDEs in mind so as to make adoption easy for developers familiar with similar products, such as Eclipse, Microsoft Visual Studio, etc. The implementation perspective is identical to how other IDEs work in that a file can be opened, modified, and run from this perspective. The difference is that in addition to traditional programming support, all steps in the classification pipeline are made editable so that users can modify certain tasks to fit their needs. For each step in the pipeline, both an implementation and analysis perspective exists to allow developers to see what is being changed on the analysis side when implementation is changed.


Gestalt is different then what is already being used for machine learning implementation because similar programs often only support either implementation or analysis perspectives or contain both but does not give the developer enough flexibility to change attributes being observed from data. 


Testing for the effectiveness of Gestalt consisted of observing 8 participants debug machine learning applications in both a basline and Gestalt manner. The difference in bugs found was the variable being measured. Solutions to the problems were created beforehand and bugs were inserted for the participants to find. The results showed a significant increase in the number of bugs found using Gestalt than standard baseline debugging.


The researchers concluded that having an IDE like Gestalt, that able to implement a classification pipeline and analyze data as it moves through the pipeline, greatly contributes to the success of developers solving machine learning problems and improves upon what is already capable using the domain-specific products being used currently.

Presentation by the developers

Discussion


I think the researchers achieved what was being attempted and effectively showed that the product improved upon what developers can do when debugging their machine learning programs. With advancements like this making machine learning more practical, it is conceivable that machine learning will continue growing in popularity and begin to be used in industry as a solution not just an experiment. This paper is interesting because it tackles one of the most common problems found today which is how to make programming easier for the developer. Future work in this field is almost guaranteed and will be integral in finding user generated solutions in a user generated world.

Paper Reading #3: Pen + Touch = New Tools

Pen + Touch = New Tools


Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton


Ken Hinckley is a Principal Researcher at Microsoft Research and has a PhD from the University of Virginia.
Koji Yatani is a PhD student at the University of Toronto and will work at Microsoft Research after graduation.
Michel Pahud is a researcher at Microsoft Research and has a PhD from the Swiss Federal Institute of Technology.
Nicole Coddington is a Senior Interaction Designer at HTC and is a former employee of Microsoft.
Jenny Rodenhouse currently works in the Interactive Entertainment Division of Microsoft after attending school at Syracuse University and the University of Washington.
Andy Wilson is a researcher at Microsoft Research and has a PhD from MIT Media Laboratory.
Hrvoje Benko is a researcher at Microsoft Research and has a PhD from Columbia University.
Bill Buxton is a Principal Researcher at Microsoft Research and has an M.S. from the University of Toronto.


This paper was presented at the UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.

Summary


In this paper, researchers from Microsoft Research closely observe how people use pen and paper in order to create an interface that makes sense on a digital surface. Through this observation, the researchers hope to create an intuitive and simple way for interaction with digital surfaces in the future without having to train users how to use it. The hypothesis for this study is that through careful study, a very efficient and easy-to-use interface can be created for use on tablets, and other such devices, that makes sense for use by all.


The first project performed by the researchers consisted of a study of participants that were asked to make a scrapbook and given the tools to do so. The researchers notices 9 common attributes that can later be used when designing an application to perform the same task that the participants were asked to do.


They decided that the best approach to using pen and touch input would be to separate the two by having the pen write and touch manipulate. After implementing basic functions (zoom, cut, paste, etc.), the researchers began to explore what new tools they could implement by combining pen and touch input where it makes sense for a user to do so. Following this mindset, stapling, tearing, copying, and much more were implemented based on the earlier study gave the application a sense of robustness without feeling overly complicated and unfamiliar. 


Participants were then asked to use the new application to perform similar tasks as the first group did and give their feedback. Most users commented on the appealing nature of the interactions and responded well to the experience. The methods for accessing certain functions were not self revealing and required instruction which had been predicted by the researchers.


The researchers conclude that their concept of "the pen writes, touch manipulates, and pen + touch yields new tools" stands up to the test and can be adapted with further study and research but the ultimate answer will not be created by developers but rather defined by what people already do.


Discussion


I think the researchers achieved their goal by effectively showing that rather than creating new tools for people to use, we should study what is already being done and adapt to that standard to gain usefulness. This paper is interesting because rather than provide a direct product it produces the idea that developers should form around convention rather than creation. I think the weakest point in this work is that this study was performed on a very small scale with no more than 12 people during either of the studies. That being said, the researchers pulled an immense amount of data out of those small samples. I think the future will be paved with research like this given how popular tablets are becoming.

Paper Reading #2: Hands-On Math

Hands-On Math: A page-based multi-touch and pen desktop for technical work and problem solving


Robert Zeleznik, Andrew Bragdon, Ferdi Adeputra, and Hsu-Sheng Ko


Robert Zeleznik is the director of research at Brown University, specifically with the Computer Graphics Group.
Andrew Bragdon is a second year PhD student at Brown University focusing on human-computer interaction.
Ferdi Adeputra is currently attending Brown University.
Hsu-Sheng Ko is currently attending Brown University.


This paper was presented at the UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.

Summary


This paper documents how researchers attempted to combine the free-form features of physical paper with the problem solving ability of computers to form a system that allows for greater efficiency when solving problems. The hypothesis for this research is that if Computer Algebra Systems (CAS) were combined with a more familiar note-taking environment, people will work more efficiently. For the purposes of this research no more than high school level math was implemented.


The first element of the project discussed is the page management features including bezel gestures, a panning bar, and page folding. The bezel gestures are used to pan across the desktop as well as page creation and deletion. These actions are all performed at a low cost to users and are easily performed as only the number of fingers used is truly important. Although users can pan easily with one finger motions across the bezel, the researchers also added support for a panning bar that is displayed after a two-finger upward swipe from the bottom bezel. This bar displays all open documents in a top-down manner allowing for broad movements at once in stead of constant swiping from side to side. Page folding was added as a way to hide parts of a solution that have no reason to be visible any more keeping the screen uncluttered. To fold a page a user simply pinches the desired portion of the paper to hide and it is replaced with a small shadow indicating a fold. To unfold a page, the user taps the shadow previously created.


The next element explained by the researchers are the gestures recognized by the system to allow for interaction with pages and text on them. Under-the-rock menus allow for hidden operations to present themselves in a semi-opaque radial menu to users once an object is being moved by one finger. If a second contact is detected in the middle of the menu, it becomes active and can then be used to select an action to perform on the selected object. Touch-Activated Pen (TAP) Gestures allow for easy switching between ink mode and anything else a user may want to do. Gestures supported include making 2D selections, inserting space, and clipboard pasting. TAP Gestures are activated by drawing a specific symbol or shape with a pen and then selecting an option from a menu, that appears simultaneously, before removing the pen from the surface. Removing the pen from the surface or continuing to draw when these TAP Gesture menus appear will result in a continuation of ink mode and will draw like normal. The last two gestures provide faster selection of common commands that do not necessarily involve objects on the surface. PalmPrints allows a user to access common commands, like changing font, by placing a palm on the surface and then selecting an action from a menu that appears beneath the fingers. FingerPose allows multiple actions to be stored for similar input because a single finger gesture with a more vertical pose can be accurately differentiated from a horizontal pose with the same finger. This allows for a user to either pan the desktop or move an object with the same finger, just in different poses.


The last element of this system is the math functionality provided to users that use the system after pages and entries have been made. By using a series of pinch, drag, and stretch gestures on specific parts of an equation, certain operations will be performed with the results displaying directly below the affected equation actively showing significant changes made by the action.


The system was tested on a group of 9 participants. Each participant was asked to perform a series of tasks and provided help when needed, although verbal help proved rather ineffective when compared to hands-on demonstrations. The conclusion of the participants was overwhelmingly positive when considering the potential but the need to simplify gestures and shrink the form factor were made apparent. Overall the system was looked upon with success, as their tests did not contradict their hypothesis, and the researchers discussed the feasibility and usefulness of integrating more complex math as the system grows.


Discussion


I think this paper is interesting as it discusses the merging of "old ways" with "new ways" and presents a very plausible way of doing it. I do not think the researchers proved their hypothesis, they just did not disprove it, which seems like a short coming. The implementation seemed to be very creative and intelligently designed but the testing lacked in tutorial material for testers resulting in confusion. The commands also seemed to be memory driven in some ways which decreases its immediate accessibility although, in the long run this would become irrelevant as people would learn it if they wanted to use it. I think ideas like this will lead to more advancements in the big picture than altogether new products because this research combines what people already do with what computers already do.

Paper Reading #1: Imaginary Interfaces, Spatial Interaction with Empty Hands and without Visual Feedback

Imaginary Interfaces, Spatial Interaction with Empty Hands and without Visual Feedback


Sean Gustafson, Daniel Bierwirth and Patrick Baudisch


Sean Gustafson is a PhD student focusing on Human Computer Interaction (HCI) at the Hasso Plattner Institute in Germany.
Daniel Bierwirth is a cofounder of Matt Halting & Company UG and the Agentur Richard GbR and has a master's degree from Hasso Plattner Institute.
Patrick Baudisch is a professor at Hasso Plattner Institute and is the head of the HCI Lab.


This paper was presented at the UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology.


Summary


This paper discusses the researchers' attempts to remove the need for visual feedback when a person is communicating a simple idea that can be shown by a simple drawing or performing an action with a series of movements.The main goal of this experiment was to evaluate the plausibility of creating a system that relies on a user's short term memory instead of providing any feedback that the user can respond to. The hypothesis for this experiment is that a person's memory can replace the need for a system to provide the user with any display or results and still support many basic interactions.


The first experiment focused on testing a user's ability to draw on a 2D imaginary plane and recall what was drawn earlier to create more complex shapes and add to what is present. The test consisted of 3 tasks.First was graffiti drawings checking for correct shapes and letters, next was repeated drawings that measured the differences in the same shape if drawn over each other right away, and last was multi-stroke drawing which combined shapes together making them more complex. The results of these tests supported the hypothesis that a user's memory can serve as enough feedback to create legible shapes because all of the users correctly performed the tasks with the multi-stroke drawings contributing the greatest amount of error while still being acceptable.


The second experiment focused more on testing a user's memory by requiring them to draw a shape, rotate, and point to a specified vertex on the shape they had drawn. The results found that when a user rotated there was a statistically significant increase in error when compared to the user standing still the entire time. The researchers also performed a follow up test that required a user to leave their left hand in front of their face in the form of an L. This resulted in significantly less error due to the use of the left hand as a reference point for later interactions


The final experiment tested a user's ability to work in a plane that they had no prior contact with. This consisted of using a user's left hand as the base for a grid using the intersection of the L as location (0,0), the tip of the index finger as location (1,0), and the tip of the thumb as position (0,1). Participants were then asked to point to various random points on the grid using their left hand as a reference. The results showed, as predicted, that the points furthest away from the left hand provided the most difficulty although some users exhibited better results than others leading the researchers to conclude that this task could be improved through training.


The results of the experiments helped to provide the researchers with more data to consider when continuing development such as support for a reference hand which would greatly reduce error. The researchers conclude the paper by discussing the plausibility of creating an Imaginary Interfaces supported device and provide a few examples of applications for the new technology such as communicating simple sketched quickly without system feedback to the user.


Discussion

Microsoft Kinect

I think the researchers achieved their goal of proving a screen, or any form of feedback for that matter, is not necessary for interaction with computers. They clearly showed that people are capable of remembering important information when interacting with something that is not present in the physical.


The closest example of this technology in the market today is Microsoft's Kinect peripheral for their XBOX gaming console. In many cases, although reactions occur on screen, users are still interacting with an interface that is completely imaginary like simulating a steering wheel even while only viewing the affected car externally.


The research presented in this paper introduces the concept of removing the interaction, as we know it, with computers and in its place allow the human's mind to take on new responsibilities. I think this concept is an interesting one that could lead to many applications that only need user input and can be performed very quickly with no obvious computer interaction. For example a button that a person presses everyday and receives no feedback from could be replaced with this technology like a garage door opener or any other unlocking mechanism for that matter that does not tell the user anything about what is occurring-it can only be evaluated by observing that the action has occurred which, in this case, serves as enough feedback given the input (button press).


A fault of this research lies only in its uniqueness as I do not think a need for this technology is satisfactorily established and it may take more experimental research before an endgame can be determined.  

Blog #0: On Computers

          Aristotle defines the soul as the purpose for a living being's existence. By this standard Aristotle stated that plants existed to grow and reproduce, animals added the ability to sense and move, and humans additionally can reason. Keeping these ideas in mind I will comment on the discussion questioning whether computers have souls. If it can be determined that computers do indeed possess souls then, the power to create souls is clearly within the realm of humanity, a fact that could have significant impact on modern philosophy and ethics.

         

          As stated above I will be using Aristotle's thoughts on what a soul is but the question remains of what defines a computer. It can easily be seen that the term computer has changed meanings through the centuries evolving from ancient calculators to machines that are responsible for life critical functions. I will use a more general definition of computers as their current iteration with processing power, dedicated resources, programmable instructions, and almost any hardware peripherals imaginable, e.g. robots.

          If we loosely think about Aristotle’s definition of a soul and apply it to a modern day factory that produces computers on an assembly line, we see that all “requirements” are met for having all levels of a soul. The assembly line’s purpose is to reproduce computers, the robots are capable of sensing and moving, and throughout the process defective units are determined and disposed of (reasoning), thus satisfying everything Aristotle mentions to some effect. The difference here is that all of these requirements were met because humans created the assembly line and instructed all of its capabilities. From here, a whole new line of questioning arises and at least 2 viewpoints can be seen.

          The first viewpoint thinks of humans as nothing more than advanced computers that have evolved over time into very complex machines that still to this day take input, process it, and produce a result in the form of movement, speech, etc. This viewpoint has no problem accepting computers as possessing infant souls much like humans began. This is further supported by the idea that if humans can create souls maybe something else did likewise to get life as we know it. The second viewpoint holds plant, animal, and human life in a different regard completely than computers and is unable to accept that humans are capable of creating souls ruling the conversation moot.

          Overall, I think the conversation is just getting started on this topic and questions of ethics will begin appearing in the future development of technology as we venture into the unknown.

Introduction Blog Assignment #-1

Andy Hampton

email: ahampt@tamu.edu

• 3rd year Senior
• I am taking this class to learn how to effectively communicate with users of my future products.
• I just started thinking about CHI principles this summer as I worked on a project as an intern with Amazon.com.
• In 10 years I expect to be a software engineer climbing the ladder.
• I think the next technological advancement in computer science will be the integration of computer and human kind that will allow for near instantaneous action and reaction.
• If I could travel back in time, I would like to meet the Wright Brothers and experience what it was like to push the boundaries of possibility.
• My favorite shoes are ones that fit and don't require any effort on my part to walk in.
• I would be fluent in a Native American language because I think it's interesting to see how their language often reflects their culture.
• I am an officer in the Flying Aggies and have had my pilot's license for 3 years.