EyeRead
Human-centered Interaction Design | 09/2022-12/2022 |
One-semester Project of MSc Human-Computer Interaction at the University of St Andrews
Introduction
This project investigates how eye movement and gaze-oriented techniques can be used to enhance the traditional user experience of operating a mobile device in a particular environment. The specific use case we will discuss is the main functionality in a reading app. The main goal is to find new interaction methods that can intuitively implement this new technology in a reading app and how it can be combined with traditional interaction methods to achieve a complete workflow. Our final proposed system is EyeRead, a mobile e-reading app that can use these eye-movement and gaze techniques on their own or in combination with other modalities (e.g., touch, voice commands) or to implement innovative ideas such as eye-movement remote control and eye-dragging in an AR environment.
Background & Motivation
The Client Demands &User Needs
" Why we conduct the user research and the UCD design "
Our client, a PhD researcher, Lei, has successfully implemented a demo interface implementing some gaze-oriented features. However, he has yet to explore how to maximize the usability of this new input modality.
Currently, most operations on mobile devices, such as navigation and selection, are accomplished via touch controls. Users generally interact with their smartphones using a variety of these gestures, such as swiping, tapping, and scrolling. Recent developments in artificial intelligence and computer vision have facilitated the introduction of eye-tracking technology directly on these devices. This allows the front-facing camera to detect the precise location of the user's gaze and follow it in real-time.
This technology has the potential to serve as an additional input modality to smartphones, going beyond traditional touch gestures and allowing users to control their devices using gaze-oriented gestures.
User Research
Contextual Inquiry and Analysis
" What we should research "
" How we conduct the research "
This part outlines the process we adopted for usage research, the purpose of which was to understand the design needs of our users and client. This began with data elicitation, where we conducted research and gathered data about our users.
Initially Define Target Users
Initially, we identified the target audience of our application to be regular users of e-reading applications such as Amazon Kindle and Apple Books, as well as users of PDF readers such as Adobe Acrobat and web browsers such as Google Chrome. However, we did not specify precise target groups at this stage, as we would need more information regarding user work roles (at the later stage of data analysis) in order to gain a deeper understanding of the potential users of a gaze-oriented reading app.
“ While there may not be a group of people who directly use this gaze -oriented technology, we may choose to conduct research with similarly targeted users.
Similar target users are considered to be those who are interested in and have a large need to read, browse, or interact with content, as well as those who are not comfortable using their hands to control the mobile devices.
”
Main Challenges
Thus, the main challenge that we encountered is that since eye-tracking is an innovative and emerging technology, there is no similar existing system that is currently used in this context. Hence, there will inevitably be a significant learning cost when it comes to users adopting our proposed system as well as a clear bias towards applications users are already familiar with. This is something that we kept in mind throughout the design of contextual inquiry. We therefore decided to focus on identifying the users' needs without focusing on one particular technology. We planned to conduct observations in the users' reguld work context, that is, reading books electronically as they would normally do.
Moreover, due to constrained time and resources, we were limited in the amount and type of data that we would have liked to collect. We adopted an abridged process, restricting the amount of users that we interviewed and observed.
Design the Process of Contextual Inquiry:
User Data Collection
From the data elicitation process, we sought to obtain three main types of usage research data.
01
Interviews & Observations & Task Analysis
-
In-depth understanding of user's reading habits in traditional reading software.
What functions do they use ?
How do they use these features ?
-
Identify user pain points when using touch gestures.
-
Discover how could gaze gesture refine these shortcomings and make them more intuitive.
The first of comprised qualitative data is from which we could understand users" reading habits, including not only what features they use but also how they use them. Moreover, we intended to identify the users' pain points when using touch gestures, in order to later discover how gaze-gestures can optimise these flaws and make them more intuitive. To accomplish this, we planned to conduct interviews and observations, in which In our interviews we ask users a series of questions and then spend a few minutes observing them perform assigned reading tasks on a traditional mobile reading application and have them gauge their feelings about using the application to accomplish these tasks. From this process, we aimed to elicit concrete work activity notes that would serve as inputs for data models at a later stage.
02
Questionnaires & Scales
-
More Questions
-
More general participants
-
Supplementary with user interview
-
More comprehensive understanding of user habits
Secondly, we aimed to acquire quantitative data to supplement the work activity notes above, gaining a more comprehensive understanding of user habits thus providing further insights. Using a questionnaire, we could elicit numerical data and process it using SPSS. Then, in a small-scope late-funnel approach (Hartson & Pyla, 2012), we would then use the results to refine the models that we constructed, filling in the gaps that work activities did not cover.
Expert Advice & Academic Researches
-
More professional suggestions
-
Consider the bottlenecks and needs of technological development
03
Thirdly, due to a lack of user knowledge about eye-tracking, we concluded that interviewing and observing users alone would not be enough to extract the insights that we needed. Therefore, we spent time conducting research by reading papers on various eye-tracking studies that the client suggested and consulting the relevant specialist. We identified five main research papers and relevant sections within those papers which contained findings and background knowledge that would be valuable to us (Wilson & Williams, 2018; Rivu et al., 2019; Feng et al., 2021; Zhao et al., 2022). From those sections, we gathered work activity notes that we used as inputs for our data models.
Design the Process of Contextual Inquiry:
Preparation for User Data Collection
Once we had a general overview of our approach to contextual inquiry, it was time to set a concrete plan, outlining the parameters for our shudy.
Interviews & Observations & Task Analysis
Firstly, we prepared an interview and observation form in order to standardise the questions and setup across all interviews. It was critical to prepare a few relevant questions that would elicit a relatively brief amount of valuable data.
Next, we identified 10 participants, who we were fortunately able to recruit on short notice, who were regular users of e-reading applications. Due to differing schedules and restricted time, each of us arranged to conduct the interviews and observations separately with our participant in a procedure that took approximately 15 minutes. Prior to this, we prepared and adapted the ethics document, participant information sheet and consent form. The users were informed accordingly and consent was obtained beforehand. Unique IDs were assigned to each participant, ensuring that confidentiality and anonymity were maintained throughout.
Questionnaires & Scales
Next, we created a questionnaire that we distributed among our colleagues via Google Forms.
The Design of the Questionnaires & Scales
Three Main Parts
The first part is to collect basic personal information related to users and electronic reading. Before the investigation, we have provided the user's informed consent and ethical review signature form.
The last two parts are all five point rating scales, each consisting of different dimensions and to evaluate different functions, facilitating subsequent factor analysis to identify the most important functions.
Short questions to compare different aspects have also been arranged for each function, facilitating subsequent difference testing and analyzing user preferences and key differences.
1. User Personal Information
This section focuses on investigating users' reading habits and frequency, as well as their familiarity with electronic devices and understanding of new technologies, which can help us understand users' reading needs, behaviors, and goals, thereby supplementing our qualitative data and summarizing notes.
2. Usability of Exiting Features
The usability of a design depends on how well its features accommodate users' needs and contexts. This section investigates the ratings of six main functions (Turning, Directory Navigation, Highlight, Taking Notes, Customized adjustment, Language, and Compensation) in two different aspects (speed - accuracy, ease of use - comfort) .
3. User Work Practice
The application of User Work Practice in UX refers to designing products or services that meet user values and expectations by understanding their needs, pain points, behaviors, and motivations, thereby improving user satisfaction and loyalty. This section mainly focuses on understanding the user's behavior and motivation, as well as their needs and attitudes, from the three aspects of Work Habits/Activities, Feature Needs, and Future attitudes.
Analysis of Contextual Data :
Data Modelling of Qualitative Data
Following the contextual inquiry, data analysis was carried out to gather and systemize the data obtained and derive insights on which we can base our design decisions in the later stages of the project. We designed three main data models, each gaining a distinct perspective on user work practice. The inputs to these models included the work activities, the research conducted regarding eye-tracking, the initial project proposal, and the client brief.
Data Elicitation resulted in a vast amount of information from our interviews and observation, questionnaire, and research. We used a WAAD as a tool to bring order to this chaos, thus consolidating all this data.
Firstly, we each converted our hand-written notes to virtual sticky notes and color-coded these work activities accordingly (blue for research, yellow for user observations). Then, in an iterative process, they were organized into headings and sub-headings (red and orange respectively). In this way, we were able to gather knowledge regarding the inconveniences (or disadvantages) of common reading apps, which we can use as a foundation on which to improve the user experience, as well as the positive remarks users had about them. The final heading related to features that participants made use of the most and how these are utilized.
User Work Role Model
We then used this organised data to model the user work roles. From this model, we gained knowledge about the work roles of these users, and we were able to classify them into groups based on whether or not they would be potential users of a gaze-oriented reading app.
The questionnaire results were especially useful in this case to gain a vaster perspective of different user groups of traditional mobile e-book reading apps.
Flow Model
This model demonstrates the users' main interaction with their reading software, which are mainly reading-related tasks (indicated by light blue arrows). The users can also define words via an online search, export notes, and highlights externally, and share content with social media apps.
Analysis of Contextual Data :
Statistical Analysis of Quantitative Data
This section involves a quantitative study of data collected from scales and questionnaires to obtain comparisons and importance rankings among functions.
we noticed that navigating between chapters and highlighting text are the tasks with the highest rating of satisfaction, while taking notes is the largest pain point. From these insights, we can conclude which are the features that we can focus on enriching with gaze-oriented gestures.
Simultaneously, to investigate pain points related to note-taking functionality in conjunction with new technologies, we conducted a human factors statistical analysis on three different gesture input keyboards provided by experts. We found that the new projected keyboard did not meet our expectations. This suggests that innovative technologies may not always outperform traditional methods, emphasizing the need for further assessment and refinement when adopting novel input methods.
Questionnaire & Scales Analysis
Goals
1. Usability of Existing Features
-
Validating Structure and Separating Dimensions (Functions) Related to Overall Satisfaction, and Assessing the Relative Importance of Functions through Principal Component Analysis
-
Comparing Differences in Speed, Accuracy, and Overall Ratings of Four Key Functions
2. User Work Practice
-
Analyze users' work practices, usage habits, and needs.
-
Explore the potential of eye-tracking technology to introduce new ways of interaction.
-
Determine users' attitudes toward future technologies and functionalities.
Usability of Existing Features
Method
Principal Component Analysis was conducted on the scale to assess construct validity, reduce dimensionality, and separate the main components (dimensions) related to overall satisfaction from the predetermined four key functions.
Dimensionality reduction revealed four principal components, which are the main dimensions determining the overall satisfaction score.
Rotating the matrix to classify these four dimensions (distinguished by different colors), we can verify that these four principal components (Y1, Y2, Y3, Y4) are highly correlated with the four functions predefined in the scales.
Through principal component analysis, four important principal components have been identified, and their weights have been determined based on the eigenvalues of each principal component.
These weights indicate the contribution of each principal component to the overall satisfaction.
According to the calculations, it is evident that principal component Y1 has the highest weight (0.55), signifying its greatest impact on overall satisfaction. This is followed by Y2 (0.20), Y3 (0.13), and Y4 (0.11). Y1, Y2, Y3, and Y4 have been obtained as corresponding functions through the matrix mentioned above.
To assess satisfaction with different features, considering various dimensions.
Paired-sample t-tests can compare differences in speed, accuracy, and overall satisfaction across different features. However, to ensure reliability, Bonferroni correction is applied to statistically test the significance of results.
Another approach is Analysis of Variance (ANOVA), allowing simultaneous comparison of differences among multiple groups in speed, accuracy, and overall satisfaction without the need for multiple t-tests. Furthermore, Multivariate Analysis of Variance (MANOVA) is typically used to compare differences among multiple dependent variables while considering the impact of multiple independent variables.
Analysis Results:
1. Significant differences (p < .05) are observed in the dimensions of speed and accuracy between directory navigation and taking notes.
2. Notable differences in overall satisfaction are found among directory navigation, highlight, turning page, and taking notes.
These findings offer insights into the satisfaction variations among different features, highlighting lower satisfaction in the aspect of taking notes. This suggests a need for particular attention to improving note-taking functionality (importance ranked two) through the application of new technologies to enhance reading efficiency.
Feature Comparison:
Note-Taking Functionality Using Different Gesture Input Methods
Regular Keyboard
The "regular keyboard" is a common physical input device that features numerous keys, with each key representing a character or function.
Slide Keyboard
The "slide keyboard" is a physical input device that can be slid, allowing users to input more characters within a limited space.
Projected Keyboard
The "projected keyboard" is a virtual input device that projects the image of a keyboard onto a surface using laser or a projector. Users can input characters by touching / eye tracking the keys on the surface.
We further assisted experts in analyzing the performance of three input methods' keyboards in terms of input speed, error rate, and satisfaction.
The results revealed that the projected keyboard performed the worst, exhibiting the slowest input speed, the highest error rate, and the lowest preference. Conversely, the regular keyboard was the top performer with significantly lower error rates, faster input speed, and higher user preference.
The Analysis Report
User Work Practice
Table 3 reflects usage patterns and demands. In terms of the usability of existing features, we observed that satisfaction with the 'Highlight' function exceeds a score of 4. It's worth noting that when questioned about habits of using 'Highlight' under limited conditions, the score dropped to around 3. This suggests that even operations perceived as advantageous may become challenging due to posture constraints.
Another notable point is the emerging emphasis on habits related to multitasking and a reluctance to gaze at the screen for extended periods. While there is a potential risk of increased workload with eye tracking, as it requires prolonged screen fixation, simultaneously, eye tracking and gaze technologies offer significant potential to introduce a new sensory dimension for innovative operations, addressing the demands of multitasking.
Table 4 represents the direct assessment of requirements, and this ranking will serve as a direct guide for our design focus. In this section, we introduce additional features based on interviews. For instance, it should encompass a immersive technique that doesn’t disrupt others and avoids interrupting our thought process.
The Analysis Report
User
Story
from interviewees
Main Target Users
From the discussion of the user work role model and the questionnaire results, we further defined precise target users. The main target groups of users (in order of priority) are those who are physically impaired, the elderly, those who multi-task when reading, advanced users, and those who are in inconvenient situations such as only being able to use one hand.
1
physically impaired
2
the elderly
3
4
advanced users
5
multi-task when reading
those in inconvenient situations
Additional Insights
Limitations of Solely Using Eye-Tracking in Apps: High Learning Cost and Counter-Productivity
Optimizing User Experience: Integrating Gaze-Oriented Gestures with Multimodal Interaction
The usability of novel interaction methods, we found, may not always surpass traditional approaches.Implementing an app using eye-tracking alone would be undesirable and inaccessible. Mainly, this is because doing so would result in a learning cost that is too high for new users and this would ultimately be counter-productive.
Preserving Core Functionality: Essential Features in the Transition to EyeRead Reading App
The flow and task structure models indicate that it is crucial that all existing functionality of current reading apps should be maintained, with all basic features being implemented in EyeRead.
In order to increase usability and enhance the user experience, gaze-oriented gestures must be added as an additional option, combined with other modalities such as touch or voice, instead of simply replacing all or most existing gestures.
Addressing the Midas Touch Problem in Eye-Tracking Design for Enhanced User Experience
Moreover, our research made us aware of the Midas Touch Problem, a phenomenon in eye-tracking where users select items inadvertently by gazing upon them unintentionally (Wilson & Williams, 2018). In our design, we must strive to mitigate this problem, which would be detrimental to the user experience and deter users from adopting the technology.
Early Sketches Stage
Scenario Sketches
Early Sketches Stage
Concept Sketches
Concept 1 : "Conservative" Simple Gesture
Prototype Iteration Stage
Conservative Concept Prototype
Key Alternatives:
Balanced Ideas
The following section describes alternative gestures and features proposed during our ideation and prototyping phase that we later decided against pursuing. It describes their pitfalls and why these ideas were eventually abandoned.
Page-turning using Point-and-hold Gestures
The initial design of the page-turning method used point-and-hold gestures. However, turning pages is the most commonly used function when reading books, and the constant and repeated use of gazing and staring increases the user's eye fatigue. Therefore, we chose the swiping gesture that gives the least eye strain.
Point-and-hold animation countdown
The original plan for using point-and-hold gestures was to introduce a circle in the centre of the button and show a countdown of 3 seconds before completing an action. However, we determined that the colour-change approach could more clearly communicate to the user what the UI signifies. However, after a discussion with our customer, we learned that altering colours rather than numbers would be easier for the user to recognise. As a result, instead of numbers, point-and-hold gestures were utilised to modify the colour of the circular UI over time.
Blink Gestures
The advantage of the blink gesture is that it is simple, and there is a low learning cost for the user to execute the motion. However, since people blink and wink periodically, it is easy to cause the Midas Touch Problem. It was crucial for us to reduce the possibility of gesture misidentification. Therefore we proceeded with other gestures. In some of the designs, this is included as an alternative option in the settings menu, but it is not intended to be an integral feature.
Undo button
In the sketching phase, the idea of an "Undo" Button was explored as another measure to mitigate the Midas Touch Problem. This helps the user return to the interface before a misoperation. However, in the prototype design, we attempt to prevent this issue directly from the gesture design while still distinguishing the tasks of turning one page and switching Chapters on the interface. As a result, the function of the "Undo" button in the prototype design is heavily conflicting with the function of the "Back" button; thus, we cancelled it.
“Out-of-the-box” Concept Prototype
设计迭代