Abstract
The project aims to develop a prototype tool that utilizes sentiment analysis from existing machine learning (ML) models to extract emotional information and visualize the results. The primary objective is to create a user-friendly system. This system not only presents the extracted emotional data alongside ML results but also assists users in collecting emotion-related information and visualisation. By providing concrete visual representations, the tool enables a better understanding of the abstract world of emotions. This, in turn, helps healthcare professionals comprehend the emotional states of their patients and aids family members and friends in understanding others ’ emotions. Ultimately, the tool facilitates improved treatment and support for individuals to understand emotions.
A Prototype Tool for Affective Computing
and Emotion Visualization
Jun - Aug 2023
Visualization System
Dissertation Project of MSc Human-Computer Interaction
Project Supervisors: Dr. Angela Miguel & Dr. Ognjen Arandelovic
School of Computer Science
University of St Andrews
15th of August 2023
In summary, the project aims to develop a prototype system capable of collecting user data, extracting underlying emotional information, and providing output and visualization tools for relevant stakeholders. It is important to note that the system will not involve the collection of real users' private data due to ethical issues but will serve as a prototype for demonstration purposes.
Overall System Architecture
Dynamic Workflow:
Real-time Reception and Visualization
When users record text and time information on the diary page, the back-end performs emotion analysis and writes the results to the JSON database. Consequently, the data received by the front-end changes in real-time based on user input, enabling dynamic visualization adjustments.
Contributions
In terms of practical application contributions, this project has developed a sophisticated tool aimed at facilitating the acquisition of textual data, enabling users to explore the underlying emotional nuances embedded within the information. Through the utilization of tangible visualization representations and interactive techniques, individuals are empowered to gain deeper insights into the abstract realm of emotions. This tool effectively mitigates the shortcomings of existing emotional visualization systems by introducing a comprehensive workflow and a seamlessly integrated, human-centric dynamic emotional analysis system. Moreover, it places emphasis on empathetic design and end-user creativity, thereby enhancing emotional interaction experiences. Theoretically, the tool introduces innovative attributes to raw emotional data, enabling the visualization of implicit emotional patterns. Furthermore, it enhances the interpretability of machine learning outcomes, providing users with a more intuitive understanding.
Evaluation
Due to ethical constraints, we were unable to secure approval for evaluating the visualization system. Therefore, future work will continue to focus on enhancing the system based on user needs and experiences. This discussion merely presents selected personal findings and insights to summarize this work.
- Interesting Findings and Thinking -
"Complex emotions and elusive machine computations"
5.29：
威廉，没有了爱，世界对我们的心又算得了什么呢？就像没有了光，阿拉丁神灯也毫无意义。只需点燃心中的火焰，最绚烂的图像就会在白墙上闪现；即使爱只给我们展示了瞬息的影子，当我们像孩子般欣赏它们，被那灿烂的幻影感动时，我们仍然是幸福的。今天我没有见到夏洛特，因为有些客人来访，我无法摆脱。我派了仆人去她家，至少能见到一位曾近距离接触过她的人。等待他回来的时候，我是多么焦急啊！当他回来时，我是多么欣喜啊！如果不是因为感到羞愧，我一定会抱住他亲吻他。有人说，把波诺纳石放在阳光下，它会吸引光线，在黑暗中发出微光。我和这个仆人的关系就像这样。想到夏洛特曾经凝视过他的容颜、他的面颊、他的衣着，使我对他无比珍视，以至于那时我愿意用千金也不愿与他分开。他的出现让我如此快乐！威廉，请别笑话我。让我们快乐的东西难道会是幻觉吗？
Chinese Version
One example diary entry
This example diary entry was simulated from a novel (https://www.gutenberg.org/files/2527/2527-h/2527-h.htm ), with the results from two sentiment analysis libraries shown on the right. Surprisingly, they yielded entirely opposite outcomes. This underscores the subjectivity and complexity of emotions, making sentiment analysis a challenging task, particularly with complex, ambiguous, or ironic texts that may elude precise algorithmic capture. Different sentiment analysis libraries may employ various algorithms and methods. Some might focus on syntactic structures, while others emphasize semantic and contextual understanding.
These differences can result in disparate interpretations of the same text, possibly reflecting issues with the reliability and accuracy of certain libraries. Future research in this field may benefit from evaluating these libraries on annotated test data to further understand their performance. While the current prototype design has considered inputting emotional labels and data through The Empathy Box interface, it failed to store real emotional data from users to compare and align with machine learning outcomes.
Therefore, future system development can continue to focus on building a more comprehensive and dynamic visualization workflow, taking into account the collection of multidimensional emotional data from users. This can provide more real datasets and insights for building sentiment analysis libraries in machine learning.
0 1
Immersive VR & Emotion Metaverse
To play, press and hold the enter key. To stop, release the enter key.
Enhance Emotional Visualization
Virtual Reality (VR) technology empowers emotional visualization by immersing users in virtual environments, enabling more intuitive and immersive interaction with data and information. Through auditory, visual, and interactive elements, VR evokes emotional responses, aiding users in understanding and managing emotions like anxiety and depression. This technology holds great potential for enhancing user experience and emotional connections.
Virtual Counseling with LLMs &
Immersive Experience
0 2
The Concept 3D model was created by three.js
Using Three.js to construct cozy room settings and figurines, simulating a counseling environment, offers users an interactive experience, aiding in their comprehension of counseling processes and environments. Such virtual settings provide a safe, controlled environment for mental health education, training, and therapy. Additionally, users can explore and practice skills and coping strategies without constraints. Compared to solely writing diaries, recording thoughts and viewpoints through dialogues allows for more natural expression of emotions and immediate feedback and support, fostering deeper understanding of emotions and issues, potentially guiding users to consider more solutions. In summary, this idea promotes awareness and practice of mental health, enhancing understanding and acceptance of counseling.
LLMs such as GPT-3 can be utilized to create virtual counselors. With robust natural language understanding and generation capabilities, LLMs can simulate human conversations and provide corresponding responses and suggestions based on user input. Therefore, LLMs are an excellent choice for constructing virtual counselors, offering personalized counseling services.
Empowering Authentic
Expression
- Dialog & Diary -
0 3
Generating Flora Based on Emotions
Creative Emotionscapes
Combining Psychological Landscapes with Sandplay Theory: Generating Corresponding Flowers, Plants, and Trees Based on Emotion Type and Intensity.
0 4
Multimodal Health Visualization
Enrich Data & Visualization
Multimodal mental health data visualization integrates multiple dimensions of data, including linguistic, visual, spatial, gesture, and aural aspects, providing a richer and more comprehensive representation compared to unimodal sentiment analysis. This holistic approach captures a broader range of individual psychological states and emotional expressions. For instance, alongside text sentiment analysis, facial expressions, body language, and vocal tone can be captured through images, videos, and sound analysis, respectively. Such integrated analyses offer a more accurate reflection of an individual's mental state and behavioral characteristics, empowering healthcare professionals and researchers with deeper, more comprehensive data insights to better understand an individual's mental health status and devise more effective treatment strategies.
0 5
The Web Application
& Mobile App
This product is a platform designed for mental health data visualization, available both as a web application and a mobile app. It allows users to input various types and sources of data, such as text, images, and audio, into the system. The data is then analyzed and processed using machine learning and deep learning techniques. Users can view the analysis results through the web application or mobile app, presented in visual formats such as charts, graphs, and heatmaps. The primary goal of this product is to help individuals better understand and manage their mental health. Additionally, it provides valuable data support for professionals in fields such as diagnosis, treatment, and research.
This platform not only assists users in visualizing personal mental health data but also fosters the creation and exchange of emotional content. Users can explore their emotional states by inputting various forms of data and visualize them for better understanding and management of their mental well-being. Additionally, they can express emotions, share experiences, and establish connections with others through the platform, gaining support and inspiration in the process. This creative exchange not only enhances individual creativity and self-expression but also strengthens community cohesion and emotional resonance.
Personal
Visualization
The Audience
The end-users comprise individuals expressing their emotions and various stakeholders, including therapists, family members, and friends, seeking to observe and comprehend these emotional states. This diverse user base constitutes the primary audience for the visualization system. Within the system’s front-end, two interfaces facilitate bidirectional interaction with users through 3D interactive visualization, fostering enhanced creativity and real-time emotional representation. Among the remaining interfaces, the diary entry interface caters to users providing input, while the timeline visualization, implemented using D3.js, serves as the principal tool for objective and professional emotion analysis, tailored to specialized audiences such as medical professionals and mental health counselors.
The System
The entire system is composed of three main components: the front-end, the back-end, and JSON databases. As illustrated in the diagram, the complete system architecture and workflow include the end-user, four front-end interfaces, a back-end interface for sentiment analysis, and JSON databases that consolidates and stores both the original front-end input and the analyzed results from the back-end.
Dynamic Workflow: Real-time Reception and Visualization
Once the front-end, back-end, and database were constructed, the fundamental framework of this dynamic visualization system/tool was established. When users record text and time information on the diary page, the back-end performs emotion analysis and writes the results to the JSON database. Consequently, the data received by the front-end changes in real-time based on user input, enabling dynamic visualization adjustments.
The Front End
The front-end mainly handles user input, sends data to the back-end, receives JSON data, and presents emotion visualization.
1. Collect and Send Input Data
The front-end code snippet provided captures user input for text and time through a form. When the form is submitted, it checks if both text and time are provided. If they are, it sends a POST request to the back-end at 'http://127.0.0.1:8000/analyze', with the text and time encapsulated as JSON. In the actual construction of the tool, we used a novel to simulate the input of 45 days' worth of diary entries .
2. Fetch JSON Data
The front-end and is responsible for fetching and processing sentiment data from a backend server and creating visualizations. The fetchData function in the front-end code fetches sentiment analysis results from a specified URL. It processes the received data by mapping it to a new format that includes date, negative, neutral, positive, and compound sentiment values, and emotion labels.
Exampe codes to fecth data in the front end
The example diary to simulate the real user input (https://www.gutenberg.org/files/2527/2527-h/2527-h.htm)
3. Creat Visualisations
D3.js : main visualisation on sentiment analysis
Upon receiving the JSON data in the front end, the D3.js library (https://d3js.org/) is utilized to create interactive charts. D3.js is a renowned JavaScript library that allows for manipulating documents based on data. It enables the binding of data to a Document Object Model (DOM) and the application of data-driven transformations to the document. Using D3.js, the complex data fetched from the sentiment analysis can be visually represented through dynamic and interactive graphical forms, thereby enhancing user engagement and comprehension of the underlying patterns and trends.
three.js : 3D interactive viusalisation on instant input
Three.js (https://threejs.org/) is a popular JavaScript library that facilitates the creation of 3D graphics on web browsers. By utilizing WebGL, three.js makes it possible to render complex 3D objects, scenes, and animations directly within the browser, without the need for specialized software or plugins.
For the purposes of this project, three.js offers the opportunity to build more empathetic physical visualizations and artistic expressions. The ability to create immersive 3D spaces and elements can lead to more engaging and intuitive understanding of the data, allowing users to explore and interact with information in a way that transcends traditional two-dimensional presentations. By adding depth and dimension, three.js can breathe life into data, turning numbers and statistics into tangible and relatable visual experiences.
D3.js:
main visualization on sentiment analysis
three.js:
3D interactive visualization on instant input
Design Inspirations
Visualization Prototypes
Various fundamental diagrams (such as river stream) and researches have been introduced, serving as strong guidance and inspiration.
Starting with initial brainstorming using the Five Design Sheet method, we incorporated findings from VADER and Tweetnlp machine learning libraries to visually represent emotions' types and intensity, following Plutchik's wheel of emotions model for the color scheme. The visualization of sentiment analysis involves two main aspects: emotions on a timeline and emotions within the text. We filtered and categorized brainstormed ideas into five visualization designs: Cross time, Diary visualization, Emotion Categories, 3D projection, and Dynamic arts. Prioritizing Ben Shneiderman's visual information seeking mantra, the timeline visualization stands out as crucial, offering an overview of emotions' evolution over time.
Plutchik's emotion model utilizes a color wheel to effectively illustrate emotions' similarities and differences:
1. Color Encoding: Each basic emotion corresponds to a specific color, with adjacent colors indicating similar emotions and opposing colors denoting contrasting emotions.
2. Saturation Encoding: Plutchik employs saturation to indicate the intensity of emotions, with more saturated colors indicating stronger emotions and less saturated colors representing weaker emotions. This intuitive approach aids in understanding emotions' strength or weakness.
The Evolution of Visualization
To enhance our visualization capabilities, I integrated timeline emotion visualization as the main overview, and incorporated additional zoom and filter functions for finer detail. This integration required linking detailed text information with the timeline visualization window and implementing timeline-driven filters.
Among these five sketching categories, the most crucial is the visualization related to the timeline, as according to Ben Shneiderman's (1996) visual information seeking mantra to design information visualization systems, the approach should be "Overview first." Thus, the visualization of emotions over time allows for a macro perspective on the overall view of emotions.
After critical discussions, more brainstormed ideas were further culled and sifted. The reasons for elimination were summarized as follows:
Limit the color palette to seven hues at most to avoid confusion caused by excessive color variations, which can hinder efficiency.
Color
Maintain the timeline’s linear format and refrain from transforming it into circular or other shapes. Circular layouts can impede comprehension, leading to visual congestion and misinterpretation of shape, color, and position due to inconsistent inner radii.
Shape
When using color to represent emotional information, ensure clarity and avoid ambiguity. For instance, employing 3D heat maps may mislead viewers into perceiving differently sized color areas based on elevation, potentially inducing cognitive bias.
Cognitive Bias
Sandplay Therapy, a non-verbal form of psychotherapy, is often utilized in mental treatment and personal growth processes (Kalff, 1980). This therapy has been widely applied in the treatment of mental health ( Bradway & McCoard, 1997; Turner, 2005). In Sandplay Therapy, individuals are guided to construct a three-dimensional "world" in a tray filled with wet or dry sand, using miniature figures, stones, shells, and other objects. The therapist observes without intervention and then explores the meaning of this world with the individual. This process can reveal the patient's inner conflicts and desires, providing a resolution in a relatively safe environment.
Sandplay Therapy
In the background introduction of the project, the primary intention of emotional computing and visualization is to aid the field of mental health in understanding and monitoring emotional information, especially for those grappling with emotional disorders such as depression and anxiety, who need particular attention to their emotions. Therefore, in addition to considering a more empathetic design in visualization, it is also vital to consider the underlying diagnostic and therapeutic significance in the long term.
To play, press and hold the enter key. To stop, release the enter key.
Emotion Physicalization.
Note: All visualization schemes for emotions to colors in this project are based on Plutchik's (2001) wheel of emotions model.
The row of flowers maps to positive emotions, which is Love, Joy, Optimism, Anticipation; the second elements of grass represents the neutral emotion of Trust, with sharp branches symbolizing Fear; the third elements of crystals corresponds to Angry, Disgust (with the color of the rock representing the type of emotion, and the rough surface mapping to negative factors); the fourth row, from left to right, depicts Sadness, Pessimism, Surprise (symbolizing tears).
Basic Emotion Projection.
Trees represent neutral emotions, with the height of the tree indicating the intensity of the emotion; lotus flowers signify positive emotions, with the number of lotuses indicating the strength of the emotion; the base of the small island symbolizes negative emotions, with a larger blue area indicating a higher intensity of negative feelings.
The concept of a sand tray is a metaphorical representation of inner emotions and desires, transforming them into tangible objects. This idea aligns with the philosophy of physical visualization, where abstract or intangible perceptions are converted into a multisensory experience. The fusion of sandplay and physical visualization gives birth to the concept of a mental landscape. Emotions can be metaphorically mapped to different natural elements, imaging sadness represented by the sea, with the ebb and flow of waves mirroring the oscillation of the emotion. Each individual's mental island may have a sunny beach symbolizing good moods, but during emotional storms, the island may become engulfed and trees toppled.
The mental landscape is a highly meaningful, creative, and empathetic visualization concept. Currently, this page only implements the retrieval of fixed 3D models (pre-modeled in Blender), but given enough time, it could fully enable the automatic selection and combination of physical elements based on emotion analysis results. Moreover, users could customize or participate in the creation of these physical elements, enhancing end-user engagement in this dynamic visualization system. I believe the powerful rendering, modeling, and animation capabilities brought by three.js make further creative work an exciting possibility.
Mental Landscape.
This mental landscape embodies the metaphor of emotions and empathy, offering insights and materials from a therapeutic perspective. With physical visualization, we can also quantitatively define the intensity of emotions.
Visualizing emotions through a psychological empathy lens still necessitates further creativity and imagination. The concept of the Empathy Box draws inspiration from Dutch artist Maarten Baas's series of "Real Time Clock" video artworks. These pieces feature live clocks manipulated by individuals in diverse ways, including one where a person appears "trapped" within the clock, continuously erasing and redrawing the hands manually. Baas's creations ingeniously depict the passage of time and highlight the intricate interplay between people and time. By situating individuals within the clock, Baas delves into the tension between subjective time experiences and physical reality (Baas, 2016).
Baas's project effectively evokes empathy by depicting anxiety within a confined space, conveying the overwhelming nature of time's passage subtly. Likewise, negative emotions like depression and anxiety can induce a sense of spatial suffocation, commonly linked with feelings of oppression, torment, and entrapment, extensively researched in psychology, neuroscience, and psychoanalysis (Gorman, 1996; Chorpita & Barlow, 1998; Barlow et al., 2000; Olatunji et al., 2007).
Drawing from the inspiration above, the Empathy Box is represented by a 3D box that changes its background color and translucency based on the emotional state of a dancing figure within Yellow, green, and blue correspond to different emotion categories (positive, neutral, negative), and the material of the box becomes opaque as the emotion shifts towards negativity.
At the peak of sadness, the box transforms into a hazy azure glass effect, obscuring the figure's movements and conveying a sense of crashing against its walls. This 3D visualization achieves an immediate portrayal of a psychological state, immersing users in a sorrowful container. Conversely, during brighter moods, the box may appear as transparent glass surrounded by warm hues, enabling joyful movement. This design facilitates empathetic visualization of emotions, allowing users to control additional interactive effects based on their psychological parameters.
Background
Millions suffer from serious mental illnesses (SMI) such as depression, with the global direct and indirect impact of these disorders estimated to be $2.5 trillion annually (Doran et al., 2017). This staggering statistic underscores the urgent need for effective interventions in mental healthcare. Visualizing mental health, encompassing aspects like mood, has been shown to enhance individual understanding and aid in treatment planning by healthcare providers (East & Havard., 2015; Murnane et al., 2018). Additionally, visualization offers researchers invaluable insights into mental health conditions, allowing for more targeted interventions (Carroll et al., 2014). Within this context, visualizing emotions specifically promotes awareness, stimulates self-reflection, and facilitates communication (Peng et al., 2023). Studies have shown that individuals who engage in emotion visualization exercises report greater emotional awareness and regulation skills (Morris et al., 2010). Therefore, emotion visualization serves as an essential tool for observing emotional trends and implementing strategies for emotional regulation, ultimately playing a vital role in promoting psychological well-being.