CHI '22: CHI Conference on Human Factors in Computing Systems

Recent research proposed eyelid gestures for people with upper-body motor impairments (UMI) to interact with smartphones without finger touch. However, such eyelid gestures were designed by researchers. It remains unknown what eyelid gestures people with UMI would want and be able to perform. Moreover, other above-the-neck body parts (e.g., mouth, head) could be used to form more gestures. We conducted a user study in which 17 people with UMI designed above-the-neck gestures for 26 common commands on smartphones. We collected a total of 442 user-defined gestures involving the eyes, the mouth, and the head. Participants were more likely to make gestures with their eyes and preferred gestures that were simple, easy-to-remember, and less likely to draw attention from others. We further conducted a survey (N=24) to validate the usability and acceptance of these user-defined gestures. Results show that user-defined gestures were acceptable to both people with and without motor impairments.

We examine touchscreen stroke-gestures and mid-air motion-gestures articulated by users with upper-body motor impairments with devices worn on the wrist, finger, and head. We analyze users’ gesture input performance in terms of production time, articulation consistency, and kinematic measures, and contrast the performance of users with upper-body motor impairments with that of a control group of users without impairments. Our results, from two datasets of 7,290 stroke-gestures and 3,809 motion-gestures collected from 28 participants, reveal that users with upper-body motor impairments take twice as much time to produce stroke-gestures on wearable touchscreens compared to users without impairments, but articulate motion-gestures equally fast and with similar acceleration. We interpret our findings in the context of ability-based design and propose ten implications for accessible gesture input with upper-body wearables for users with upper-body motor impairments.

Individuals with spinal cord injury (SCI) need to perform numerous self-care behaviors, some very frequently. Pressure reliefs (PRs), which prevent life-threatening pressure ulcers (PUs), are one such behavior. We conducted a qualitative study with seven individuals with severe SCI—who depend on power wheelchairs—to explore their current PR behavior and the potential for technology to facilitate PR adherence. While our participants were highly motivated to perform PRs because of prior PUs, we found that their understanding of how and when to perform a PR differed by individual, and that while they sometimes forgot to perform PR, in other cases contextual factors made it difficult to perform a PR. Our findings provide insight into the complexity of this design space, identify design considerations for designing technology to facilitate these behaviors, and demonstrate the opportunity for personal informatics to be more inclusive by supporting the needs of this population.

People with limited mobility often use multiple devices when interacting with computing systems, but little is known about the impact these multi-modal configurations have on daily computing use. A deeper understanding of the practices, preferences, obstacles, and workarounds associated with accessible multi-modal input can uncover opportunities to create more accessible computer applications and hardware. We explored how people with limited mobility use multi-modality through a three-part investigation grounded in the context of video games. First, we surveyed 43 people to learn about their preferred devices and configurations. Next, we conducted semi-structured interviews with 14 participants to understand their experiences and challenges with using, configuring, and discovering input setups. Lastly, we performed a systematic review of 74 YouTube videos to illustrate and categorize input setups and adaptations in-situ. We conclude with a discussion on how our findings can inform future accessibility research for current and emerging computing technologies.

End-users can potentially style and customize websites by editing them through in-browser developer tools. Unfortunately, end-users lack the knowledge needed to translate high-level styling goals into low-level code edits. We present Stylette, a browser extension that enables users to change the style of websites by expressing goals in natural language. By interpreting the user’s goal with a large language model and extracting suggestions from our dataset of 1.7 million web components, Stylette generates a palette of CSS properties and values that the user can apply to reach their goal. A comparative study (N=40) showed that Stylette lowered the learning curve, helping participants perform styling changes 35% faster than those using developer tools. By presenting various alternatives for a single goal, the tool helped participants familiarize themselves with CSS through experimentation. Beyond CSS, our work can be expanded to help novices quickly grasp complex software or programming languages.

We demonstrate that recent natural language processing (NLP) techniques introduce a new paradigm of vocabulary learning that benefits from both micro and usage-based learning by generating and presenting the usages of foreign words based on the learner’s context. Then, without allocating dedicated time for studying, the user can become familiarized with how the words are used by seeing the example usages during daily activities, such as Web browsing. To achieve this, we introduce VocabEncounter, a vocabulary-learning system that suitably encapsulates the given words into materials the user is reading in near real time by leveraging recent NLP techniques. After confirming the system’s human-comparable quality of generating translated phrases by involving crowdworkers, we conducted a series of user studies, which demonstrated its effectiveness on learning vocabulary and its favorable experiences. Our work shows how NLP-based generation techniques can transform our daily activities into a field for vocabulary learning.

Context-aware applications have the potential to act opportunistically to facilitate human experiences and activities, from reminding us of places to perform personal activities, to identifying coincidental moments to engage in digitally-mediated shared experiences. However, despite the availability of context-detectors and programming frameworks for defining how such applications should trigger, designers lack support for expressing their human concepts of a situation and the experiences and activities they afford (e.g., situations to toss a frisbee) when context-features are made available at the level of locations (e.g., parks). This paper introduces Affinder, a block-based programming environment that supports constructing concept expressions that effectively translate their conceptions of a situation into a machine representation using available context features. During pilot testing, we discovered three bridging challenges that arise when expressing situations that cannot be encoded directly by a single context-feature. To overcome these bridging challenges, Affinder provides designers (1) an unlimited vocabulary search for discovering features they may have forgotten; (2) prompts for reflecting and expanding their concepts of a situation and ideas for foraging for context-features; and (3) simulation and repair tools for identifying and resolving issues with the precision of concept expressions on real use-cases. In a comparison study, we found that Affinder’s core functions helped designers stretch their concepts of how to express a situation, find relevant context-features matching their concepts, and recognize when the concept expression operated differently than intended on real-world cases. These results show that Affinder and tools that support bridging can improve a designer’s ability to express their concepts of a human situation into detectable machine representations—thus pushing the boundaries of how computing systems support our activities in the world.

Shared control is an emerging interaction paradigm in which a human and an AI partner collaboratively control a system. Shared control unifies human and artificial intelligence, making the human’s interactions with computers more accessible, safe, precise, effective, creative, and playful. This form of interaction has independently emerged in contexts as varied as mobility assistance, driving, surgery, and digital games. These domains each have their own problems, terminology, and design philosophies. Without a common language for describing interactions in shared control, it is difficult for designers working in one domain to share their knowledge with designers working in another. To address this problem, we present a dimension space for shared control, based on a survey of 55 shared control systems from six different problem domains. This design space analysis tool enables designers to classify existing systems, make comparisons between them, identify higher-level design patterns, and imagine solutions to novel problems.

In this study, we investigate how supporting serendipitous discovery and analysis of online product reviews can encourage readers to explore reviews more comprehensively prior to making purchase decisions. We propose two interventions — Exploration Metrics that can help readers understand and track their exploration patterns through visual indicators and a Bias Mitigation Model that intends to maximize knowledge discovery by suggesting sentiment and semantically diverse reviews. We designed, developed, and evaluated a text analytics system called Serendyze, where we integrated these interventions. We asked 100 crowd workers to use Serendyze to make purchase decisions based on product reviews. Our evaluation suggests that exploration metrics enabled readers to efficiently cover more reviews in a balanced way, and suggestions from the bias mitigation model influenced readers to make confident data-driven decisions. We discuss the role of user agency and trust in text-level analysis systems and their applicability in domains beyond review exploration.

Saliency methods — techniques to identify the importance of input features on a model’s output — are a common step in understanding neural network behavior. However, interpreting saliency requires tedious manual inspection to identify and aggregate patterns in model behavior, resulting in ad hoc or cherry-picked analysis. To address these concerns, we present Shared Interest: metrics for comparing model reasoning (via saliency) to human reasoning (via ground truth annotations). By providing quantitative descriptors, Shared Interest enables ranking, sorting, and aggregating inputs, thereby facilitating large-scale systematic analysis of model behavior. We use Shared Interest to identify eight recurring patterns in model behavior, such as cases where contextual features or a subset of ground truth features are most important to the model. Working with representative real-world users, we show how Shared Interest can be used to decide if a model is trustworthy, uncover issues missed in manual analyses, and enable interactive probing.

Choosing what to eat requires navigating a large volume of information and various competing factors. While recommendation systems are an effective approach to assist users with this culinary decision-making, they typically prioritise similarity to a query or user profile to give relevant results. This can expose users to an increasingly narrow band of phenomena, which could compromise dietary diversity, a factor in dietary quality. We designed Q-Chef, which combines a recipe recommendation system with a personalised model of surprise, and conducted a study to identify if surprise-eliciting recipes affect food decisions. Our study utilises a rigorous thematic analysis with over 40 participants to explore how computational models of surprise influence recipe choice. We also explored how these factors differed when people were presented with “surprising-yet-tasty” recipes, as opposed to just “tasty” recipes, and identified that being presented with surprising choices is more likely to elicit situational interest and prompt reflection on choices. We conclude with a set of suggestions for the design of future surprise-eliciting recipe systems.

Large repositories of products, patents and scientific papers offer an opportunity for building systems that scour millions of ideas and help users discover inspirations. However, idea descriptions are typically in the form of unstructured text, lacking key structure that is required for supporting creative innovation interactions. Prior work has explored idea representations that were either limited in expressivity, required significant manual effort from users, or dependent on curated knowledge bases with poor coverage. We explore a novel representation that automatically breaks up products into fine-grained functional aspects capturing the purposes and mechanisms of ideas, and use it to support important creative innovation interactions: functional search for ideas, and exploration of the design space around a focal problem by viewing related problem perspectives pooled from across many products. In user studies, our approach boosts the quality of creative search and inspirations, substantially outperforming strong baselines by 50-60%.

Often, emotional disorders are overlooked due to their lack of awareness, resulting in potential mental issues. Recent advances in sensing and inference technology provide a viable path to wearable facial-expression-based emotion recognition. However, most prior work has explored only laboratory settings and few platforms are geared towards end-users in everyday lives or provide personalized emotional suggestions to promote self-regulation. We present EmoGlass, an end-to-end wearable platform that consists of emotion detection glasses and an accompanying mobile application. Our single-camera-mounted glasses can detect seven facial expressions based on partial face images. We conducted a three-day out-of-lab study (N=15) to evaluate the performance of EmoGlass. We iterated on the design of the EmoGlass application for effective self-monitoring and awareness of users’ daily emotional states. We report quantitative and qualitative findings, based on which we discuss design recommendations for future work on sensing and enhancing awareness of emotional health.

Prior research has studied the detrimental impact of algorithmic management on gig workers and strategies that workers devise in response. However, little work has investigated alternative platform designs to promote worker well-being, particularly from workers’ own perspectives. We use a participatory design approach wherein workers explore their algorithmic imaginaries to co-design interventions that center their lived experiences, preferences, and well-being in algorithmic management. Our interview and participatory design sessions highlight how various design dimensions of algorithmic management, including information asymmetries and unfair, manipulative incentives, hurt worker well-being. Workers generate designs to address these issues while considering competing interests of the platforms, customers, and themselves, such as information translucency, incentives co-configured by workers and platforms, worker-centered data-driven insights for well-being, and collective driver data sharing. Our work offers a case study that responds to a call for designing worker-centered digital work and contributes to emerging literature on algorithmic work.

Even entrepreneurs whose businesses are not technological (e.g., handmade goods) need to be able to use a wide range of computing technologies in order to achieve their business goals. In this paper, we follow a participatory action research approach and collaborate with various stakeholders at an entrepreneurial co-working space to design “Tech Help Desk”, an on-going technical service for entrepreneurs. Our model for technical assistance is strategic, in how it is designed to fit the context of local entrepreneurs, and responsive, in how it prioritizes emergent needs. From our engagements with 19 entrepreneurs and support personnel, we reflect on the challenges with existing technology support for non-technological entrepreneurs. Our work highlights the importance of ensuring technological support services can adapt based on entrepreneurs’ ever-evolving priorities, preferences and constraints. Furthermore, we find technological support services should maintain broad technical support for entrepreneurs’ long tail of computing challenges.

It is difficult to design systems that honor the complex and often contradictory emotions that can be surfaced by sensitive encounters with recommender systems. To explore the design and ethical considerations in this space, we interviewed 20 people who had recently seen sensitive content through Facebook’s Memories feature. Interviewees typically described how (1) expectedness, (2) context of viewing, and (3) what we describe as “affective sense-making” were important factors for how they perceived “bittersweet” content, a sensitizing concept from our interviews that we expand upon. To address these user needs, we pose provocations to support critical work in this area and we suggest that researchers and designers: (1) draw inspiration from no/low-technology artifacts, (2) use empirical research to identify contextual features that have negative impacts on users, and (3) conduct user studies on affective sense-making.

CAUTION: This paper discusses difficult subject matter related to death and relationships.

In this paper we present Immersive Speculative Enactments (ISEs), a novel concept that extends conventional Speculative Enactments to Virtual Reality. Through ISEs, participants are immersed in a speculative world depicted by the designers and can engage with it in its truest envisioned form. We explore this concept via four scenarios with increasing technological uncertainty: a glimpse in the daily life of the parent of a newborn baby; a Mixed Reality experience supporting hybrid classrooms; two wearable devices that present a pet’s emotional state and needs; and an enactment on the effect of communication delay across interplanetary distances. We discuss the concept of ISEs and contrast them to other forms of speculation, provide guidelines on how to design them, as well as reflecting on the challenges, limitations, and potential associated with the role of ISEs in the HCI discourse.

Many people have experienced mindlessly scrolling on social media. We investigated these experiences through the lens of normative dissociation: total cognitive absorption, characterized by diminished self-awareness and reduced sense of agency. To explore user experiences of normative dissociation and how design affects the likelihood of normative dissociation, we deployed Chirp, a custom Twitter client, to 43 U.S. participants. Experience sampling and interviews revealed that sometimes, becoming absorbed in normative dissociation on social media felt like a beneficial break. However, people also reported passively slipping into normative dissociation, such that they failed to absorb any content and were left feeling like they had wasted their time. We found that designed interventions–including custom lists, reading history labels, time limit dialogs, and usage statistics–reduced normative dissociation. Our findings demonstrate that interaction designs intended to capture attention likely do so by harnessing people’s natural inclination to seek normative dissociation experiences. This suggests that normative dissociation may be a more productive framing than addiction for discussing social media overuse.

HCI researchers have increasingly examined how social context shapes health behaviors. Much of this work operates at the interpersonal level. Communities such as churches play important roles in supporting wellbeing and addressing health inequities. While some work has investigated creating digital health tools for religious populations, few have explicitly focused on the incorporation of community support in the form of prayer support. Embedding health interventions in any community has the potential to support or challenge the community’s dynamics. We report on findings from interviews with 17 church members who used a church-based mHealth application over a 4-week period and provide guidelines for developers based on these results. Through their use of the system, participants characterized several community dynamics including a desire for social intimacy, communicating care, creating opportunities for fellowship, maintaining privacy and discretion, and building community connections, and how these dynamics influence their aspirations for a church-based health app.

Recovery from addiction is a journey that requires a lifetime of support from a strong network of peers. Many people seek out this support through online communities, like those on Reddit. However, as these communities developed outside of existing aid groups and medical practice, it is unclear how they enable recovery. Their scale also limits researchers’ ability to engage through traditional qualitative research methods. To study these groups, we performed a topic-guided thematic analysis that used machine-generated topic models to purposively sample from two recovery subreddits: r/stopdrinking and r/OpiatesRecovery. We show that these communities provide access to an experienced and accessible support group whose discussions include consequences, reflections, and celebrations, but that also play a distinct metacommunicative role in supporting formal treatment. We discuss how these communities can act as knowledge sources to improve in-person recovery support and medical practice, and how computational techniques can enable HCI researchers to study communities at scale.

Apps for depression can increase access to mental health care but concerns abound with disparities between academic development of apps and those available through app stores. Reviews highlighted ethical shortcomings of these self-management tools, with a need for greater insight into how ethical issues are experienced by users. We addressed these gaps by exploring user reviews of such apps to better understand user experiences and ethical issues. We conducted a thematic analysis of 2,217 user reviews sampled from 40 depression apps in Google Play and Apple App Store, totaling over 77,500 words. Users reported positive and negative experiences, with ethical implications evident in areas of benefits, adverse effects, access, usability and design, support, commercial models, autonomy, privacy, and transparency. We integrated our elements of ethically designed apps for depression and principles of nonmaleficence, beneficence, justice, autonomy, and virtue, and we conclude with implications for ethical design of apps for depression.

Immersive interactive technologies such as virtual reality (VR) have the potential to foster well-being. While VR applications have been successfully used to evoke positive emotions through the presetting of light, colour and scenery, the experiential potential of allowing users to independently create a virtual environment (VE) has not yet been sufficiently addressed. To that end, we explore how the autonomous design of a VE can affect emotional engagement and well-being. We present Mood Worlds – a VR application allowing users to visualise their emotions by self-creating a VE. In an exploratory evaluation (N=16), we found that Mood Worlds is an effective tool supporting emotional engagement. Additionally, we found that an autonomous creation process in VR increases positive emotions and well-being. Our work shows that VR can be an effective tool to visualise emotions, thereby increasing positive affect. We discuss opportunities and design requirements for VR as positive technology.

Analysis of human motion data can reveal valuable insights about the utilization of space and interaction of humans with their environment. To support this, we present AvatAR, an immersive analysis environment for the in-situ visualization of human motion data, that combines 3D trajectories with virtual avatars showing people’s detailed movement and posture. Additionally, we describe how visualizations can be embedded directly into the environment, showing what a person looked at or what surfaces they touched, and how the avatar’s body parts can be used to access and manipulate those visualizations. AvatAR combines an AR HMD with a tablet to provide both mid-air and touch interaction for system control, as well as an additional overview device to help users navigate the environment. We implemented a prototype and present several scenarios to show that AvatAR can enhance the analysis of human motion data by making data not only explorable, but experienceable.

The nascent field of mixed reality is seeing an ever-increasing need for user studies and field evaluation, which are particularly challenging given device heterogeneity, diversity of use, and mobile deployment. Immersive analytics tools have recently emerged to support such analysis in situ, yet the complexity of the data also warrants an ex-situ analysis using more traditional non-immersive visual analytics setups. To bridge the gap between both approaches, we introduce ReLive: a mixed-immersion visual analytics framework for exploring and analyzing mixed reality user studies. ReLive combines an in-situ virtual reality view with a complementary ex-situ desktop view. While the virtual reality view allows users to relive interactive spatial recordings replicating the original study, the synchronized desktop view provides a familiar interface for analyzing aggregated data. We validated our concepts in a two-step evaluation consisting of a design walkthrough and an empirical expert user study.

As mixed-reality (MR) technologies become more mainstream, the delineation between data visualisations displayed on screens or other surfaces and those floating in space becomes increasingly blurred. Rather than the choice of using either a 2D surface or the 3D space for visualising data being a dichotomy, we argue that users should have the freedom to transform visualisations seamlessly between the two as needed. However, the design space for such transformations is large, and practically uncharted. To explore this, we first establish an overview of the different states that a data visualisation can take in MR, followed by how transformations between these states can facilitate common visualisation tasks. We then describe a design space of how these transformations function, in terms of the different stages throughout the transformation, and the user interactions and input parameters that affect it. This design space is then demonstrated with multiple exemplary techniques based in MR.

The helical axis is a common tool used in biomechanical modeling to parameterize the motion of rigid objects. It encodes an object’s rotation around and translation along a unique axis. Visualizations of helical axes have helped to make kinematic data tangible. However, the analysis process often remains tedious, especially if complex motions are examined. We identify multiple key challenges: the absence of interactive tools for the computation and handling of helical axes, visual clutter in axis representations, and a lack of contextualization. We solve these issues by providing the first generalized framework for kinematic analysis with helical axes. Axis sets can be computed on-demand, interactively filtered, and explored in multiple coordinated views. We iteratively developed and evaluated the HAExplorer with active biomechanics researchers. Our results show that the techniques we introduce open up the possibility to analyze non-planar, compound, and interdependent motion data.

We present the findings of four studies related to the visualization of sleep data on wearables with two form factors: smartwatches and fitness bands. Our goal was to understand the interests, preferences, and effectiveness of different sleep visualizations by form factor. In a survey, we showed that wearers were mostly interested in weekly sleep duration, and nightly sleep phase data. Visualizations of this data were generally preferred over purely text-based representations, and the preferred chart type for fitness bands, and smartwatches was often the same. In one in-person pilot study, and two crowdsourced studies, we then tested the effectiveness of the most preferred representations for different tasks, and found that participants performed simple tasks effectively on both form factors but more complex tasks benefited from the larger smartwatch size. Lastly, we reflect on our crowdsourced study methodology for testing the effectiveness of visualizations for wearables. Supplementary material is available at https://osf.io/yz8ar/.

Recent work has highlighted that emotion is key to the user experience with data stories. However, limited attention has been paid to negative emotions specifically. This work investigates the outcomes of negative emotions in the context of serious data stories and examines how they can be augmented by design methods from the perspectives of both storytellers and viewers. First, we conducted a workshop with 9 data story experts to understand the possible benefits of eliciting negative emotions in serious data stories and 19 potential design methods that contribute to negative emotions. Based on the findings from the workshop, we then conducted a lab study with 35 participants to explore the outcomes of eliciting negative emotions as well as the effectiveness of the design methods. The results indicated that negative emotions mainly facilitated contemplative experiences and long-term memory. Besides, the design methods showed varied effectiveness in augmenting negative emotions and being recalled.

Chatbots have garnered interest as conversational interfaces for a variety of tasks. While general design guidelines exist for chatbot interfaces, little work explores analytical chatbots that support conversing with data. We explore Gricean Maxims to help inform the basic design of effective conversational interaction. We also draw inspiration from natural language interfaces for data exploration to support ambiguity and intent handling. We ran Wizard of Oz studies with 30 participants to evaluate user expectations for text and voice chatbot design variants. Results identified preferences for intent interpretation and revealed variations in user expectations based on the interface affordances. We subsequently conducted an exploratory analysis of three analytical chatbot systems (text + chart, voice + chart, voice-only) that implement these preferred design variants. Empirical evidence from a second 30-participant study informs implications specific to data-driven conversation such as interpreting intent, data orientation, and establishing trust through appropriate system responses.

Making time estimates, such as how long a given task might take, frequently leads to inaccurate predictions because of an optimistic bias. Previous attempts to alleviate this bias, including decomposing the task into smaller components and listing potential surprises, have not shown any major improvement. This article builds on the premise that these procedures may have failed because they involve compound probabilities and mixture distributions which are difficult to compute in one’s head. We hypothesize that predictive visualizations of such distributions would facilitate the estimation of task durations. We conducted a crowdsourced study in which 145 participants provided different estimates of overall and sub-task durations and we used these to generate predictive visualizations of the resulting mixture distributions. We compared participants’ initial estimates with their updated ones and found compelling evidence that predictive visualizations encourage less optimistic estimates.

Changing a Twitter account’s privacy setting between public and protected changes the visibility of past tweets. By inspecting the privacy setting of more than 100K Twitter users over 3 months, we noticed that over 40% of those users changed their privacy setting at least once with around 16% changing it over 5 times. This observation motivated us to explore the reasons why people switch their privacy settings. We studied these switching phenomena quantitatively by comparing the tweeting behaviour of users when public vs protected, and qualitatively using two follow-up surveys (n=100, n=324) to understand potential reasoning behind the observed behaviours. Our quantitative analysis shows that users who switch privacy settings mention others and share hashtags more when their setting is public. Our surveys highlighted that users turn protected to share personal content and regulate boundaries while they turn public to interact with others in ways the protected setting prevents.

People feel concerned, angry, and powerless when subjected to surveillance, data breaches and other privacy-violating experiences with institutions (PVEIs). Collective action may empower groups of people affected by a PVEI to jointly demand redress, but a necessary first step is for the collective to agree on demands. We designed a sensitizing prototype to explore how to shepherd a collective to generate a unified set of demands for redress in response to a triggering PVEI. We found that collectives can converge on high-priority concerns and demands for redress, and that many of their demands indicated preferences for broad reform. We then gathered a panel of security and privacy experts to react to the collective’s demands. Experts were dismissive, preferring incremental measures that cleanly mapped onto existing legal structures. We argue this misalignment may help uphold the power chasm between data-harvesting institutions and the individuals whose personal data they monetize.

There is limited information regarding how users employ password managers in the wild and why they use them in that manner. To address this knowledge gap, we conduct observational interviews with 32 password manager users. Using grounded theory, we identify four theories describing the processes and rationale behind participants’ usage of password managers. We find that many users simultaneously use both a browser-based and a third-party manager, using each as a backup for the other, with this new paradigm having intriguing usability and security implications. Users also eschew generated passwords because these passwords are challenging to enter and remember when the manager is unavailable, necessitating new generators that create easy-to-enter and remember passwords. Additionally, the credential audits provided by most managers overwhelm users, limiting their utility and indicating a need for more proactive and streamlined notification systems. We also discuss mobile usage, adoption and promotion, and other related topics.

Users avoid engaging with privacy policies because they are lengthy and complex, making it challenging to retrieve relevant information. In response, research proposed contextual privacy policies (CPPs) that embed relevant privacy information directly into their affiliated contexts. To date, CPPs are limited to concept showcases. This work evolves CPPs into a production tool that automatically extracts and displays concise policy information. We first evaluated the technical functionality on the US’s 500 most visited websites with 59 participants. Based on our results, we further revised the tool to deploy it in the wild with 11 participants over ten days. We found that our tool is effective at embedding CPP information on websites. Moreover, we found that the tool’s usage led to more reflective privacy behavior, making CPPs powerful in helping users understand the consequences of their online activities. We contribute design implications around CPP presentation to inform future systems design.

People share photos on Social Networks Sites, but at the same time want to keep some photo content private. This tension between sharing and privacy has led researchers to try to solve this problem, but without considering users’ needs. To fill this gap, we present a novel interface that expands privacy options beyond recipient-control (R). Our system can also flag sensitive content (C) and obfuscate (O) it (RCO). We then describe the results of a two-step experiment that compares RCO with two alternative interfaces - (R) which mimics existing SNS privacy options by providing recipient control, and a system that in addition to recipient control also flags sensitive content (RC). Results suggest RC performs worse than R regarding perceived privacy risks, willingness to share, and user experience. However, RCO, which provides obfuscation options, restores these metrics to the same levels as R. We conclude by providing insights on system implementation.

UI designers often correct false affordances and improve the discoverability of features when users have trouble determining if elements are tappable. We contribute a novel system that models the perceived tappability of mobile UI elements with a vision-based deep neural network and helps provide design insights with dataset-level and instance-level explanations of model predictions. Our system retrieves designs from similar mobile UI examples from our dataset using the latent space of our model. We also contribute a novel use of an interpretability algorithm, XRAI, to generate a heatmap of UI elements that contribute to a given tappability prediction. Through several examples, we show how our system can help automate elements of UI usability analysis and provide insights for designers to iterate their designs. In addition, we share findings from an exploratory evaluation with professional designers to learn how AI-based tools can aid UI design and evaluation for tappability issues.

The Bayesian information gain (BIG) framework has garnered significant interest as an interaction method for predicting a user’s intended target based on a user’s input. However, the BIG framework is constrained to goal-oriented cases, which renders it difficult to support changing goal-oriented cases such as design exploration. During the design exploration process, the design direction is often undefined and may vary over time. The designer’s mental model specifying the design direction is sequentially updated through the information-retrieval process. Therefore, tracking the change point of a user’s goal is crucial for supporting an information exploration. We introduce the BIGexplore framework for changing goal-oriented cases. BIGexplore detects transitions in a user’s browsing behavior as well as the user’s next target. Furthermore, a user study on BIGexplore confirms that the computational cost is significantly reduced compared with the existing BIG framework, and it plausibly detects the point where the user changes goals.

The simulation of user behavior with deep reinforcement learning agents has shown some recent success. However, the inverse problem, that is, inferring the free parameters of the simulator from observed user behaviors, remains challenging to solve. This is because the optimization of the new action policy of the simulated agent, which is required whenever the model parameters change, is computationally impractical. In this study, we introduce a network modulation technique that can obtain a generalized policy that immediately adapts to the given model parameters. Further, we demonstrate that the proposed technique improves the efficiency of user simulator-based inference by eliminating the need to obtain an action policy for novel model parameters. We validated our approach using the latest user simulator for point-and-click behavior. Consequently, we succeeded in inferring the user’s cognitive parameters and intrinsic reward settings with less than 1/1000 computational power to those of existing methods.

Reach redirection is an illusion-based virtual reality (VR) interaction technique where a user’s virtual hand is shifted during a reach in order to guide their real hand to a physical location. Prior works have not considered the underlying sensorimotor processes driving redirection. In this work, we propose adapting a sensorimotor model for goal-directed reach to obtain a model for visually-redirected reach, specifically by incorporating redirection as a sensory bias in the state estimate used by a minimum jerk motion controller. We validate and then leverage this model to develop a Model Predictive Control (MPC) approach for reach redirection, enabling the real-time generation of spatial warping according to desired optimization criteria (e.g., redirection goals) and constraints (e.g., sensory thresholds). We illustrate this approach with two example criteria – redirection to a desired point and redirection along a desired path – and compare our approach against existing techniques in a user evaluation.

When giving input with a button, users follow one of two strategies: (1) react to the output from the computer or (2) proactively act in anticipation of the output from the computer. We propose a technique to quantify reactiveness and proactiveness to determine the degree and characteristics of each input strategy. The technique proposed in this study uses only screen recordings and does not require instrumentation beyond the input logs. The likelihood distribution of the time interval between the button inputs and system outputs, which is uniquely determined for each input strategy, is modeled. Then the probability that each observed input/output pair originates from a specific strategy is estimated along with the parameters of the corresponding likelihood distribution. In two empirical studies, we show how to use the technique to answer questions such as how to design animated transitions and how to predict a player’s score in real-time games.

By CHI 2022, fifteen years will have passed since the emergence of Sustainable HCI (SHCI), which now constitutes an important subfield of HCI. In this paper, we draw on two SHCI corpora to ask: Has SHCI progressed? How has the field responded to prominent critiques? Have we identified and adopted constructive strategies for impacting environmental unsustainability? We further show the wide array of competencies SHCI researchers have been called to develop, and how this has been reflected in subsequent work. Our analysis identifies significant shifts in the SHCI landscape, toward research that is diverse and holistic, but also away from efforts to address the urgent climate crisis. We posit that SHCI has tended to take on far more than it could reasonably expect to deliver, and propose ‘Green Policy informatics’ as a pathway that enables SHCI to leverage a more traditional HCI skillset in addressing climate change.

The COVID-19 pandemic continues to affect the daily life of college students, impacting their social life, education, stress levels and overall mental well-being. We study and assess behavioral changes of N=180 undergraduate college students one year prior to the pandemic as a baseline and then during the first year of the pandemic using mobile phone sensing and behavioral inference. We observe that certain groups of students experience the pandemic very differently. Furthermore, we explore the association of self-reported COVID-19 concern with students’ behavior and mental health. We find that heightened COVID-19 concern is correlated with increased depression, anxiety and stress. We evaluate the performance of different deep learning models to classify student COVID-19 concerns with an AUROC and F1 score of 0.70 and 0.71, respectively. Our study spans a two-year period and provides a number of important insights into the life of college students during this period.

HCI is increasingly concerned with health information quality and spread of misinformation on social media. Despite many major platforms having been adopted across the world, the situated evaluation and sharing of health information is underexplored across diverse health systems and cultural and political contexts. Drawing on semi-structured interviews, we study the navigation of health information on social media in urban and rural South India, backdropped by plural knowledges around health and the specific politics and sociality of health and social media in this setting. We use Ivan Illich’s concept of tools for conviviality [49] to distinguish between how people creatively use tools versus how tools manage and impose values on people—participants aimed to use health information towards care beyond institutionalized healthcare, but insidious misinformation and information-sharing practices served to commodify, spark uncertainty in, and discipline caring behavior. We use our findings to expand understandings of the use of health information on social media and how positionality shapes how people are affected by and respond to misinformation. We also draw attention to the structural aspects of health misinformation in the Indian context and how the design of social media platforms might play a role in addressing it.

The role of HCI in informal caregivers’ lives has been a focus of research for some time. Yet to gain significance in HCI, are the implications of healthcare systems’ transformation into a personalised care paradigm, where citizens gain choice and control over the delivery of their care. We provide a first HCI paper to examine self-directed care budgets for disabled citizens, where care funding is controlled by the individual. We explore how digital technology can assist citizens, promoting peer support to create meaningful, personalised healthcare infrastructures. This qualitative study contributes insights from interviews and focus groups with 24 disabled citizens, informal caregivers and healthcare officers, to provide understanding of their experiences and practices. These insights highlight relational care, invisible labour, power struggles with authorities and how citizens seek socio-technical capability. We contribute design implications for self-directed care budgets and HCI research concerned with developing technologies that support this population.

Home health aides are a vulnerable group of frontline caregivers who provide personal and medically-oriented care in patients’ homes. Their work is difficult and unpredictable, involving a mix of physical and emotional labor as they adapt to patients’ changing needs. Our paper presents an exploratory, qualitative study with 32 participants, that investigates design opportunities for Interactive Voice Assistants (IVAs) to support aides’ essential care work. We explore challenges and opportunities for IVAs to (1) fill gaps in aides’ access to information and care coordination, (2) assist with decision making and task completion, (3) advocate on behalf of aides, and (4) provide emotional support. We then discuss key implications of our work, including how materiality may impact perceived ownership and usage of IVAs, the need to carefully consider tensions around surveillance, accountability, data collection, and reporting, and the challenges of centering aides as essential workers in complex home health care contexts.

Many of us daily encounter shadow and reflected light patterns alongside macro-level changes in ambient light levels. These are caused by elements—opaque objects, glass, mirrors, even clouds—in our environment interfacing with sunlight or artificial indoor lighting. Inspired by these phenomena, we explored ways of creating digitally-supported displays that use light, shade and reflection for output and harness the energy they need to operate from the sun or indoor ambient light. Through a set of design workshops we developed exemplar devices: SolarPix, ShadMo and GlowBoard. We detail their function and implementation, as well as evidencing their technical viability. The designs were informed by material understandings from the Global North and Global South and demonstrated in a cross-cultural workshop run in parallel in India and South Africa where community co-designers reflected on their uses and value given lived experience of their communication practices and unreliable energy networks.

In recognition of food's significant experiential pleasures, culinary practitioners and designers are increasingly exploring novel combinations of computing technologies and food. However, despite much creative endeavors, proposals and prototypes have so far largely maintained a traditional divide, treating food and technology as separate entities. In contrast, we present a “Research through Design” exploration of the notion of food as computational artifact: wherein food itself is the material of computation. We describe the Logic Bonbon, a dessert that can hydrodynamically regulate its flavor via a fluidic logic system. Through a study of experiencing the Logic Bonbon and reflection on our design practice, we offer a provisional account of how food as computational artifact can mediate new interactions through a novel approach to food-computation integration, that promotes an enriched future of Human-Food Interaction.

Slab-based ceramics are constructed by rolling out flat sheets of clay, cutting out a pattern, and then folding the cut clay to build a three-dimensional design. Slabforge is an open-source web-based software application that supports slab-based ceramics. It enables users to design a range of simple 3D forms and then generate flat patterns and matching 3D-printable slump molds that support the construction of those forms. This paper discusses the development of the software in the context of our own ceramics practice and then describes the results of a study in which students in an introductory ceramics course used Slabforge to create tea sets. We use both of these experiences to motivate a critical reflection on the relationships between materials, craft, digital fabrication, and software, introducing three themes of friction that we encountered during the course of this project.

Fluid fiber, with fluid flowing in a tube, is an attractive material to flexibly and dynamically display digital information. While it has recently attracted attention from the HCI field, there is currently little knowledge about this material, limited to controlling the position of the droplets to present representational information like letters and numbers. To develop a broader and deepened understanding of this material and its potential for display design, we conducted a study based on a design workshop where art and design practitioners engaged in creation practice with a toolkit we designed and developed. The toolkit includes hardware components for controlling bubbles and droplets and a GUI design tool for arranging the fluid layout. Our research reveals the structural and expressive affordance of such a fluid fiber for displaying information, highlighting the unique value of fluidity as an intuitive form to express life, emotion, movement and changes.

While systems that use Artificial Intelligence (AI) are increasingly becoming part of everyday technology use, we do not fully understand how AI changes design processes. A structured understanding of how designers work with AI is needed to improve the design process and educate future designers. To that end, we conducted interviews with designers who participated in projects which used AI. While past work focused on AI systems created by experienced designers, we focus on the perspectives of a diverse sample of interaction designers. Our results show that the design process of an interactive system is affected when AI is integrated and that design teams adapt their processes to accommodate AI. Based on our data, we contribute four approaches adopted by interaction designers working with AI: a priori, post-hoc, model-centric, and competence-centric. Our work contributes a pragmatic account of how design processes for AI systems are enacted.

Enterprises have recently adopted AI to human resource management (HRM) to evaluate employees’ work performance evaluation. However, in such an HRM context where multiple stakeholders are complexly intertwined with different incentives, it is problematic to design AI reflecting one stakeholder group's needs (e.g., enterprises, HR managers). Our research aims to investigate what tensions surrounding AI in HRM exist among stakeholders and explore design solutions to balance the tensions. By conducting stakeholder-centered participatory workshops with diverse stakeholders (including employees, employers/HR teams, and AI/business experts), we identified five major tensions: 1) divergent perspectives on fairness, 2) the accuracy of AI, 3) the transparency of the algorithm and its decision process, 4) the interpretability of algorithmic decisions, and 5) the trade-off between productivity and inhumanity. We present stakeholder-centered design ideas for solutions to mitigate these tensions and further discuss how to promote harmony among various stakeholders at the workplace.

AI-based decision support tools (ADS) are increasingly used to augment human decision-making in high-stakes, social contexts. As public sector agencies begin to adopt ADS, it is critical that we understand workers’ experiences with these systems in practice. In this paper, we present findings from a series of interviews and contextual inquiries at a child welfare agency, to understand how they currently make AI-assisted child maltreatment screening decisions. Overall, we observe how workers’ reliance upon the ADS is guided by (1) their knowledge of rich, contextual information beyond what the AI model captures, (2) their beliefs about the ADS’s capabilities and limitations relative to their own, (3) organizational pressures and incentives around the use of the ADS, and (4) awareness of misalignments between algorithmic predictions and their own decision-making objectives. Drawing upon these findings, we discuss design implications towards supporting more effective human-AI decision-making.

Creating presentation slides is a critical but time-consuming task for data scientists. While researchers have proposed many AI techniques to lift data scientists’ burden on data preparation and model selection, few have targeted the presentation creation task. Based on the needs identified from a formative study, this paper presents NB2Slides, an AI system that facilitates users to compose presentations of their data science work. NB2Slides uses deep learning methods as well as example-based prompts to generate slides from computational notebooks, and take users’ input (e.g., audience background) to structure the slides. NB2Slides also provides an interactive visualization that links the slides with the notebook to help users further edit the slides. A follow-up user evaluation with 12 data scientists shows that participants believed NB2Slides can improve efficiency and reduces the complexity of creating slides. Yet, participants questioned the future of full automation and suggested a human-AI collaboration paradigm.

Despite impressive performance in many benchmark datasets, AI models can still make mistakes, especially among out-of-distribution examples. It remains an open question how such imperfect models can be used effectively in collaboration with humans. Prior work has focused on AI assistance that helps people make individual high-stakes decisions, which is not scalable for a large amount of relatively low-stakes decisions, e.g., moderating social media comments. Instead, we propose conditional delegation as an alternative paradigm for human-AI collaboration where humans create rules to indicate trustworthy regions of a model. Using content moderation as a testbed, we develop novel interfaces to assist humans in creating conditional delegation rules and conduct a randomized experiment with two datasets to simulate in-distribution and out-of-distribution scenarios. Our study demonstrates the promise of conditional delegation in improving model performance and provides insights into design for this novel paradigm, including the effect of AI explanations.

Recent HCI research has sought to develop guidelines—‘heuristics’, ‘best practices’, ‘principles’ and so on—for voice user interfaces (VUI) to aid both practitioners and researchers in improving the quality of VUI-based design. However, limited research is available on how such design knowledge is conceptualised and used by industry practitioners. We present a small interview-based study conducted with 9 experienced VUI industry practitioners. Their concerns range from terminological challenges associated with VUI design knowledge, the role of codifications of such knowledge like design guidelines alongside their practical design work, through to their views on the value of ‘harmonisation’ of VUI design knowledge. Given the complex—albeit preliminary—picture that emerges, we argue for HCI's deeper consideration of how design knowledge meshes with the contingencies of practice, so that VUI design knowledge—such as design guidelines developed in HCI—delivers the most potential value for industry practice.

Can you speak the way you desire without feeling the pressure to conform to standards of speaking? In this study, we investigated the impact of user-controlled voice manipulation and listening to recordings of model speakers on self-perceptions of voice and speech. Quantitative analysis showed that there was a significant improvement in the perceived confidence of tone by listening to model speakers, but there were no significant improvements due to voice manipulation. Qualitative analysis of interviews revealed that participants responded positively to the visual and auditory feedback provided by the voice manipulation software. The participants also evaluated the quality of model speakers to decide whether or not they wanted to refer to them for speech practice. Based on the results of these analyses, we summarized the design implications for a speech practice system that would allow further investigation of the impact of the system on self-perceptions of speech performance.

Conversational agents are being widely adopted across several domains to serve a variety of purposes ranging from providing intelligent assistance to companionship. Recent literature has shown that users develop intuitive folk theories and a metaphorical understanding of conversational agents (CAs) due to the lack of a mental model of the agents. However, investigation of metaphorical agent representation in the HCI community has mainly focused on the human level, despite non-human metaphors for agents being prevalent in the real world. We adopted Lakoff and Turner’s ‘Great Chain of Being’ framework to systematically investigate the impact of using non-human metaphors to represent conversational agents on worker engagement in crowdsourcing marketplaces. We designed a text-based conversational agent that assists crowd workers in task execution. Through a between-subjects experimental study (N = 341), we explored how different human and non-human metaphors affect worker engagement, the perceived cognitive load of workers, intrinsic motivation, and their trust in the agents. Our findings bridge the gap of how users experience CAs with non-human metaphors in the context of conversational crowdsourcing.

Voice interaction has long been envisioned as enabling users to transform physical interaction into hands-free, such as allowing fine-grained control of instructional videos without physically disengaging from the task at hand. While significant engineering advances have brought us closer to this ideal, we do not fully understand the user requirements for voice interactions that should be supported in such contexts. This paper presents an ecologically-valid wizard-of-oz elicitation study exploring realistic user requirements for an ideal instructional video playback control while cooking. Through the analysis of the issued commands and performed actions during this non-linear and complex task, we identify (1) patterns of command formulation, (2) challenges for design, and (3) how task and voice-based commands are interwoven in real-life. We discuss implications for the design and research of voice interactions for navigating instructional videos while performing complex tasks.

Although there is a potential demand for customizing voices, most customization is limited to the visual appearance of a figure (e.g., avatars). To better understand the users’ need, we first conducted an online survey with 104 participants. Then we conducted a semi-structured interview with a prototype with 14 participants to identify design considerations for supporting voice customization. The results show that there is a desire for voice customization especially for non-face-to-face conversations with someone unfamiliar. In addition, the findings revealed that different voices are favored for different contexts from a better version of one’s own voice for improving delivery to a completely different voice for securing identity. As future work, we plan to extend this study by investigating voice synthesis techniques for end-users who wish to design their own voices for various contexts.

Existing designs helping people manage their social media use include: 1) external supports that monitor and limit use; 2) internal supports that change the interface itself. Here, we design and deploy Chirp, a mobile Twitter client, to independently examine how users experience external and internal supports. To develop Chirp, we identified 16 features that influence users’ sense of agency on Twitter through a survey of 129 participants and a design workshop. We then conducted a four-week within-subjects deployment with 31 participants. Our internal supports (including features to filter tweets and inform users when they have exhausted new content) significantly increased users’ sense of agency, while our external supports (a usage dashboard and nudges to close the app) did not. Participants valued our internal supports and said that our external supports were for “other people.” Our findings suggest that design patterns promoting agency may serve users better than screen time tools.

From zooming on smartphones and mid-air gestures to deformable user interfaces, thumb-index pinching grips are used in many interaction techniques. However, there is still a lack of systematic understanding of how the accuracy and efficiency of such grips are affected by various factors such as counterforce, grip span, and grip direction. Therefore, in this paper, we contribute an evaluation (N = 18) of thumb-index pinching performance in a visual targeting task using scales up to 75 items. As part of our findings, we conclude that the pinching interaction between the thumb and index finger is a promising modality also for one-dimensional input on higher scales. Furthermore, we discuss and outline implications for future user interfaces that benefit from pinching as an additional and complementary interaction modality.

Research output related to digital nudging has increased ten-fold over the last five years. Nudging in the digital realm differs from its analog counterpart in important ways. For instance, online, choice architectures can be interconnected and personalized using real time data. In the face of this development, it is crucial to understand the current state of the literature and to map out possible research gaps. This paper addresses this issue and provides a systematic review of empirical studies where digital nudges have been evaluated. The systematic review covers 73 peer-reviewed papers containing 109 separate studies where 231 digital nudges have been evaluated. Our results lead to nine open research questions to be addressed in the future by the research community.

Interactions with our personal and family pictures are essential to continued social reminiscence, leading to long-term benefits, including reduced social isolation. Previous research has identified how designs of digital picture tools fall short of physical options specifically in terms of reminiscence. However, the relative prompting abilities of different digital interactions, including the types of memories prompted like external facts or person-centred memories, have not yet been explored. To investigate this, we present a controlled study of the memories prompted by three digital picture interactions (slideshow, gallery, and tabletop) on personal touchscreen devices. We find differences in how these tools and the interactions they support prompt reminiscence. In particular, gallery views prompt significantly fewer memories than either the tabletop or slideshow. Slideshows prompt significantly more external, factual memories, but not more person-centred memories, which are key to reminiscence. This has implications for the overall social usability of digital picture interactions.

To better ground technical (systems) investigation and interaction design of cross-device experiences, we contribute an in-depth survey of existing multi-device practices, including fragmented workflows across devices and the way people physically organize and configure their workspaces to support such activity. Further, this survey documents a historically significant moment of transition to a new future of remote work, an existing trend dramatically accelerated by the abrupt switch to work-from-home (and having to contend with the demands of home-at-work) during the COVID-19 pandemic. We surveyed 97 participants, and collected photographs of home setups and open-ended answers to 50 questions categorized in 5 themes. We characterize the wide range of multi-device physical configurations and identify five usage patterns, including: partitioning tasks, integrating multi-device usage, cloning tasks to other devices, expanding tasks and inputs to multiple devices, and migrating between devices. Our analysis also sheds light on the benefits and challenges people face when their workflow is fragmented across multiple devices. These insights have implications for the design of multi-device experiences that support people’s fragmented workflows.

Augmented Reality (AR) experiences tightly associate virtual contents with environmental entities. However, the dissimilarity of different environments limits the adaptive AR content behaviors under large-scale deployment. We propose ScalAR, an integrated workflow enabling designers to author semantically adaptive AR experiences in Virtual Reality (VR). First, potential AR consumers collect local scenes with a semantic understanding technique. ScalAR then synthesizes numerous similar scenes. In VR, a designer authors the AR contents’ semantic associations and validates the design while being immersed in the provided scenes. We adopt a decision-tree-based algorithm to fit the designer’s demonstrations as a semantic adaptation model to deploy the authored AR experience in a physical scene. We further showcase two application scenarios authored by ScalAR and conduct a two-session user study where the quantitative results prove the accuracy of the AR content rendering and the qualitative results show the usability of ScalAR.

In many domains, strategic knowledge is documented and shared through checklists and handbooks. In software engineering, however, developers rarely share strategic knowledge for approaching programming problems, in contrast to other artifacts and despite its importance to productivity and success. To understand barriers to sharing, we simulated a programming strategy knowledge-sharing platform, asking experienced developers to articulate a programming strategy and others to use these strategies while providing feedback. Throughout, we asked strategy authors and users to reflect on the challenges they faced. Our analysis revealed that developers could share strategic knowledge. However, they struggled in choosing a level of detail and understanding the diversity of the potential audience. While authors required substantial feedback, users struggled to give it and authors to interpret it. Our results suggest that sharing strategic knowledge differs from sharing code and raises challenging questions about how knowledge-sharing platforms should support search and feedback.

The layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, have mismatches with their visual representation, or consists of generic or app-specific types that are difficult to analyze and model. In this paper, we propose the CLAY pipeline that uses a deep learning approach for denoising UI layouts, allowing us to automatically improve existing mobile UI layout datasets at scale. Our pipeline takes both the screenshot and the raw UI layout, and annotates the raw layout by removing incorrect nodes and assigning a semantically meaningful type to each node. To experiment with our data-cleaning pipeline, we create the CLAY dataset of 59,555 human-annotated screen layouts, based on screenshots and raw layouts from Rico, a public mobile UI corpus. Our deep models achieve high accuracy with F1 scores of 82.7% for detecting layout objects that do not have a valid visual representation and 85.9% for recognizing object types, which significantly outperforms a heuristic baseline. Our work lays a foundation for creating large-scale high quality UI layout datasets for data-driven mobile UI research and reduces the need of manual labeling efforts that are prohibitively expensive.

Developers perform online sensemaking on a daily basis, such as researching and choosing libraries and APIs. Prior research has introduced tools that help developers capture information from various sources and organize it into structures useful for subsequent decision-making. However, it remains a laborious process for developers to manually identify and clip content, maintaining its provenance and synthesizing it with other content. In this work, we introduce a new system called Crystalline that automatically collects and organizes information into tabular structures as the user searches and browses the web. It leverages natural language processing to automatically group similar criteria together to reduce clutter, and uses passive behavioral signals such as mouse movement and dwell time to infer what information to collect and how to visualize and prioritize it. Our user study suggests that developers are able to create comparison tables about 20% faster with a 60% reduction in operational cost without sacrificing the quality of the tables.

Modern software development requires developers to find and effectively utilize new APIs and their documentation, but documentation has many well-known issues. Despite this, developers eventually overcome these issues but have no way of sharing what they learned. We investigate sharing this documentation-specific information through annotations, which have advantages over developer forums as the information is contextualized, not disruptive, and is short, thus easy to author. Developers can also author annotations to support their own comprehension. In order to support the documentation usage behaviors we found, we built the Adamite annotation tool, which provides features such as multiple anchors, annotation types, and pinning. In our user study, we found that developers are able to create annotations that are useful to themselves and are able to utilize annotations created by other developers when learning a new API, with readers of the annotations completing 67% more of the task, on average, than the baseline.

During recent crises like COVID-19, microblogging platforms have become popular channels for affected people seeking assistance such as medical supplies and rescue operations from emergency responders and the public. Despite this common practice, the affordances of microblogging services for help-seeking during crises that needs immediate attention are not well understood. To fill this gap, we analyzed 8K posts from COVID-19 patients or caregivers requesting urgent medical assistance on Weibo, the largest microblogging site in China. Our mixed-methods analyses suggest that existing microblogging functions need to be improved in multiple aspects to sufficiently facilitate help-seeking in emergencies, including capabilities of search and tracking requests, ease of use, and privacy protection. We also find that people tend to stick to certain well-established functions for publishing requests, even after better alternatives emerge. These findings have implications for designing microblogging tools to better support help requesting and responding during crises.

Previous qualitative work has documented that platform workers place an immense importance on their reputation due to the use of algorithmic management by online labor platforms. We provide a general experimental method, which can be used across platforms and time, for numerically quantifying the intensity with which platform workers experience reputation system-based algorithmic management. Our method works via an experiment where workers choose between a monetary bonus or a positive review. We demonstrate this method by measuring the value that freelancers assigned to positive feedback on Upwork in June 2020. The median freelancer in our sample valued a single positive review at ∼ $49 USD. We also find that less experienced freelancers valued a positive review more highly than those with more experience. Qualitative data collected during the experiment indicates that many freelancers considered issues related to reputation system-based algorithmic management while choosing between the monetary reward and the positive review.

Digital lutherie is a sub-domain of digital craft focused on creating digital musical instruments: high-performance devices for musical expression. It represents a nuanced and challenging area of human-computer interaction that is well established and mature, offering the opportunity to observe designers’ work on highly demanding human-computer interfaces. This paper explores how and why digital luthiers choose their tools and how these tools relate to the challenges they face. Findings from 27 standardised open-ended interviews with prominent digital luthiers from commercial, research, independent and artistic backgrounds are analysed through reflexive thematic analysis. Our discussion explores their perspectives, finding that a process of pragmatic rationalisation and environmental influences play a significant role in tool selection. We also present how challenges faced by digital luthiers relate to social creativity and meta-design. These findings build upon the existing literature that examines the designer-tool relationship.

HCI researchers have been gradually shifting attention from individual users to communities when engaging in research, design, and system development. However, our field has yet to establish a cohesive, systematic understanding of the challenges, benefits, and commitments of community-collaborative approaches to research. We conducted a systematic review and thematic analysis of 47 computing research papers discussing participatory research with communities for the development of technological artifacts and systems, published over the last two decades. From this review, we identified seven themes associated with the evolution of a project: from establishing community partnerships to sustaining results. Our findings suggest that several tensions characterize these projects, many of which relate to the power and position of researchers, and the computing research environment, relative to community partners. We discuss the implications of our findings and offer methodological proposals to guide HCI, and computing research more broadly, towards practices that center communities.

It is challenging for customers to select appearance building products (e.g., skincare products, weight loss programs) that suit them personally as such products usually demonstrate efficacy only after long-term usage. Although e-retailers generally provide product descriptions or other customers’ reviews, users often find it hard to relate to their own situations. In this work, we proposed a pipeline to display envisioned users’ appearance after long-term use of appearance building products to deliver their efficacy on each individual visually. We selected skincare as a case and developed SkincareMirror which predicts skincare effects on users’ facial images by analyzing product function labels, efficacy ratings, and skin models’ images. The results of a between-subjects study (N=48) show that (1) SkincareMirror outperforms the baseline shopping site in terms of perceived usability, usefulness, user satisfaction and helps users select products faster; (2) SkincareMirror is especially effective to males and users with limited product domain knowledge.

The COVID-19 pandemic has led governments worldwide to introduce various measures restricting human activity and mobility. Along with the administration of COVID-19 vaccinations and rapid testing, socio-technological solutions such as digital COVID-19 certificates have been considered as a strategy to lessen these restrictions and allow the resumption of routine activities. Using a mixed-methods approach – a survey (n=1008) and 27 semi-structured interviews – this study explores the attitudes of residents in the Republic of Ireland towards the idea of introducing digital COVID-19 certificates. We examine the topics of acceptability, fairness, security and privacy of COVID-related personal data, and practical considerations for implementation. Our study reveals the conditional and contextual nature of the acceptability of digital certificates, identifying specific factors that affect it, associated data practices, and related public concerns and expectations of such technologies.

Contact tracers assist in containing the spread of highly infectious diseases such as COVID-19 by engaging community members who receive a positive test result in order to identify close contacts. Many contact tracers rely on community member’s recall for those identifications, and face limitations such as unreliable memory. To investigate how technology can alleviate this challenge, we developed a visualization tool using de-identified location data sensed from campus WiFi and provided it to contact tracers during mock contact tracing calls. While the visualization allowed contact tracers to find and address inconsistencies due to gaps in community member’s memory, it also introduced inconsistencies such as false-positive and false-negative reports due to imperfect data, and information sharing hesitancy. We suggest design implications for technologies that can better highlight and inform contact tracers of potential areas of inconsistencies, and further present discussion on using imperfect data in decision making.

Digital contact tracing can limit the spread of infectious diseases. Nevertheless, barriers remain to attain sufficient adoption. In this study, we investigate how willingness to participate in contact tracing is affected by two critical factors: the modes of data collection and the type of data collected. We conducted a scenario-based survey study among 220 respondents in the United States (U.S.) to understand their perceptions about contact tracing associated with automated and manual contact tracing methods. The findings indicate a promising use of smartphones and a combination of public health officials and medical health records as information sources. Through a quantitative analysis, we describe how different modalities and individual demographic factors may affect user compliance when participants are asked to provide four key information pieces for contact tracing.

During crises like COVID-19, individuals are inundated with conflicting and time-sensitive information that drives a need for rapid assessment of the trustworthiness and reliability of information sources and platforms. This parallels evolutions in information infrastructures, ranging from social media to government data platforms. Distinct from current literature, which presumes a static relationship between the presence or absence of trust and people’s behaviors, our mixed-methods research focuses on situated trust, or trust that is shaped by people’s information-seeking and assessment practices through emerging information platforms (e.g., social media, crowdsourced systems, COVID data platforms). Our findings characterize the shifts in trustee (what/who people trust) from information on social media to the social media platform(s), how distrust manifests skepticism in issues of data discrepancy, the insufficient presentation of uncertainty, and how this trust and distrust shift over time. We highlight the deep challenges in existing information infrastructures that influence trust and distrust formation.

COVID-19 has demonstrated the importance of digital contact tracing apps in reducing the spread of disease. Despite people widely expressing interest in using contact tracing apps, actual installation rates have been low in many parts of the world. Prior studies suggest that decisions to use these apps are largely shaped by pandemic beliefs, social influences, perceived benefits and harms, and other factors. However, there is a gap in understanding what factors motivate intention, but not subsequent behavior of actual adoption. Reporting on a survey of 290 U.S. residents, we disentangle the intention-behavior gap by investigating factors associated with installing a contact tracing app from those associated with intending to install, but not actually installing. Our results suggest that social norms can be leveraged to span the intention-behavior gap, and that a privacy paradox may influence people’s adoption decisions. We present recommendations for technologies that enlist individuals to address collective challenges.

Deceptive visualizations are visualizations that, whether intentionally or not, lead the reader to an understanding of the data which varies from the actual data. Examples of deceptive visualizations can be found in every digital platform, and, despite their widespread use in the wild, there have been limited efforts to alert laypersons to common deceptive visualization practices. In this paper, we present a tool for annotating line charts in the wild that reads line chart images and outputs text and visual annotations to assess the line charts for distortions and help guide the reader towards an honest understanding of the chart data. We demonstrate the usefulness of our tool through a series of case studies on real-world charts. Finally, we perform a crowdsourced experiment to evaluate the ability of the proposed tool to educate readers about potentially deceptive visualization practices.

Designing a data physicalization requires a myriad of different considerations. Despite the cross-disciplinary nature of these considerations, research currently lacks a synthesis across the different communities data physicalization sits upon, including their approaches, theories, and even terminologies. To bridge these communities synergistically, we present a design space that describes and analyzes physicalizations according to three facets: context (end-user considerations), structure (the physical structure of the artifact), and interactions (interactions with both the artifact and data). We construct this design space through a systematic review of 47 physicalizations and analyze the interrelationships of key factors when designing a physicalization. This design space cross-pollinates knowledge from relevant HCI communities, providing a cohesive overview of what designers should consider when creating a data physicalization while suggesting new design possibilities. We analyze the design decisions present in current physicalizations, discuss emerging trends, and identify underlying open challenges.

Physicalizations represent data through their tangible and material properties. In contrast to screen-based visualizations, there is currently very limited understanding of how to label or annotate physicalizations to support people in interpreting the data encoded by the physicalization. Because of its spatiality, contextualization through labeling or annotation is crucial to communicate data across different orientations. In this paper, we study labeling approaches as part of the overall construction process of bar chart physicalizations. We designed a toolkit of physical tokens and paper data labels and asked 16 participants to construct and contextualize their own data physicalizations. We found that (i) the construction and contextualization of physicalizations is a highly intertwined process, (ii) data labels are integrated with physical constructs in the final design, and (iii) these are both influenced by orientation changes. We contribute with an understanding of the role of data labeling in the creation and contextualization of physicalizations.

Data visualizations are now widely used across many disciplines. However, many of them are not easily accessible for visually impaired people. In this work, we use three-staged mixed methods to understand the current practice of accessible visualization design for visually impaired people. We analyzed 95 visualizations from various venues to inspect how they are made inaccessible. To understand the rationale and context behind the design choices, we also conducted surveys with 144 practitioners in the U.S. and follow-up interviews with ten selected survey participants. Our findings include the difficulties of handling modern complex and interactive visualizations and the lack of accessibility support from visualization tools in addition to personal and organizational factors making it challenging to perform accessible design practices.

Matrix visualizations are widely used to display large-scale network, tabular, set, or sequential data. They typically only encode a single value per cell, e.g., through color. However, this can greatly limit the visualizations’ utility when exploring multivariate data, where each cell represents a data point with multiple values (referred to as details). Three well-established interaction approaches can be applicable in multivariate matrix visualizations (or MMV): focus+context, pan&zoom, and overview+detail. However, there is little empirical knowledge of how these approaches compare in exploring MMV. We report on two studies comparing them for locating, searching, and contextualizing details in MMV. We first compared four focus+context techniques and found that the fisheye lens overall outperformed the others. We then compared the fisheye lens, to pan&zoom and overview+detail. We found that pan&zoom was faster in locating and searching details, and as good as overview+detail in contextualizing details.

Massively multiplayer online role-playing games create virtual communities that support heterogeneous “social roles” determined by gameplay interaction behaviors under a specific social context. For all social roles, formal roles are pre-defined, obvious, and explicitly ascribed to the people holding the roles, whereas informal roles are not well-defined and unspoken. Identifying the informal roles and understanding their subtle changes are critical to designing sociability mechanisms. However, it is nontrivial to understand the existence and evolution of such roles due to their loosely defined, interconvertible, and dynamic characteristics. We propose a visual analytics system, RoleSeer, to investigate informal roles from the perspectives of behavioral interactions and depict their dynamic interconversions and transitions. Two cases, experts’ feedback, and a user study suggest that RoleSeer helps interpret the identified informal roles and explore the patterns behind role changes. We see our approach’s potential in investigating informal roles in a broader range of social games.

Movement-based video games can provide engaging play experiences, and also have the potential to encourage physical activity. However, existing design guidelines for such games overwhelmingly focus on non-disabled players. Here, we explore wheelchair users’ perspectives on movement-based games as an enjoyable play activity. We created eight game concepts as discussion points for semi-structured interviews (N=6) with wheelchair users, and used Interpretative Phenomenological Analysis to understand their perspectives on physical activity and play. Themes focus on independent access, challenges in social settings, and the need for comprehensive adaptation. We also conducted an online survey (N=21) using the same game concepts, and thematic analysis highlighted the importance of adequate challenge, and considerations around multiplayer experiences. Based on these findings, we re-contextualize and expand guidelines for movement-based games previously established by Mueller and Isbister to include disabled players, and suggest design strategies that take into account their perspectives on play.

Integrating fabrication activities into existing video games provides opportunities for players to construct objects from their gameplay and bring the digital content into the physical world. In our prior work, we outlined a framework and developed a toolkit for integrating fabrication activities within existing digital games. Insights from our prior study highlighted the challenge of aligning fabrication mechanics with the existing game mechanics in order to strengthen the player aesthetics.

In this paper, we address this challenge and build on our prior work by adding fabrication components to the Mechanics-Dynamics-Aesthetics (MDA) framework. We use this f-MDA framework to analyze the 47 fabrication events from the prior study. We list the new player-object aesthetics that emerge from integrating the existing game mechanics with fabrication mechanics. We identify connections between these emergent player-object aesthetics and the existing game mechanics. We discuss how designers can use this mapping to identify potential game mechanics for integrating with fabrication activities.

Dynamically drawn content (e.g., handwritten text) in learning videos is believed to improve users’ engagement and learning over static powerpoint-based ones. However, evidence from existing literature is inconclusive. With the emergence of Optical Head-Mounted Displays (OHMDs), recent work has shown that video learning can be adapted for on-the-go scenarios. To better understand the role of dynamic drawing, we decoupled dynamically drawn text into two factors (font style and motion of appearance) and studied their impact on learning performance under two usage scenarios (while seated with desktop and walking with OHMD). We found that although letter-traced text was more engaging for some users, most preferred learning with typeface text that displayed the entire word at once and achieved better recall (46.7% higher), regardless of the usage scenarios. Insights learned from the studies can better inform designers on how to present text in videos for ubiquitous access.

Instructional videos for physical training have gained popularity in recent years among sport and fitness practitioners, due to the proliferation of affordable and ubiquitous forms of online training. Yet, learning movement this way poses challenges: lack of feedback and personalised instructions, and having to rely on personal imitation capacity to learn movements. We address some of these challenges by exploring visual cues’ potential to help people imitate movements from instructional videos. With a Research through Design approach, focused on strength training, we augmented an instructional video with different sets of visual cues: directional cues, body highlights, and metaphorical visualizations. We tested each set with ten practitioners over three recorded sessions, with follow-up interviews. Through thematic analysis, we derived insights on the effect of each set of cues for supporting movement learning. Finally, we generated design takeaways to inform future HCI work on visual cues for instructional training videos.

How-to videos are often shot using camera angles that may not be optimal for learning motor tasks, with a prevalent use of third-person perspective. We present immersivePOV, an approach to film how-to videos from an immersive first-person perspective using a head-mounted 360° action camera. immersivePOV how-to videos can be viewed in a Virtual Reality headset, giving the viewer an eye-level viewpoint with three Degrees of Freedom. We evaluated our approach with two everyday motor tasks against a baseline first-person perspective and a third-person perspective. In a between-subjects study, participants were assigned to watch the task videos and then replicate the tasks. Results suggest that immersivePOV reduced perceived cognitive load and facilitated task learning. We discuss how immersivePOV can also streamline the video production process for content creators. Altogether, we conclude that immersivePOV is an effective approach to film how-to videos for learners and content creators alike.

Most video-based learning content is designed for desktops without considering mobile environments. We (1) investigate the gap between mobile learners’ challenges and video engineers’ considerations using mixed methods and (2) provide design guidelines for creating mobile-friendly MOOC videos. To uncover learners’ challenges, we conducted a survey (n=134) and interviews (n=21), and evaluated the mobile adequacy of current MOOCs by analyzing 41,722 video frames from 101 video lectures. Interview results revealed low readability and situationally-induced impairments as major challenges. The content analysis showed a low guideline compliance rate for key design factors. We then interviewed 11 video production engineers to investigate design factors they mainly consider. The engineers mainly focus on the size and amount of content while lacking consideration for color, complex images, and situationally-induced impairments. Finally, we present and validate guidelines for designing mobile-friendly MOOCs, such as providing adaptive and customizable visual design and context-aware accessibility support.

Labeled datasets are essential for supervised machine learning. Various data labeling tools have been built to collect labels in different usage scenarios. However, developing labeling tools is time-consuming, costly, and expertise-demanding on software development. In this paper, we propose a conceptual framework for data labeling and OneLabeler based on the conceptual framework to support easy building of labeling tools for diverse usage scenarios. The framework consists of common modules and states in labeling tools summarized through coding of existing tools. OneLabeler supports configuration and composition of common software modules through visual programming to build data labeling tools. A module can be a human, machine, or mixed computation procedure in data labeling. We demonstrate the expressiveness and utility of the system through ten example labeling tools built with OneLabeler. A user study with developers provides evidence that OneLabeler supports efficient building of diverse data labeling tools.

Graph analytics is currently performed using a combination of code, symbolic algebra, and network visualizations. The analyst has to work with symbolic and abstract forms of data to construct and analyze graphs. We locate unique design opportunities at the intersection of computer vision and graph analytics, by utilizing visual variables extracted from images/videos and some direct manipulation and pen interaction techniques. We also summarize commonly used graph operations and graphical representations (graphs, simplicial complexes, hypergraphs), and map them to a few brushes and direct manipulation actions. The mapping enables us to visually construct and analyze a wide range of graphs on top of images, videos, and sketches. The design framework is implemented as a sketch-based notebook interface to demonstrate the design possibilities. User studies with scientists from various fields reveal innovative use cases for such an embodied interaction paradigm for graph analytics.

Data documents play a central role in recording, presenting, and disseminating data. Despite the proliferation of applications and systems designed to support the analysis, visualization, and communication of data, writing data documents remains a laborious process, requiring a constant back-and-forth between data processing and writing tools. Interviews with eight professionals revealed that their workflows contained numerous tedious, repetitive, and error-prone operations. The key issue that we identified is the lack of persistent connection between text and data. Thus, we developed CrossData, a prototype that treats text-data connections as persistent, interactive, first-class objects. By automatically identifying, establishing, and leveraging text-data connections, CrossData enables rich interactions to assist in the authoring of data documents. An expert evaluation with eight users demonstrated the usefulness of CrossData, showing that it not only reduced the manual effort in writing data documents but also opened new possibilities to bridge the gap between data exploration and writing.

Online learners are hugely diverse with varying prior knowledge, but most instructional videos online are created to be one-size-fits-all. Thus, learners may struggle to understand the content by only watching the videos. Providing scaffolding prompts can help learners overcome these struggles through questions and hints that relate different concepts in the videos and elicit meaningful learning. However, serving diverse learners would require a spectrum of scaffolding prompts, which incurs high authoring effort. In this work, we introduce Promptiverse, an approach for generating diverse, multi-turn scaffolding prompts at scale, powered by numerous traversal paths over knowledge graphs. To facilitate the construction of the knowledge graphs, we propose a hybrid human-AI annotation tool, Grannotate. In our study (N=24), participants produced 40 times more on-par quality prompts with higher diversity, through Promptiverse and Grannotate, compared to hand-designed prompts. Promptiverse presents a model for creating diverse and adaptive learning experiences online.

Data science is characterized by evolution: since data science is exploratory, results evolve from moment to moment; since it can be collaborative, results evolve as the work changes hands. While existing tools help data scientists track changes in code, they provide less support for understanding the iterative changes that the code produces in the data. We explore the idea of visualizing differences in datasets as a core feature of exploratory data analysis, a concept we call Diff in the Loop (DITL). We evaluated DITL in a user study with 16 professional data scientists and found it helped them understand the implications of their actions when manipulating data. We summarize these findings and discuss how the approach can be generalized to different data science workflows.

While much work is underway within the context of posthuman design, this research is often described from a dominantly human perspective. It rarely accounts for the creative capacities of nonhumans in design, such as materials, tools, and software. There is a need to further engage with posthuman theories conceptually, materially, and methodologically. We approach this challenge through Ron Wakkary's concept of repertoires: actions the human designer can take to increase participation of nonhumans in design research practice. This paper reports on potential repertoires' development by exploring three approaches from outside of HCI: describing the landscape, noticing, and translations. We use these methods to account for weaving events that the first author was engaged in. Through critical reflection of these accounts, we contribute three repertoires and an example of applying the theoretical framework of Designing Things.

Flavobacteria, which can be found in marine environments, are able to grow in highly organized colonies producing vivid iridescent colorations. While much is known about the biology of these organisms, their design potential as responsive media in user interfaces has not been explored. Our paper aims at bridging this gap by providing insights into the type, degree, and duration of change in Flavobacteria’s expression, i.e., their living aesthetics. We present a tool to capture and characterize these changes concerning form, texture and iridescent color. To support the long-term study of their living aesthetics, we designed Flavorium. This bio-digital artifact provides the necessary habitat conditions for Flavobacteria to thrive for a month. Granting insights into the responsive behavior of this organism, this work presents a design space, vocabulary, and application concepts to inspire HCI and design scholars to investigate the complex temporal qualities of living media for future user interfaces.

Sustainability is critical to our planet and thus our designs. Within HCI, there is a tension between the desire to create interactive electronic systems and sustainability. In this paper, we present the design of an interactive system comprising components that are entirely decomposable. We leverage the inherent material properties of natural materials, such as paper, leaf skeletons, and chitosan, along with silver nanowires to create a new system capable of being electrically controlled as a portable heater. This new decomposable system, capable of wirelessly heating to >70°C, is flexible, lightweight, low-cost, and reusable, and it maintains its functionality over long periods of heating and multiple power cycles. We detail its design and present a series of use cases, from enabling a novel resealable packaging system to acting as a catalyst for shape-changing designs and beyond. Finally, we highlight the important decomposable property of the interactive system when it meets end-of-life.

Given the ongoing environmental crisis and recent calls within HCI to engage with its cascading effects on the more-than-human world, this paper introduces the concept of the eco-technical interface as a critical zone at which designers can surface and subvert issues of multispecies relations such as nonhuman instrumentalization. The eco-technical interface represents the sites at which human, nonhuman, and technological interfaces overlap, ranging from remote sensing for conservation to smart devices for precision agriculture to community science platforms for species identification. Here, we highlight the pervasiveness of the eco-technical interface as a set of sites for further HCI inquiry, engage with the politics and instrumentalizing tendencies at three particular sites, and demonstrate tactics for cultivating attunement to, reflexively accounting for, and subverting instrumentalization in multispecies encounter.

Little is known about non-Western social media users’ motivations for adopting behaviors that protect them against pervasive threats to their privacy, security, and personal well-being. Drawing on Rogers’ Protection Motivation Theory (PMT), this survey study explores Caribbean people’s (N=551) perceptions of safety threats and the factors contributing to their intention to adopt protective behaviors. Our analysis revealed that prior victimization was associated with increased perceptions of vulnerability and severity of harms, which, in turn, influenced elevated safety protection behaviors. For harassment-related harms in particular, participants’ trust in social media sites increased their intention to adopt protective behaviors. We observe significant country-to-country differences, which we contextualize through interviews with experts throughout the region. Our findings provide a new understanding of users’ mental models, behaviors, and attitudes with respect to online safety. We conclude by discussing theoretical and practical implications and outline opportunities for the design of inclusive and culturally-aware safety tools.

Deepfakes are synthetic content generated using advanced deep learning and AI technologies. The advancement of technology has created opportunities for anyone to create and share deepfakes much easier. This may lead to societal concerns based on how communities engage with it. However, there is limited research available to understand how communities perceive deepfakes. We examined deepfake conversations on Reddit from 2018 to 2021—including major topics and their temporal changes as well as implications of these conversations. Using a mixed-method approach—topic modeling and qualitative coding, we found 6,638 posts and 86,425 comments discussing concerns of the believable nature of deepfakes and how platforms moderate them. We also found Reddit conversations to be pro-deepfake and building a community that supports creating and sharing deepfake artifacts and building a marketplace regardless of the consequences. Possible implications derived from qualitative codes indicate that deepfake conversations raise societal concerns. We propose that there are implications for Human Computer Interaction (HCI) to mitigate the harm created from deepfakes.

In January 2021, WhatsApp announced an update to their privacy policy, sparking an outcry that saw millions of users install other messaging apps such as Telegram and Signal. This presented a rare opportunity to study users’ experiences when trying to leave the world’s most popular communication app. We conducted surveys in February and May with 1525 WhatsApp users from Mexico, Spain, South Africa, and the United Kingdom. Over a quarter wanted to switch at least part of their communication to other apps, but 74% of them failed to do so. By May, 27% had increased their use of other apps, and only 16% used WhatsApp less. Beyond network effects, users struggled with making informed choices of alternative apps and with differences in their design and functionality. We suggest messaging interoperability as an approach to alleviate switching costs and discuss implications for HCI research and competition regulation of digital services.

We explore privacy perceptions, beliefs and practices of low-literate, low-income users in Pakistan, a patriarchal and religious context with a literacy rate of approx. 68% and where 59% of mobile users have less than 6 years of formal education. Through a qualitative study with 40 participants (17 male and 23 female) we examine the cultural, religious, and familial structures that impact users perceptions, management, and control of their personal privacy. We reveal significant gendered differences in privacy understandings, privacy preserving practices and the access to privacy related knowledge. Our work also highlights the seminal impact religious beliefs have on men and women’s understandings and management of privacy and the prolific use of after-market modified apps to support users specific privacy needs. The privacy concerns raised by our participants provide HCI researchers with valuable insights into designing privacy affordances for vulnerable and diverse populations beyond Western, educated, industrialized, rich and democratic contexts.

In our data-centric world, most services rely on collecting and using personal data. The EU's General Data Protection Regulation (GDPR) aims to enhance individuals’ control over their data, but its practical impact is not well understood. We present a 10-participant study, where each participant filed 4-5 data access requests. Through interviews accompanying these requests and discussions scrutinising returned data, it appears that GDPR falls short of its goals due to non-compliance and low-quality responses. Participants found their hopes to understand providers’ data practices or harness their own data unmet. This causes increased distrust without any subjective improvement in power, although more transparent providers do earn greater trust. We propose designing more effective, data-inclusive and open policies and data access systems to improve both customer relations and individual agency, and also that wider public use of GDPR rights could help with delivering accountability and motivating providers to improve data practices.

High-fidelity driving simulators can act as testbeds for designing in-vehicle interfaces or validating the safety of novel driver assistance features. In this system paper, we develop and validate the safety of a mixed reality driving simulator system that enables us to superimpose virtual objects and events into the view of participants engaging in real-world driving in unmodified vehicles. To this end, we have validated the mixed reality system for basic driver cockpit and low-speed driving tasks, comparing the use of the system with non-headset and with the headset driving conditions, to ensure that participants behave and perform similarly using this system as they would otherwise. This paper outlines the operational procedures and protocols for using such systems for cockpit tasks (like using the parking brake, reading the instrument panel, and turn signaling) as well as basic low-speed driving exercises (such as steering around corners, weaving around obstacles, and stopping at a fixed line) in ways that are safe, effective, and lead to accurate, repeatable data collection about behavioral responses in real-world driving tasks.

We present a new and practical method for capturing user body pose in virtual reality experiences: integrating cameras into handheld controllers, where batteries, computation and wireless communication already exist. By virtue of the hands operating in front of the user during many VR interactions, our controller-borne cameras can capture a superior view of the body for digitization. Our pipeline composites multiple camera views together, performs 3D body pose estimation, uses this data to control a rigged human model with inverse kinematics, and exposes the resulting user avatar to end user applications. We developed a series of demo applications illustrating the potential of our approach and more leg-centric interactions, such as balancing games and kicking soccer balls. We describe our proof-of-concept hardware and software, as well as results from our user study, which point to imminent feasibility.

This paper presents Centaur, an input system that enables tangible interaction on displays, e.g., untouchable computer monitors. Centaur’s tangibles are built from low-cost optical mouse sensors, or can alternatively be emulated by commercial optical mice already available. They are trackable when put on the display, rendering a real-time and high-precision tangible interface. Even for ordinary personal computers, enabling Centaur requires no new hardware and installation burden. Centaur’s cost-effectiveness and wide availability open up new opportunities for tangible user interface (TUI) users and practitioners. Centaur’s key innovation lies in its tracking method. It embeds high-frequency light signals into different portions of the display content as location beacons. When the tangibles are put on the screen, they are able to sense the light signals with their optical mouse sensors, and thus determine the locations accordingly. We develop four applications to showcase the potential usage of Centaur.

Conductive thread is a common material in e-textile toolkits that allows practitioners to create connections between electronic components sewn on fabric. When powered, conductive threads are used as resistive heaters to activate thermochromic dyes or pigments on textiles to create interactive, aesthetic, and ambient textile displays. In this work, we introduce Embr, a creative framework for supporting hand-embroidered liquid crystal textile displays (LCTDs). This framework includes a characterization of conductive embroidery stitches, an expanded repertoire of thermal formgiving techniques, and a thread modeling tool used to simulate mechanical, thermal, and electrical behaviors of LCTDs. Through exemplar artifacts, we annotate a morphological design space of LCTDs and discuss the tensions and opportunities of satisfying the wider range of electrical, craft, cultural, aesthetic, and functional concerns inherent to e-textile practices.

Online synchronous tutoring allows for immediate engagement between instructors and audiences over distance. However, tutoring physical skills remains challenging because current telepresence approaches may not allow for adequate spatial awareness, viewpoint control of the demonstration activities scattered across an entire work area, and the instructor’s sufficient awareness of the audience. We present Asteroids, a novel approach for tangible robotic telepresence, to enable workbench-scale physical embodiments of remote people and tangible interactions by the instructor. With Asteroids, the audience can actively control a swarm of mini-telepresence robots, change camera positions, and switch to other robots’ viewpoints. Demonstrators can perceive the audiences’ physical presence while using tangible manipulations to control the audience’s viewpoints and presentation flow. We conducted an exploratory evaluation for Asteroids with 12 remote participants in a model-making tutorial scenario with an architectural expert demonstrator. Results suggest our unique features benefitted participants’ engagement, sense of presence, and understanding.

Designers reportedly struggle with design optimization tasks where they are asked to find a combination of design parameters that maximizes a given set of objectives. In HCI, design optimization problems are often exceedingly complex, involving multiple objectives and expensive empirical evaluations. Model-based computational design algorithms assist designers by generating design examples during design, however they assume a model of the interaction domain. Black box methods for assistance, on the other hand, can work with any design problem. However, virtually all empirical studies of this human-in-the-loop approach have been carried out by either researchers or end-users. The question stands out if such methods can help designers in realistic tasks. In this paper, we study Bayesian optimization as an algorithmic method to guide the design optimization process. It operates by proposing to a designer which design candidate to try next, given previous observations. We report observations from a comparative study with 40 novice designers who were tasked to optimize a complex 3D touch interaction technique. The optimizer helped designers explore larger proportions of the design space and arrive at a better solution, however they reported lower agency and expressiveness. Designers guided by an optimizer reported lower mental effort but also felt less creative and less in charge of the progress. We conclude that human-in-the-loop optimization can support novice designers in cases where agency is not critical.

As algorithms are increasingly augmenting and substituting human decision-making, understanding how the introduction of computational agents changes the fundamentals of human behavior becomes vital. This pertains to not only users, but also those parties who face the consequences of an algorithmic decision. In a controlled experiment with 480 participants, we exploit an extended version of two-player ultimatum bargaining where responders choose to bargain with either another human, another human with an AI decision aid or an autonomous AI-system acting on behalf of a passive human proposer. Our results show strong responder preferences against the algorithm, as most responders opt for a human opponent and demand higher compensation to reach a contract with autonomous agents. To map these preferences to economic expectations, we elicit incentivized subject beliefs about their opponent’s behavior. The majority of responders maximize their expected value when this is line with approaching the human proposer. In contrast, responders predicting income maximization for the autonomous AI-system overwhelmingly override economic self-interest to avoid the algorithm.

People consider recommendations from AI systems in diverse domains ranging from recognizing tumors in medical images to deciding which shoes look cute with an outfit. Implicit in the decision process is the perceived expertise of the AI system. In this paper, we investigate how people trust and rely on an AI assistant that performs with different levels of expertise relative to the person, ranging from completely overlapping expertise to perfectly complementary expertise. Through a series of controlled online lab studies where participants identified objects with the help of an AI assistant, we demonstrate that participants were able to perceive when the assistant was an expert or non-expert within the same task and calibrate their reliance on the AI to improve team performance. We also demonstrate that communicating expertise through the linguistic properties of the explanation text was effective, where embracing language increased reliance and distancing language reduced reliance on AI.

Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks ranging from online comment toxicity to misinformation detection to medical diagnosis, different groups in society may have irreconcilable disagreements about ground truth labels. Supervised ML today resolves these label disagreements implicitly using majority vote, which overrides minority groups’ labels. We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier’s prediction. For example, a jury learning model for online toxicity might centrally feature women and Black jurors, who are commonly targets of online harassment. To enable jury learning, we contribute a deep learning architecture that models every annotator in a dataset, samples from annotators’ models to populate the jury, then runs inference to classify. Our architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent. A field evaluation finds that practitioners construct diverse juries that alter 14% of classification outcomes.

Is recommendation the new search? Recommender systems have shortened the search for information in everyday activities such as following the news, media, and shopping. In this paper, we address the challenges of capturing the situational needs of the user and linking them to the available datasets with the concept of Mindsets. Mindsets are categories such as “I’m hungry” and “Surprise me” designed to lead the users to explicitly state their intent, control the recommended content, save time, get inspired, and gain shortcuts for a satisficing exploration of POI recommendations. In our methodology, we first compiled Mindsets with a card sorting workshop and a formative evaluation. Using the insights gathered from potential end users, we then quantified Mindsets by linking them to POI utility measures using approximated lexicographic multi-objective optimisation. Finally, we ran a summative evaluation of Mindsets and derived guidelines for designing novel categories for recommender systems.

Street harassment is a widespread problem that can constrain people’s freedom to enjoy public spaces safely, along with many other negative psychological impacts. However, very little research has looked at how immersive technology can help in addressing it. We conducted three studies to investigate the design decisions, ethical issues and efficacy of an immersive simulation of street harassment: an online design study (n=20), an interview study with experts working in the area (n=9), and a comparative lab study investigating design, ethics and efficacy (n=44). Our results deepen understanding of the design decisions that contribute to a realistic psychological experience, such as the effects of screen-based video vs passive VR vs interactive VR. They also highlight important ethical issues such as traumatisation and potential for victim blaming, and how they can be approached in an ethical manner. Finally, they provide insights into efficacy in terms of perceived usefulness, competence and empathy.

The contemporary understanding of gender continues to highlight the complexity and variety of gender identities beyond a binary dichotomy regarding one’s biological sex assigned at birth. The emergence and popularity of various online social spaces also makes the digital presentation of gender even more sophisticated. In this paper, we use non-cisgender as an umbrella term to describe diverse gender identities that do not match people’s sex assigned at birth, including Transgender, Genderfluid, and Non-binary. We especially explore non-cisgender individuals’ identity practices and their challenges in novel social Virtual Reality (VR) spaces where they can present, express, and experiment their identity in ways that traditional online social spaces cannot provide. We provide one of the first empirical evidence of how social VR platforms may introduce new and novel phenomena and practices of approaching diverse gender identities online. We also contribute to re-conceptualizing technology-supported identity practices by highlighting the role of (re)discovering the physical body online and informing the design of the emerging metaverse for supporting diverse gender identities in the future.

Smartphone addiction refers to the problematic use of smartphones, which can negatively impact one’s quality of life and even health. We conducted a two-week technology probe study to explore the use of technologies aimed at improving smartphone addiction among seven dyads of adolescents and their parents. Interviews conducted during and after the probe study revealed that manually reporting lifestyle and well-being data could provide motivation to improve one’s lifestyle and well-being by moderating phone use. Sharing smartphone use data with parents was also shown to head off negative communication loops and foster opportunities to overcome the smartphone addiction.

Caseworkers are trained to write detailed narratives about families in Child-Welfare (CW) which informs collaborative high-stakes decision-making. Unlike other administrative data, these narratives offer a more credible source of information with respect to workers’ interactions with families as well as underscore the role of systemic factors in decision-making. SIGCHI researchers have emphasized the need to understand human discretion at the street-level to be able to design human-centered algorithms for the public sector. In this study, we conducted computational text analysis of casenotes at a child-welfare agency in the midwestern United States and highlight patterns of invisible street-level discretionary work and latent power structures that have direct implications for algorithm design. Casenotes offer a unique lens for policymakers and CW leadership towards understanding the experiences of on-the-ground caseworkers. As a result of this study, we highlight how street-level discretionary work needs to be supported by sociotechnical systems developed through worker-centered design. This study offers the first computational inspection of casenotes and introduces them to the SIGCHI community as a critical data source for studying complex sociotechnical systems.

Content creators—social media personalities with large audiences on platforms like Instagram, TikTok, and YouTube—face a heightened risk of online hate and harassment. We surveyed 135 creators to understand their personal experiences with attacks (including toxic comments, impersonation, stalking, and more), the coping practices they employ, and gaps they experience with existing solutions (such as moderation or reporting). We find that while a majority of creators view audience interactions favorably, nearly every creator could recall at least one incident of hate and harassment, and attacks are a regular occurrence for one in three creators. As a result of hate and harassment, creators report self-censoring their content and leaving platforms. Through their personal stories, their attitudes towards platform-provided tools, and their strategies for coping with attacks and harms, we inform the broader design space for how to better protect people online from hate and harassment.

We know surprisingly little about the prevalence and severity of cybercrime in the U.S. Yet, in order to prioritize the development and distribution of advice and technology to protect end users, we require empirical evidence regarding cybercrime. Measuring crime, including cybercrime, is a challenging problem that relies on a combination of direct crime reports to the government – which have known issues of under-reporting – and assessment via carefully-designed self-report surveys. We report on the first large-scale, nationally representative academic survey (n=11,953) of consumer cybercrime experiences in the U.S. Our analysis answers four research questions: (1) What is the prevalence and (2) the monetary impact of these cybercrimes we measure in the U.S.?, (3) Do inequities exist in victimization?, and (4) Can we improve cybercrime measurement by leveraging social-reporting techniques used to measure physical crime? Our analysis also offers insight toward improving future measurement of cybercrime and protecting users.

Survivors of intimate partner violence (IPV) face complex threats to their digital privacy and security. Prior work has established protocols for directly helping them mitigate these harms; however, there remains a need for flexible and pluralistic systems that can support survivors’ long-term needs. This paper describes the design and development of sociotechnical infrastructure that incorporates feminist notions of care to connect IPV survivors experiencing technology abuse with volunteer computer security consultants. We present findings from a mixed methods study that draws on data from an 8-month, real-world deployment, as well as interviews with 7 volunteer technology consultants and 18 IPV professionals. Our findings illuminate emergent challenges in safely and adaptively providing computer security advice as care. We discuss implications of these findings for feminist approaches to computer security and privacy, and provide broader lessons for interventions that aim to directly assist at-risk and marginalized people experiencing digital insecurity.

The possibility that common users are successfully recruited in cyberattacks represents a considerable vulnerability because it implies that citizens can legitimize cyberattacks instead of condemning them. We propose to adopt an argumentative approach to identify which premises allow such legitimization. To showcase this approach, we created four short narratives describing cyberattacks involving generic users and covering different motives for the attacks: profit, recreation, revenge, and ideology. A sample of 16 participants read the four narratives and was afterward interviewed to express their position on the attacks described. All interview transcripts were then analyzed with an argumentative approach, and 15 premises were found to account for the different positions taken. We describe the premises, their distribution across the four narratives, and discuss the implications of this approach for cybersecurity.

The success of Wikipedia and other user-generated content communities has been driven by the openness of recruiting volunteers globally, but this openness has also led to a persistent lack of trust in its content. Despite several attempts at developing trust indicators to help readers more quickly and accurately assess the quality of content, challenges remain for practical deployment to general consumers. In this work we identify and address three key challenges: empirically determining which metrics from prior and existing community approaches most impact reader trust; 2) validating indicator placements and designs that are both compact yet noticed by readers; and 3) demonstrating that such indicators can not only lower trust but also increase perceived trust in the system when appropriate. By addressing these, we aim to provide a foundation for future tools that can practically increase trust in user generated content and the sociotechnical systems that generate and maintain them.

Play is an essential part of the human experience and can be found throughout the lifespan. While play has long been of interest to the HCI community, research has often focused on the technologies supporting game play, the potential outcomes of play (e.g., skill-building, health improvements), or play among children. This paper explores what play looks like in online communities that are not specifically game-based and consist primarily of adults. From online ethnographic work of the ARMY (i.e., Adorable Representative M.C. for Youth), fandom of the South Korean musical group BTS, we explore how BTS and ARMY collaboratively construct a playful social environment using various social media platforms. A contribution of this work is to expand our conceptualization of how adults create playful places that are not specifically game-based and highlights the role of socio-technical systems in their community building.

In recent years, political crowdfunding campaigns have emerged through which politicians raise money to fund their election campaigns. Divisive issues discussed in these campaigns may not only motivate donations but also could have a broader priming effect on people’s social opinions. In the U.S., more than one-third of the population with moderate opinions show a tendency to swing their opinion based on recent and more accessible events. In this paper, we ask: can such campaigns further prime people’s responses to partisan topics, even when we discuss those topics in a non-political context? To answer this question, we analyzed the influence of exposure to a political candidate’s crowdfunding campaign on responses to a subsequently seen, unrelated scientific topic that is not inherently political but is seen as partisan in the U.S. (climate change). We found that exposure to an attitude-inconsistent political candidate’s crowdfunding campaign (a campaign that is counter to someone’s existing political beliefs) can have a significant priming effect on subsequently seen politically charged topics. This effect may occur due to the activation of in-group identity by the candidate’s partisan campaign. Guided by these findings, we investigated elements that can mitigate this self-categorization effect. We found that carefully designed content following framing techniques such as schema framing and threat/safety framing can mitigate people’s sense of self-categorization toward non-political topics.

Why do some peer production projects do a better job at engaging potential contributors than others? We address this question by comparing three Indian language Wikipedias, namely, —Malayalam, Marathi, and Kannada. We found that although the three projects share goals, technological infrastructure, and a similar set of challenges, Malayalam Wikipedia’s community engages language speakers in contributing at a much higher rate than the others. Drawing from a grounded theory analysis of interviews with 18 community participants from the three projects, we found that experience with participatory governance and free/open-source software in the Malayalam community supported high engagement of contributors. Counterintuitively, we found that financial resources intended to increase participation in the Marathi and Kannada communities hindered the growth of these communities. Our findings underscore the importance of social and cultural context in the trajectories of peer production communities.

Online investigations are increasingly conducted by individuals with diverse skill levels and experiences, with mixed results. Novice investigations often result in vigilantism or doxxing, while expert investigations have greater success rates and fewer mishaps. Many of these experts are involved in a community of practice known as Open Source Intelligence (OSINT), with an ethos and set of techniques for conducting investigations using only publicly available data. Through semi-structured interviews with 14 expert OSINT investigators from nine different organizations, we examine the social dynamics of this community, including the collaboration and competition patterns that underlie their investigations. We also describe investigators’ use of and challenges with existing OSINT tools, and implications for the design of social computing systems to better support crowdsourced investigations.

Real-world IoT enhanced spaces involve diverse proximity- and gesture-based interactions between users and IoT devices/objects. Prototyping such interactions benefits various applications like the conceptual design of ubicomp space. AR (Augmented Reality) prototyping provides a flexible way to achieve early-stage designs by overlaying digital contents on real objects or environments. However, existing AR prototyping approaches have focused on prototyping AR experiences or context-aware interactions from the first-person view instead of full-body proxemic and gestural (pro-ges for short) interactions of real users in the real world. In this work, we conducted interviews to figure out the challenges of prototyping pro-ges interactions in real-world IoT enhanced spaces. Based on the findings, we present ProGesAR, a mobile AR tool for prototyping pro-ges interactions of a subject in a real environment from a third-person view, and examining the prototyped interactions from both the first- and third- person views. Our interface supports the effects of virtual assets dynamically triggered by a single subject, with the triggering events based on four features: location, orientation, gesture, and distance. We conduct a preliminary study by inviting participants to prototype in a freeform manner using ProGesAR. The early-stage findings show that with ProGesAR, users can easily and quickly prototype their design ideas about pro-ges interactions.

There is a growing focus on computational thinking (CT) in terms of supporting children’s understanding of everyday technologies. But unlike other technologies, Augmented Reality (AR) has received limited attention. In this paper, we present ExposAR – a collaborative cross-device AR system enabling children to create, use, and reflect on AR technologies through the co-creation of a simple AR application. With ExposAR, we explore three core design principles: 1) reify computational concepts, 2) support collaborative cross-device authoring, and 3) incorporate the user’s own world. These principles were developed through a co-design process with teachers and evaluated with 46 primary school students. We found that the collaborative authoring process with ExposAR supported students in understanding AR concepts and challenged their perspectives on AR. With these results, we bring AR to the CT agenda and contribute novel design principles for exposing the underlying mechanisms and implications of AR.

Performing with technology is a complex and challenging task. Artists who use novel technologies, such as Virtual Reality, have to develop strategies of monitoring, maintenance, and recovery from errors with as minimal impact on the ongoing performance as possible. In this paper we draw on two case studies of mixed-reality performances and document strategies of Stage Managing VR Performance, Choreographing for Cables, Consistency & Charging, Improvising Interventions, and Priming Participants. We discuss how these practices expose areas ripe with potential for tool development, and how they can also be used to inform the design of interaction with other technologies, such as the Internet of Things.

Flying dreams have the potential to evoke a feeling of empowerment (or self-efficacy, confidence in our ability to succeed) and self-transcendent experience (STE), which have been shown to contribute to an individual’s overall well-being. However, these exceptional dreaming experiences remain difficult to induce at will. Inspired by the potential of Virtual Reality (VR) to support profound emotional experiences, we explored if a VR flying interface with more embodied self-motion cues could contribute to the benefits associated with flying dreams (i.e., STE and empowerment). Our results indicated that a flying interface with more self-motion cues indeed better supported STE and empowerment. We derived several design considerations: obscurity, extraordinary light and supportive setting. Our results contribute to the discourse around design guidelines for self-transcendence and empowerment in VR, which may further be applied to the improvement of mental well-being.

We present an exploratory study on the accessibility of images in publications when viewed with color vision deficiencies (CVDs). The study is based on 1,710 images sampled from a visualization dataset (VIS30K) over five years. We simulated four CVDs on each image. First, four researchers (one with a CVD) identified existing issues and helpful aspects in a subset of the images. Based on the resulting labels, 200 crowdworkers provided  30,000 ratings on present CVD issues in the simulated images. We analyzed this data for correlations, clusters, trends, and free text comments to gain a first overview of paper figure accessibility. Overall, about 60 % of the images were rated accessible. Furthermore, our study indicates that accessibility issues are subjective and hard to detect. On a meta-level, we reflect on our study experience to point out challenges and opportunities of large-scale accessibility studies for future research directions.

There are many methods for projecting spherical maps onto the plane. Interactive versions of these projections allow the user to centre the region of interest. However, the effects of such interaction have not previously been evaluated. In a study with 120 participants we find interaction provides significantly more accurate area, direction and distance estimation in such projections. The surface of 3D sphere and torus topologies provides a continuous surface for uninterrupted network layout. But how best to project spherical network layouts to 2D screens has not been studied, nor have such spherical network projections been compared to torus projections. Using the most successful interactive sphere projections from our first study, we compare spherical, standard and toroidal layouts of networks for cluster and path following tasks with 96 participants, finding benefits for both spherical and toroidal layouts over standard network layouts in terms of accuracy for cluster understanding tasks.

An extensive body of work in visual analytics has examined how users conduct analyses in scientific and academic settings, identifying and categorizing user goals and the actions they undertake to achieve them. However, most of this work has studied the analysis process in simulated or isolated environments, leading to a gap in connecting these findings to large-scale business (enterprise) contexts, where visual analysis is most needed to make sense of the large amounts of data being generated. In this work, we conducted digital ”field” observations to understand how users conduct visual analyses in an enterprise setting, where they operate within a large ecosystem of systems and people. From these observations, we identified four common objectives, six recurring visual investigation patterns, and five emergent themes. We also performed a quantitative analysis of logs over 2530 user sessions from a second visual analysis product to validate that our patterns were not product-specific.

Icon arrays are graphical displays in which a subset of identical shapes are filled to convey probabilities. They are widely used for communicating probabilities to the general public. A primary design decision concerning icon arrays is how to fill and arrange these shapes. For example, a designer could fill the shapes from top to bottom or in a random fashion. We investigated the effect of different arrangements in icon arrays on probability perception. We showed participants icon arrays depicting probabilities between 0% and 100% in six different arrangements. Participants were more accurate in estimating probabilities when viewing the top, row, and diagonal arrangements, but they overestimated the proportions with the central arrangement and underestimated the proportions with the edge arrangement. They were biased to either overestimate or underestimate when viewing the random arrangement depending on the objective proportions, following a cyclical pattern consistent with existing findings in the psychophysics literature.

Data visualization is pervasive in the lives of children as they encounter graphs and charts in early education and online media. In spite of this prevalence, our guidelines and understanding of how children perceive graphs stem primarily from studies conducted with adults. Previous psychology and education research indicates that children’s cognitive abilities are different from adults. Therefore, we conducted a classic graphical perception study on a population of children aged 8–12 enrolled in the Ivy After School Program in Boston, MA and adult computer science students enrolled in Northeastern University to determine how accurately participants judge differences in particular graphical encodings. We record the accuracy of participants’ answers for five encodings most commonly used with quantitative data. The results of our controlled experiment show that children have remarkably similar graphical perception to adults, but are consistently less accurate at interpreting the visual encodings. We found similar effectiveness rankings, relative differences in error between the different encodings, and patterns of bias across encoding types. Based on our findings, we provide design guidelines and recommendations for creating visualizations for children. This paper and all supplemental materials are available at https://osf.io/ygrdv.

Human connection is essential for our personal well-being and a building block for a well-functioning society. There is a prominent interest in the potential of technology for mediating social connection, with a wealth of systems designed to foster the feeling of connection between strangers, friends, and family. By surveying this design landscape we present a transitional definition of mediated genuine connection and nine design strategies embodied within 50 design artifacts: affective self-disclosure, reflection on unity, shared embodied experience, transcendent emotions, embodied metaphors, interpersonal distance, touch, provocations, and play. In addition to drawing on design practice-based knowledge we also identify underlying psychological theories that can inform these strategies. We discuss design considerations pertaining to sensory modalities, vulnerability–comfort trade-offs, consent, situatedness in context, supporting diverse relationships, reciprocity, attention directedness, pursuing generalized knowledge, and questions of ethics. We hope to inspire and enrich designers’ understanding of the possibilities of technology to better support a mediated genuine feeling of connection.

Digital calendars and other technologies for social event planning leave little space to communicate uncertainty regarding time, place or the ability to attend an event. However, narratives of certainty can be detrimental and lead to the marginalisation of those who find it hard to cope with rigid and strictly paced schedules, such as people with health conditions or caring responsibilities. In this paper, we explore uncertainty as the starting point and leading principle behind digital scheduling tools. We present Haze, a speculative tool and user interface, designed to gain insights on participants’ perceptions of uncertainty-based scheduling scenarios. We report on two qualitative studies (total of 21 participants), which indicate that a change in perspective towards uncertainty can challenge moral assumptions around certainty, increase temporal empathy, and indeed support those who are particularly affected by uncertainty. These findings help shift and expand the repertoire of temporality and discuss moral and social responsibilities for design and HCI.

The COVID-19 pandemic has forced workers around the world to switch their working paradigms from on-site to video-mediated communication. Despite the advantages of videoconferencing, diverse circumstances have prevented people from focusing on their work. One of the most typical problems they face is that various surrounding factors distract them during their meetings. This study focuses on conditions in which remote workers are distracted by factors that disturb, interrupt, or restrict them during their meetings. We aim to explore the various problem situations and user needs. To understand users’ pain points and needs, focus group interviews and participatory design workshops were conducted to learn about participants’ troubled working experiences over the past two years and the solutions they expected. Our study provides a unified framework of distracting factors by which to understand causes of poor user experience and reveals valuable implications to improve videoconferencing experiences.

Personal informatics (PI) systems have been developed to support reflection. While reflection is considered an indispensable activity in PI use, how and when reflection occurs is still under-studied. In this paper, we present an analysis of the interactive features of 123 commercial PI apps, revealing that reflective practices are unevenly supported. The lack of features that encourage user-driven reflection, scaffolding for setting goals and configuring data collection and presentation, and consideration of wider implications stand to limit meaning-making and frustrate nuanced insight generation based on lived experiences. Based on our findings, we discuss how reflection is currently misrepresented in personal informatics tools, identify and characterize the gaps between theoretical research on reflection and interface features in current apps, and offer suggestions about how reflection could be better supported.

This paper reports on Zoom Obscura – an artist-based design research project, responding to the ubiquity of video-conferencing as a technical and cultural phenomenon throughout the Covid-19 pandemic. As enterprise software, such as Zoom, rapidly came to mediate even the most personal and intimate interactions, we supported and collaborated with seven independent artists to explore technical and creative interventions in video-conferencing. Our call for participation sought critical interventions that would help users counter, and regain agency in regard to the various ways in which personal data is captured, transmitted and processed in video-conferencing tools. In this design study, we analyse post-hoc how each of the seven projects employed aspects of counterfunctional design to achieve these aims. Each project reveals different avenues and strategies for counterfunctionality in video-conferencing software, as well as opportunities to design critically towards interactions and experiences that challenge existing norms and expectations around these platforms.

Social media can facilitate numerous benefits, ranging from facilitating access to social, instrumental, financial, and other support, to professional development and civic participation. However, these benefits may not be generalizable to all users. Therefore, we conducted an ethnographic case study with eight Autistic young adults, ten staff members, and four parents to understand how Autistic users of social media engage with others, as well as any unintended consequences of use. We leveraged an affordances perspective to understand how Autistic young adults share and consume user-generated content, make connections, and engage in networked interactions with others via social media. We found that they often used a literal interpretation of digital affordances that sometimes led to negative consequences including physical harm, financial loss, social anxiety, feelings of exclusion, and inadvertently damaging their social relationships. We make recommendations for redesigning social media affordances to be more inclusive of neurodiverse users.

Vaccine hesitancy has always been a public health concern, and anti-vaccine campaigns that proliferate disinformation have gained traction across the US in the last 25 years. The demographics of resistance are varied, with health, religious, and, increasingly, political concerns cited as reasons. With the COVID-19 pandemic igniting the fastest development of vaccines to date, mis- and disinformation about them have become inflammatory, with campaigning allegedly including racial targeting. Through a primarily qualitative investigation, this study inductively examines a large online vaccine discussion space that invokes references to the unethical Tuskegee Syphilis Study to understand how tactics of racial targeting of Black Americans might appear publicly. We find that such targeting is entangled with a genuine discussion about medical racism and vaccine hesitancy. Across 12 distinct voices that address race, medical racism, and vaccines, we discuss how mis- and disinformation sit alongside accurate information in a “polyvocal” space.

Online harm is a prevalent issue in adolescents’ online lives. Restorative justice teaches us to focus on those who have been harmed, ask what their needs are, and engage in the offending party and community members to collectively address the harm. In this research, we conducted interviews and design activities with harmed adolescents to understand their needs to address online harm. They also identified the key stakeholders relevant to their needs, the desired outcomes, and the preferred timing to achieve them. We identified five central needs of harmed adolescents: sensemaking, emotional support and validation, safety, retribution, and transformation. We find that addressing the needs of those who are harmed online usually requires concerted efforts from multiple stakeholders online and offline. We conclude by discussing how platforms can implement design interventions to meet some of these needs.

Social Network Services (SNSs) evoke diverse affective experiences. While most are positive, many authors have documented both the negative emotions that can result from browsing SNS and their impact: Facebook depression is a common term for the more severe results. However, while the importance of the emotions experienced on SNSs is clear, methods to catalog them, and systems to detect them, are less well developed. Accordingly, this paper reports on two studies using a novel contextually triggered Experience Sampling Method to log surveys immediately after using Instagram, a popular image-based SNS, thus minimizing recall biases. The first study improves our understanding of the emotions experienced while using SNSs. It suggests that common negative experiences relate to appearance comparison and envy. The second study captures smartphone sensor data during Instagram sessions to detect these two emotions, ultimately achieving peak accuracies of 95.78% (binary appearance comparison) and 93.95% (binary envy).

We collected Instagram Direct Messages (DMs) from 100 adolescents and young adults (ages 13-21) who then flagged their own conversations as safe or unsafe. We performed a mixed-method analysis of the media files shared privately in these conversations to gain human-centered insights into the risky interactions experienced by youth. Unsafe conversations ranged from unwanted sexual solicitations to mental health related concerns, and images shared in unsafe conversations tended to be of people and convey negative emotions, while those shared in regular conversations more often conveyed positive emotions and contained objects. Further, unsafe conversations were significantly shorter, suggesting that youth disengaged when they felt unsafe. Our work uncovers salient characteristics of safe and unsafe media shared in private conversations and provides the foundation to develop automated systems for online risk detection and mitigation.

Voice-based Conversational Agents (CAs) are increasingly being used by children. Through a review of 38 research papers, this work maps trends, themes, and methods of empirical research on children and CAs in HCI research over the last decade. A thematic analysis of the research found that work in this domain focuses on seven key topics: ascribing human-like qualities to CAs, CAs? support of children?s learning, the use and role of CAs in the home and family context, CAs? support of children?s play, children?s storytelling with CA, issues concerning the collection of information revealed by CAs, and CAs designed for children with differing abilities. Based on our findings, we identify the needs to account for children’s intersectional identities and linguistic and cultural diversity and theories from multiple disciples in the design of CAs, develop heuristics for child-centric interaction with CAs, to investigate implications of CAs on social cognition and interpersonal relationships, and to examine and design for multi-party interactions with CAs for different domains and contexts.

Educational technologies offer benefits in the classroom but there are barriers to their successful integration, including teachers’ pedagogical beliefs and their skills and experience. Participatory Design (PD) approaches offer one way in which teachers can be directly involved in the design of classroom technologies, however PD processes alone fail to address the challenges of integrating technology within existing practices. In this paper we propose co-teaching as a novel form of co-design practice. We describe a two year longitudinal Co-Teaching project resulting in the development and use of three digital designs for the classroom. Using the TPACK model to guide our reflections we offer insights into the ways that co-teaching can support the design and integration of educational technologies. We suggest that co-teaching as a form of co-design practice offers a way to move teachers from passive adopters of technology to active participants in the design and integration of educational technologies.

Social robots are increasingly introduced into children’s lives as educational and social companions, yet little is known about how these products might best be introduced to their environments. The emergence of the “unboxing” phenomenon in media suggests that introduction is key to technology adoption where initial impressions are made. To better understand this phenomenon toward designing a positive unboxing experience in the context of social robots for children, we conducted three field studies with families of children aged 8 to 13: (1) an exploratory free-play activity (n = 12); (2) a co-design session (n = 11) that informed the development of a prototype box and a curated unboxing experience; and (3) a user study (n = 9) that evaluated children’s experiences. Our findings suggest the unboxing experience of social robots can be improved through the design of a creative aesthetic experience that engages the child socially to guide initial interactions and foster a positive child-robot relationship.

Children are being presented with augmented reality (AR) in different contexts, such as education and gaming. However, little is known about how children conceptualize AR, especially AR headsets. Prior work has shown that children's interaction behaviors and expectations of technological devices can be quite different from adults’. It is important to understand children's mental models of AR headsets to design more effective experiences for them. To elicit children's perceptions, we conducted four participatory design sessions with ten children on designing content for imaginary AR headsets. We found that children expect AR systems to be highly intelligent and to recognize and virtually transform surroundings to create immersive environments. Also, children are in favor of using these devices for difficult tasks but prefer to work on their own for easy tasks. Our work contributes new understanding on how children comprehend AR headsets and provides recommendations for designing future headsets for children.

Advances in speech recognition, language processing and natural interaction have led to an increased industrial and academic interest. While the robustness and usability of such systems are steadily increasing, speech-based systems are still susceptible to recognition errors. This makes intelligent error handling of utmost importance for the success of those systems. In this work, we integrated anticipatory error handling for a voice-controlled video game where the game would perform a locally optimized action in respect to goal completion and obstacle avoidance, when a command is not recognized. We evaluated the user experience of our approach versus traditional, repetition-based error handling (). Our results indicate that implementing anticipatory error handling can improve the usability of a system, if it follows the intention of the user. Otherwise, it impairs the user experience, even when deciding for technically optimal decisions.

Competitive first-person shooter games are played over a network, where latency can degrade player performance. To better understand latency’s impact, a promising approach is to study how latency affects individual game actions, such as moving and shooting. While target selection (aiming and shooting at an opponent) is fairly well studied, navigation (moving an avatar into position) is not. This paper presents results from a 30-person user study that evaluates the impact of latency on first-person navigation using a custom “hide and seek” game that isolates avatar movement in a manner intended to be similar to movement in a first-person shooter game. Analysis of the results shows latency has pronounced effects on player performance (score and seek positioning), with subjective opinions on Quality of Experience following suit.

Learnability is a core aspect of software usability. Video games are not an exception, as game designers need to teach players how to play their creations. We analyzed 40 contemporary video games to identify how video games approach learning experiences. We found that games have advanced far beyond using simple tutorials or demonstration screens and adopt a range of repeatable and reusable design strategies using visual cues to facilitate learning. We provide a detailed descriptive framework of these design strategies, elucidating how and when they can be used, and describing how the visual cues are used to build them. Our research can be useful for both general HCI researchers and practitioners seeking to tap into the rich ideas from video game learnability design looking for practical solutions for their work.

Games have become a popular way of collecting human subject data, based on the premise that they are more engaging than surveys or experiments, but generate equally valid data. However, this premise has not been empirically tested. In response, we designed a game for eliciting linguistic data following Intrinsic Elicitation – a design approach aiming to minimise validity threats in data collection games – and compared it to an equivalent linguistics experiment as control. In a preregistered study and replication (n=96 and n=136), using two different ways of operationalising accuracy, the game generated substantially more enjoyment (d=.70,.73) and substantially less accurate data (d=-.68, -.40) – though still more accurate than random responding. We conclude that for certain data types data collection games may present a serious trade-off between participant enjoyment and data quality, identify possible causes of lower data quality for future research, reflect on our design approach, and urge games HCI researchers to use careful controls where appropriate.

Random Reward Mechanisms (RRMs) in video games are systems in which rewards are issued probabilistically upon certain trigger conditions, such as completing gameplay tasks, exceeding a playtime quota, or making in-game purchases. We investigated the relationship between RRM implementations and user experience. Video analysis of 35 RRM systems allowed for the creation of a classification system based on contrasting observed dimensions. Interviews with 14 video game players provided insights into how factors such as the affordances of non-optimal rewards and the trade-off between random luck and skill impact player perception and interaction with RRMs. We additionally investigated the relationship between auditory, visual, and gameplay design decisions and player expectations for RRM reward presentations, finding that the resources required to obtain the reward and the relative value of the reward impact its expected presentation. Finally, we applied our findings to propose design methodologies for creating engaging and significant RRM systems.

Most prior work on human-AI interaction is set in communities that indicate skepticism towards AI, but we know less about contexts where AI is viewed as aspirational. We investigated the perceptions around AI systems by drawing upon 32 interviews and 459 survey respondents in India. Not only do Indian users accept AI decisions (79.2% respondents indicate acceptance), we find a case of AI authority—AI has a legitimized power to influence human actions, without requiring adequate evidence about the capabilities of the system. AI authority manifested into four user attitudes of vulnerability: faith, forgiveness, self-blame, and gratitude, pointing to higher tolerance for system misfires, and introducing potential for irreversible individual and societal harm. We urgently call for calibrating AI authority, reconsidering success metrics and responsible AI approaches and present methodological suggestions for research and deployments in India.

It is not well understood why people continue to use privacy-invasive apps they consider creepy. We conducted a scenario-based study (n = 751) to investigate how the intention to use an app is influenced by affective perceptions and privacy concerns. We show that creepiness is one facet of affective discomfort, which is becoming normalized in app use. We found that affective discomfort can be negatively associated with the intention to use a privacy-invasive app. However, the influence is mitigated by other factors, including data literacy, views regarding app data practices, and ambiguity of the privacy threat. Our findings motivate a focus on affective discomfort when designing user experiences related to privacy-invasive data practices. Treating affective discomfort as a fundamental aspect of user experience requires scaling beyond the point where the thumb meets the screen and accounting for entrenched data practices and the sociotechnical landscape within which the practices are embedded.

While artificial intelligence (AI) is increasingly applied for decision-making processes, ethical decisions pose challenges for AI applications. Given that humans cannot always agree on the right thing to do, how would ethical decision-making by AI systems be perceived and how would responsibility be ascribed in human-AI collaboration? In this study, we investigate how the expert type (human vs. AI) and level of expert autonomy (adviser vs. decider) influence trust, perceived responsibility, and reliance. We find that participants consider humans to be more morally trustworthy but less capable than their AI equivalent. This shows in participants’ reliance on AI: AI recommendations and decisions are accepted more often than the human expert’s. However, AI team experts are perceived to be less responsible than humans, while programmers and sellers of AI systems are deemed partially responsible instead.

Decision-making involves biases from past experiences, which are difficult to perceive and eliminate. We investigate a specific type of anchoring bias, in which decision-makers are anchored by their own recent decisions, e.g. a college admission officer sequentially reviewing students. We propose an algorithm that identifies existing anchored decisions, reduces sequential dependencies to previous decisions, and mitigates decision inaccuracies post-hoc with 2% increased agreement to ground-truth on a large-scale college admission decision data set. A crowd-sourced study validates this algorithm on product preferences (5% increased agreement). To avoid biased decisions ex-ante, we propose a procedure that presents instances in an order that reduces anchoring bias in real-time. Tested in another crowd-sourced study, it reduces bias and increases agreement to ground-truth by 7%. Our work reinforces individuals with similar characteristics to be treated similarly, independent of when they were reviewed in the decision-making process.

Machine learning tools have been deployed in various contexts to support human decision-making, in the hope that human-algorithm collaboration can improve decision quality. However, the question of whether such collaborations reduce or exacerbate biases in decision-making remains underexplored. In this work, we conducted a mixed-methods study, analyzing child welfare call screen workers’ decision-making over a span of four years, and interviewing them on how they incorporate algorithmic predictions into their decision-making process. Our data analysis shows that, compared to the algorithm alone, workers reduced the disparity in screen-in rate between Black and white children from 20% to 9%. Our qualitative data show that workers achieved this by making holistic risk assessments and adjusting for the algorithm’s limitations. Our analyses also show more nuanced results about how human-algorithm collaboration affects prediction accuracy, and how to measure these effects. These results shed light on potential mechanisms for improving human-algorithm collaboration in high-risk decision-making contexts.

In this paper, we explore how expressive auditory gestures added to the speech of a pedagogical agent influence the human-agent relationship and learning outcomes. In a between-subjects experiment, 41 participants assumed the role of a tutor to teach a voice-based agent. The agent used either: expressive interjections (e.g.,“yay”, “hmm”, “oh”), brief expressive musical executions, or no auditory gestures at all (control condition), throughout the interaction. Overall, the results indicate that both gestures can positively affect the interaction, but in particular, interjections can significantly increase feelings of emotional rapport with the agent and enhance motivation in learners. The implications of our findings are discussed as our work adds to the understanding of conversational agent design and can be useful for education as well as other domains in which dialogue systems are used.

Though recent technological advances have enabled note-taking through different modalities (e.g., keyboard, digital ink, voice), there is still a lack of understanding of the effect of the modality choice on learning. In this paper, we compared two note-taking input modalities—keyboard and voice—to study their effects on participants’ understanding of learning content. We conducted a study with 60 participants in which they were asked to take notes using voice or keyboard on two independent digital text passages while also making a judgment about their performance on an upcoming test. We built mixed-effects models to examine the effect of the note-taking modality on learners’ text comprehension, the content of notes and their meta-comprehension judgement. Our findings suggest that taking notes using voice leads to a higher conceptual understanding of the text when compared to typing the notes. We also found that using voice triggers generative processes that result in learners taking more elaborate and comprehensive notes. The findings of the study imply that note-taking tools designed for digital learning environments could incorporate voice as an input modality to promote effective note-taking and higher conceptual understanding of the text.

Avatar customization is known to positively affect crucial outcomes in numerous domains. However, it is unknown whether audial customization can confer the same benefits as visual customization. We conducted a preregistered 2 x 2 (visual choice vs. visual assignment x audial choice vs. audial assignment) study in a Java programming game. Participants with visual choice experienced higher avatar identification and autonomy. Participants with audial choice experienced higher avatar identification and autonomy, but only within the group of participants who had visual choice available. Visual choice led to an increase in time spent, and indirectly led to increases in intrinsic motivation, immersion, time spent, future play motivation, and likelihood of game recommendation. Audial choice moderated the majority of these effects. Our results suggest that audial customization plays an important enhancing role vis-à-vis visual customization. However, audial customization appears to have a weaker effect compared to visual customization. We discuss the implications for avatar customization more generally across digital applications.

Video programs are important, accessible educational resources for young children, especially those from an under-resourced backgrounds. These programs’ potential can be amplified if children are allowed to socially interact with media characters during their video watching. This paper presents the design and empirical investigation of interactive science-focused videos in which the main character, powered by a conversational agent, engaged in contingent conversation with children by asking children questions and providing responsive feedback. We found that children actively interacted with the media character in the conversational videos and their parents spontaneously provided support in the process. We also found that the children who watched the conversational video performed better in the immediate, episode-specific science assessment compared to their peers who watched the broadcast, non-interactive version of the same episode. Several design implications are discussed for using conversational technologies to better support child active learning and parent involvement in video watching.

Smart speakers are increasingly being adopted by families in the U.S. In many cases, a smart speaker is shared by family members, which might make the sense of ownership uncertain. Through a diary and interview-based study with 20 Asian Indian parents and teenagers living in the U.S., this study through thematic analysis highlights various aspects of smart speakers that support or hinder the fulfilment of the psychological needs – self-efficacy, self-identity, territoriality, autonomy, and accountability and responsibility – that foster a sense of ownership. The paper also discusses the experiences of a cultural group that is not well represented in prior HCI work, and thus, it will add useful nuances and knowledge about inclusivity to the study of smart-speaker technology in HCI. Finally, it contributes six actionable design guidelines, including guidelines that relate to maintaining conversational context across different interactions and fostering a sense of ownership among users who share smart speakers, through the use of territorial markers.

To address difficulties with affect dysregulation in youth diagnosed with autism spectrum disorder (ASD), we designed and developed an end-to-end vibrotactile breathing pacer system and evaluated its usability. In this paper we describe the system architecture and the features we deployed for this system based on expert advice and reviews. Through piloting this system with one child diagnosed with ASD, we learned that our system was used in ways we did and did not anticipate. For example, the paced-breathing personalization procedure did not meet the attention span of the pilot participant but two instead of one pacer devices encouraged caregiver’s involvement. This paper details our learnings and concludes with a list of system design guidelines at the system architecture level. To the best of our knowledge, this is the first fully functional vibrotactile system designed for ASD children that withstood usability testing in vitro for two weeks.

This paper presents FlexHaptics, a design method for creating custom haptic input interfaces. Our approach leverages planar compliant structures whose force-deformation relationship can be altered by adjusting the geometries. Embedded with such structures, a FlexHaptics module exerts a fine-tunable haptic effect (i.e., resistance, detent, or bounce) along a movement path (i.e., linear, rotary, or ortho-planar). These modules can work separately or combine into an interface with complex movement paths and haptic effects. To enable the parametric design of FlexHaptic modules, we provide a design editor that converts user-specified haptic properties into underlying mechanical structures of haptic modules. We validate our approach and demonstrate the potential of FlexHaptic modules through six application examples, including a slider control for a painting application and a piano keyboard interface on touchscreens, a tactile low vision timer, VR game controllers, and a compound input device of a joystick and a two-step button.

From creating input devices to rendering tangible information, the field of HCI is interested in using kinematic mechanisms to create human-computer interfaces. Yet, due to fabrication and design challenges, it is often difficult to create kinematic devices that are compact and have multiple reconfigurable motional degrees of freedom (DOFs) depending on the interaction scenarios. In this work, we combine compliant mechanisms (CMs) with tensioning cables to create dynamically reconfigurable kinematic mechanisms. The devices’ kinematics (DOFs) is enabled and determined by the layout of bendable rods. The additional cables function as on-demand motion constraints that can dynamically lock or unlock the mechanism's DOFs as they are tightened or loosened. We provide algorithms and a design tool prototype to help users design such kinematic devices. We also demonstrate various HCI use cases including a kinematic haptic display, a haptic proxy, and a multimodal input device.

We present Shape-Haptics, an approach for designers to rapidly design and fabricate passive force feedback mechanisms for physical interfaces. Such mechanisms are used in everyday interfaces and tools, and they are challenging to design. Shape-Haptics abstracts and broadens the haptic expression of this class of force feedback systems through 2D laser cut configurations that are simple to fabricate. They leverage the properties of polyoxymethylene plastic and comprise a compliant spring structure that engages with a sliding profile during tangible interaction. By shaping the sliding profile, designers can easily customize the haptic force feedback delivered by the mechanism. We provide a computational design sandbox to facilitate designers to explore and fabricate Shape-Haptics mechanisms. We also propose a series of applications that demonstrate the utility of Shape-Haptics in creating and customizing haptics for different physical interfaces.

This work presents PITAS, a thin-sheet robotic material composed of a reversible phase transition actuating layer and a heating/sensing layer. The synthetic sheet material enables non-expert makers to create shape-changing devices that can locally or remotely convey physical information such as shape, color, texture and temperature changes. PITAS sheets can be manipulated into various 2D shapes or 3D geometries using subtractive fabrication methods such as laser, vinyl, or manual cutting or an optional additive 3D printing method for creating 3D objects. After describing the design of PITAS, this paper also describes a study conducted with thirteen makers to gauge the accessibility, design space, and limitations encountered when PITAS is used as a soft robotic material while designing physical information communication devices. Lastly, this work reports on the results of a mechanical and electrical evaluation of PITAS and presents application examples to demonstrate its utility.

Abstract: FabricatINK explores the personal fabrication of irregularly-shaped low-power displays using electronic ink (E ink). E ink is a programmable bicolour material used in traditional form-factors such as E readers. It has potential for more versatile use within the scope of personal fabrication of custom-shaped displays, and it has the promise to be the pre-eminent material choice for this purpose. We appraise technical literature to identify properties of E ink, suited to fabrication. We identify a key roadblock, universal access to E ink as a material, and we deliver a method to circumvent this by upcycling broken electronics. We subsequently present a novel fabrication method for irregularly-shaped E ink displays. We demonstrate our fabrication process and E ink’s versatility through ten prototypes showing different applications and use cases. By addressing E ink as a material for display fabrication, we uncover the potential for users to create custom-shaped truly bistable displays.

This paper presents Interactive Robotic Plastering (IRoP), a system enabling designers and skilled workers to engage intuitively with an in-situ robotic plastering process. The research combines three elements: interactive design tools, an augmented reality interface, and a robotic spraying system. Plastering is a complex process relying on tacit knowledge and craftsmanship, making it difficult to simulate and automate. However, our system utilizes a controller-based interaction system to enable diverse users to interactively create articulated plasterwork in-situ. A customizable computational toolset converts human intentions into robotic motions while respecting robotic and material constraints. To accomplish this, we developed both an interactive computational model to translate the data from a motion-tracking system into robotic trajectories using design and editing tools as well as an audio-visual guidance system for in-situ projection. We then conducted two user-studies of designers and skilled workers who used IRoP to design and fabricate a full-scale demonstrator.

Soft actuators with integrated sensing have shown utility in a variety of applications such as assistive wearables, robotics, and interactive input devices. Despite their promise, these actuators can be difficult to both design and fabricate. As a solution, we present a workflow for computationally designing and digitally fabricating soft pneumatic actuators via a machine knitting process. Machine knitting is attractive as a fabrication process because it is fast, digital (programmable), and provides access to a rich material library of functional yarns for specified mechanical behavior and integrated sensing. Our method uses elastic stitches to construct non-homogeneous knitting structures, which program the bending of actuators when inflated. Our method also integrates pressure and swept frequency capacitive sensing structures using conductive yarns. The entire knitted structure is fabricated automatically in a single machine run. We further provide a computational design interface for the user to interactively preview actuators’ quasi-static shape when authoring elastic stitches. Our sensing-integrated actuators are cost-effective, easy to design, robust to large actuation, and require minimal manual post-processing. We demonstrate five use-cases of our actuators in relevant application settings.

Soma design is intended to increase our ability to appreciate through all our senses and lead to more meaningful interactions with the world. We contribute a longer-term study of soma design that shows evidence of this promise. Using storytelling approaches we draw on qualitative data from a three-month study of the soma mat and breathing light in four households. We tell stories of people’s becomings in the world as they learn of new possibilities for their somas; and as their somas transform. We show how people drew on their somaesthetic experiences with the prototypes to find their way through troubled times; and how through continued engagement some felt compelled to make transformations in how they live their lives. We discuss the implications for the overarching soma design program, focusing on what is required to design for ways of leading a better life.

Attending to the challenges of describing first-person experience, this article illustrates different uses of the Focusing method in interaction design and HCI, offering a systematic way of accessing the subtle qualities of lived experiences for design use. In this approach, the implicit bodily knowledge -or felt sense- becomes the material capture of aesthetic experiences used to inform data collection, ideation and prototyping. We offer a high-level, yet systematic coverage of Focusing applied to two case studies, informing both a set of instructions to use the method and a series of design considerations to adopt this understudied tool of introspection in interaction design research and practice.

We articulate vulnerability as an ethical stance in soma design processes and discuss the conditions of its emergence. We argue that purposeful vulnerability – an act of taking risk, exposing oneself, and resigning part of one’s autonomy – is a necessary although often neglected part of design, and specifically soma design, which builds on felt experience and stimulates designers to engage with the non-habitual by challenging norms, habitual movements, and social interactions. With the help of ethnography, video analysis, and micro-phenomenological interviews, we document an early design exploration around drones, describing how vulnerability is accomplished in collaboration between members of the design team and the design materials. We (1) define vulnerability as an active ethical stance; (2) make vulnerability visible as a necessary but often neglected part of an exploratory design process; and (3) discuss the conditions of its emergence, demonstrating the importance of deliberating ethics within the design process.

Distracting mobile notifications are a high-profile problem but previous research suggests notification management tools are underused because of the barriers users face in relation to the perceived benefits. We posit that users might be more motivated to personalize if they could view contextual data for how personalizations would have impacted their recent notifications. We propose the ‘Reflective Spring Cleaning’ approach to support notification management through infrequent personalization with visualization of collected notification data. To simplify and contextualize key trends in a user’s notifications, we framed these visualizations within a novel who-what-when data abstraction. We evaluated it through a four-week longitudinal study: 21 participants logged their notifications before and after a personalization session that included suggestions for notification management contextualized against visualizations of their recent notifications. A debriefing interview described their new experience after two more weeks of logging. Our approach encouraged users to critically reflect on their notifications, which frequently inspired them to personalize and improved the experience of the majority.

Supervised machine learning utilizes large datasets, often with ground truth labels annotated by humans. While some data points are easy to classify, others are hard to classify, which reduces the inter-annotator agreement. This causes noise for the classifier and might affect the user’s perception of the classifier’s performance. In our research, we investigated whether the classification difficulty of a data point influences how strongly a prediction mistake reduces the “perceived accuracy”. In an experimental online study, 225 participants interacted with three fictive classifiers with equal accuracy (73%). The classifiers made prediction mistakes on three different types of data points (easy, difficult, impossible). After the interaction, participants judged the classifier’s accuracy. We found that not all prediction mistakes reduced the perceived accuracy equally. Furthermore, the perceived accuracy differed significantly from the calculated accuracy. To conclude, accuracy and related measures seem unsuitable to represent how users perceive the performance of classifiers.

Machine learning models need to provide contrastive explanations, since people often seek to understand why a puzzling prediction occurred instead of some expected outcome. Current contrastive explanations are rudimentary comparisons between examples or raw features, which remain difficult to interpret, since they lack semantic meaning. We argue that explanations must be more relatable to other concepts, hypotheticals, and associations. Inspired by the perceptual process from cognitive psychology, we propose the XAI Perceptual Processing Framework and RexNet model for relatable explainable AI with Contrastive Saliency, Counterfactual Synthetic, and Contrastive Cues explanations. We investigated the application of vocal emotion recognition, and implemented a modular multi-task deep neural network to predict and explain emotions from speech. From think-aloud and controlled studies, we found that counterfactual explanations were useful and further enhanced with semantic cues, but not saliency explanations. This work provides insights into providing and evaluating relatable contrastive explainable AI for perception applications.

Model explanations such as saliency maps can improve user trust in AI by highlighting important features for a prediction. However, these become distorted and misleading when explaining predictions of images that are subject to systematic error (bias) by perturbations and corruptions. Furthermore, the distortions persist despite model fine-tuning on images biased by different factors (blur, color temperature, day/night). We present Debiased-CAM to recover explanation faithfulness across various bias types and levels by training a multi-input, multi-task model with auxiliary tasks for explanation and bias level predictions. In simulation studies, the approach not only enhanced prediction accuracy, but also generated highly faithful explanations about these predictions as if the images were unbiased. In user studies, debiased explanations improved user task performance, perceived truthfulness and perceived helpfulness. Debiased training can provide a versatile platform for robust performance and explanation faithfulness for a wide range of applications with data biases.

Feedback in creativity support tools can help crowdworkers to improve their ideations. However, current feedback methods require human assessment from facilitators or peers. This is not scalable to large crowds. We propose Interpretable Directed Diversity to automatically predict ideation quality and diversity scores, and provide AI explanations — Attribution, Contrastive Attribution, and Counterfactual Suggestions — to feedback on why ideations were scored (low), and how to get higher scores. These explanations provide multi-faceted feedback as users iteratively improve their ideations. We conducted formative and controlled user studies to understand the usage and usefulness of explanations to improve ideation diversity and quality. Users appreciated that explanation feedback helped focus their efforts and provided directions for improvement. This resulted in explanations improving diversity compared to no feedback or feedback with scores only. Hence, our approach opens opportunities for explainable AI towards scalable and rich feedback for iterative crowd ideation and creativity support tools.

Deep learning models for image classification suffer from dangerous issues often discovered after deployment. The process of identifying bugs that cause these issues remains limited and understudied. Especially, explainability methods are often presented as obvious tools for bug identification. Yet, the current practice lacks an understanding of what kind of explanations can best support the different steps of the bug identification process, and how practitioners could interact with those explanations. Through a formative study and an iterative co-creation process, we build an interactive design probe providing various potentially relevant explainability functionalities, integrated into interfaces that allow for flexible workflows. Using the probe, we perform 18 user-studies with a diverse set of machine learning practitioners. Two-thirds of the practitioners engage in successful bug identification. They use multiple types of explanations, e.g. visual and textual ones, through non-standardized sequences of interactions including queries and exploration. Our results highlight the need for interactive, guiding, interfaces with diverse explanations, shedding light on future research directions.

We present AirRacket, perceptual modeling and design of ungrounded, directional force feedback for virtual racket sports. Using compressed air propulsion jets to provide directional impact forces, we iteratively designed for three popular sports that span a wide range of force magnitudes: ping-pong, badminton, and tennis. To address the limited force magnitude of ungrounded force feedback technologies, we conducted a perception study which discovered the novel illusion that users perceive larger impact force magnitudes with longer impact duration, by an average factor of 2.57x. Through a series of formative, perceptual, and user experience studies with a combined total of 72 unique participants, we explored several perceptual designs using force magnitude scaling and duration scaling methods to expand the dynamic range of perceived force magnitude. Our user experience evaluation showed that perceptual designs can significantly improve realism and preference vs. physics-based designs for ungrounded force feedback systems.

Virtual reality (VR) has increasingly been used in many areas, and the need to deliver notifications in VR is also expected to increase accordingly. However, untimely interruptions could largely impact the experience in VR. Identifying opportune times to deliver notifications to users allows for notifications to be scheduled in a way that minimizes disruption. We conducted a study to investigate the use of sensor data available on an off-the-shelf VR device and additional contextual information, including current activity and engagement of users, to predict opportune moments for sending notifications using deep learning models. Our analysis shows that using mainly sensor features could achieve 72% recall, 71% precision and 0.86 area under receiver operating characteristic (AUROC); performance can be further improved to 81% recall, 82% precision, and 0.93 AUROC if information about activity and summarized user engagement is included.

Virtual Reality (VR) bicycle simulations aim to recreate the feeling of riding a bicycle and are commonly used in many application areas. However, current solutions still create mismatches between the visuals and physical movement, which causes VR sickness and diminishes the cycling experience. To reduce VR sickness in bicycle simulators, we conducted two controlled lab experiments addressing two main causes of VR sickness: (1) steering methods and (2) cycling trajectory. In the first experiment (N = 18) we compared handlebar, HMD, and upper-body steering methods. In the second experiment (N = 24) we explored three types of movement in VR (1D, 2D, and 3D trajectories) and three countermeasures (airflow, vibration, and dynamic Field-of-View) to reduce VR sickness. We found that handlebar steering leads to the lowest VR sickness without decreasing cycling performance and airflow suggests to be the most promising method to reduce VR sickness for all three types of trajectories.

Dancing is a universal human activity, and also a domain of enduring significance in Human-Computer Interaction (HCI) research. However, there has been limited investigation into how computing supports the experiences of recreational dancers. Concurrently, a diverse and sizeable dance community has been emerging in VRChat. Little is known about these dancers’ experiences, motivations, and practices. Yet shedding light into these could inform both VR technology development and the design of systems that better support embodied and complex social interactions. To bridge this gap, we interviewed participants active in the VRChat dance scene. Through thematic analysis, we identified six central facets of their experiences related to freedom, community, dance as an individual experience, dance as a shared experience, dance as a performance, and self-expression and -exploration. Based on these findings, we discuss emerging tensions and highlight beneficial impacts of dancing in VR as well as problems that still await resolving.

Simulator sickness has been one of the major obstacles toward making virtual reality (VR) widely accepted and used. For example, virtual navigation produces vection, which is the illusion of self-motion as one perceives bodily motion despite no movement actually occurs. This, in turn, causes a sensory conflict between visual and actual (or vestibular) motion and sickness. In this study, we explore a method to reduce simulator sickness by visually mixing the optical flow patterns that are in the reverse direction of the virtual visual motion. As visual motion is mainly detected and perceived by the optical flow, artificial mixing in the reverse flow is hypothesized to induce a cancellation effect, thereby reducing the degree of the conflict with the vestibular sense and sickness. To validate our hypothesis, we developed a real-time algorithm to visualize the reverse optical flow and conducted experiments by comparing the before and after sickness levels in seven virtual navigation conditions. The experimental results confirmed the proposed method was effective for reducing the simulator sickness in a statistically significant manner. However, no dependency to the motion type or degrees of freedom were found. Significant distraction and negative influence to the sense of presence and immersion were observed only when the the artificially added reverse optical flow patterns were rather visually marked with high contrast to the background content.

Researchers reference realism in digital games without sufficient specificity. Without clarity about the dimensions of realism, we cannot assess how and when to aim for a higher degree of realism, when lower realism suffices, or when purposeful unrealism is ideal for a game and can benefit player experience (PX). To address this conceptual gap, we conducted a systematic review using thematic synthesis to distinguish between types of realism currently found in the digital games literature. We contribute qualitative themes that showcase contradictory design goals of realism/unrealism. From these themes, we created a framework (i.e., a hierarchical taxonomy and mapping) of realism dimensions in digital games as a conceptual foundation. Our themes and framework enable a workable specificity for designing or analyzing types of realism, equip future work to explore effects of specific realism types on PX, and offer a starting point for similar efforts in non-game applications.

Visual effects and elements in video games and interactive virtual environments can be applied to transfer (or delegate) non-visual perceptions (e.g., proprioception, presence, pain) to players and users, thus increasing perceptual diversity via the visual modality. Such elements or effects are referred to as visual delegates (VDs). Current findings on the experiences that VDs can elicit relate to specific VDs, not to VDs in general. Deductive and comprehensive VD evaluation frameworks are lacking. We analyzed VDs in video games to generalize VDs in terms of their visual properties. We conducted a systematic paper analysis to explore player and user experiences observed in association with specific VDs in user studies. We conducted semi-structured interviews with expert players to determine their preferences and the impact of VD properties. The resulting VD framework (VD-frame) contributes to a more strategic approach to identifying the impact of VDs on player and user experiences.

Board gaming is a popular hobby that increasingly features the inclusion of technology, yet little research has sought to understand how board game player experience is impacted by digital augmentation or to inform the design of new technology-enhanced games. We present a mixed-methods study exploring how the presence of music and sound effects impacts the player experience of a board game. We found that the soundtrack increased the enjoyment and tension experienced by players during game play. We also found that a soundtrack provided atmosphere surrounding the gaming experience, though players did not necessarily experience this as enhancing the world-building capabilities of the game. We discuss how our findings can inform the design of new games and soundtracks as well as future research into board game player experience.

Sound effects (SFX) complement the visual feedback provided by gamification elements in gamified systems. However, the impact of SFX has not been systematically studied. To bridge this gap, we investigate the effects of SFX—supplementing points (as a gamification element)—on task performance and user experience in a gamified image classification task. We created 18 SFX, studied their impact on perceived valence and arousal (N = 49) and selected four suitable SFX to be used in a between-participants user study (N = 317). Our findings show that neither task performance, affect, immersion, nor enjoyment were significantly affected by the sounds. Only the pressure/tension factor differed significantly, indicating that low valence sounds should be avoided to accompany point rewards. Overall, our results suggest that SFX seem to have less impact than expected in gamified systems. Hence, using SFX in gamification should be a more informed choice and should receive more attention in gamification research.

Shared control wheelchairs can help users to navigate through crowds by enabling the person to drive the wheelchair while receiving support in avoiding pedestrians. To date, research into shared control has largely overlooked the perspectives of wheelchair users. In this paper, we present two studies that aim to address this gap. The first study involved a series of semi-structured interviews with wheelchair users which highlighted the presence of two different interaction loops, one between the user and the wheelchair and a second one between the user and the crowd. In the second study we engaged with wheelchair users and designers to co-design appropriate feedback loops for future shared control interaction interfaces. Based on the results of the co-design session, we present design implications for shared control wheelchair around the need for empathy, embodiment and social awareness; situational awareness and adaptability; and selective information management.

Searching for the meaning of an unfamiliar sign-language word in a dictionary is difficult for learners, but emerging sign-recognition technology will soon enable users to search by submitting a video of themselves performing the word they recall. However, sign-recognition technology is imperfect, and users may need to search through a long list of possible results when seeking a desired result. To speed this search, we present a hybrid-search approach, in which users begin with a video-based query and then filter the search results by linguistic properties, e.g., handshape. We interviewed 32 ASL learners about their preferences for the content and appearance of the search-results page and filtering criteria. A between-subjects experiment with 20 ASL learners revealed that our hybrid search system outperformed a video-based search system along multiple satisfaction and performance metrics. Our findings provide guidance for designers of video-based sign-language dictionary search systems, with implications for other search scenarios.

Collaborative writing is an integral part of academic and professional work. Although some prior research has focused on accessibility in collaborative writing, we know little about how visually impaired writers work in real-time with sighted collaborators or how online editing tools could better support their work. Grounded in formative interviews and observations with eight screen reader users, we built Co11ab, a Google Docs extension that provides configurable audio cues to facilitate understanding who is editing (or edited) what and where in a shared document. Results from a design exploration with fifteen screen reader users, including three naturalistic sessions of use with sighted colleagues, reveal how screen reader users understand various auditory representations and use them to coordinate real-time collaborative writing. We revisit what collaboration awareness means for screen reader users and discuss design considerations for future systems.

Animated GIF images have become prevalent in internet culture, often used to express richer and more nuanced meanings than static images. But animated GIFs often lack adequate alternative text descriptions, and it is challenging to generate such descriptions automatically, resulting in inaccessible GIFs for blind or low-vision (BLV) users. To improve the accessibility of animated GIFs for BLV users, we provide a system called Ga11y (pronounced “galley”), for creating GIF annotations. Ga11y combines the power of machine intelligence and crowdsourcing and has three components: an Android client for submitting annotation requests, a backend server and database, and a web interface where volunteers can respond to annotation requests. We evaluated three human annotation interfaces and employ the one that yielded the best annotation quality. We also conducted a multi-stage evaluation with 12 BLV participants from the United States and China, receiving positive feedback.

Mobile experience sampling methods (ESMs) are widely used to measure users’ affective states by randomly sending self-report requests. However, this random probing can interrupt users and adversely influence users’ emotional states by inducing disturbance and stress. This work aims to understand how ESMs themselves may compromise the validity of ESM responses and what contextual factors contribute to changes in emotions when users respond to ESMs. Towards this goal, we analyze 2,227 samples of the mobile ESM data collected from 78 participants. Our results show ESM interruptions positively or negatively affected users’ emotional states in at least 38% of ESMs, and the changes in emotions are closely related to the contexts users were in prior to ESMs. Finally, we discuss the implications of using the ESM and possible considerations for mitigating the variability in emotional responses in the context of mobile data collection for affective computing.

With the growing prevalence of affective computing applications, Automatic Emotion Recognition (AER) technologies have garnered attention in both research and industry settings. Initially limited to speech-based applications, AER technologies now include analysis of facial landmarks to provide predicted probabilities of a common subset of emotions (e.g., anger, happiness) for faces observed in an image or video frame. In this paper, we study the relationship between AER outputs and self-reports of affect employed by prior work, in the context of information work at a technology company. We compare the continuous observed emotion output from an AER tool to discrete reported affect obtained via a one-day combined tool-use and diary study (N = 15). We provide empirical evidence showing that these signals do not completely align, and find that using additional workplace context only improves alignment up to 58.6%. These results suggest affect must be studied in the context it is being expressed, and observed emotion signal should not replace internal reported affect for affective computing applications.

We frequently utilize face emojis to express emotions in digital communication. But how wholly and precisely do such pictographs sample the emotional spectrum, and are there gaps to be closed? Our research establishes emoji intensity scales for seven basic emotions: happiness, anger, disgust, sadness, shock, annoyance, and love. In our survey (N = 1195), participants worldwide assigned emotions and intensities to 68 face emojis. According to our results, certain feelings, such as happiness or shock, are visualized by manifold emojis covering a broad spectrum of intensities. Other feelings, such as anger, have limited and only very intense representative visualizations. We further emphasize that the cultural background influences emojis’ perception: for instance, linear-active cultures (e.g., UK, Germany) rate the intensity of such visualizations higher than multi-active (e.g., Brazil, Russia) or reactive cultures (e.g., Indonesia, Singapore). To summarize, our manuscript promotes future research on more expressive, culture-aware emoji design.

“Time spent on platform” is a widely used measure in many studies examining social media use and well-being, yet the current literature presents unresolved findings about the relationship between time on platform and well-being. In this paper, we consider the moderating effect of people’s mindsets about social media — whether they think a platform is good or bad for themselves and for society more generally. Combining survey responses from 29,284 participants in 15 countries with server-logged data of Facebook use, we found that when people thought that Facebook was good for them and for society, time spent on the platform was not significantly associated with well-being. Conversely, when they thought Facebook was bad, greater time spent was associated with lower well-being. On average, there was a small, negative correlation between time spent and well-being and the causal direction is not known. Beliefs had a stronger moderating relationship when time-spent measures were self-reported rather than coming from server logs. We discuss potential mechanisms for these results and implications for future research on well-being and social media use.

ASMR (Autonomous Sensory Meridian Response) has grown to immense popularity on YouTube and drawn HCI designers’ attention to its effects and applications in design. YouTube ASMR creators incorporate visual elements, sounds, motifs of touching and tasting, and other scenarios in multisensory video interactions to deliver enjoyable and relaxing experiences to their viewers. ASMRtists engage viewers by social, physical, and task attractions. Research has identified the benefits of ASMR in mental wellbeing. However, ASMR remains an understudied phenomenon in the HCI community, constraining designers’ ability to incorporate ASMR in video-based designs. This work annotates and analyzes the interaction modalities and parasocial attractions of 2663 videos to identify unique experiences. YouTube comment sections are also analyzed to compare viewers’ responses to different ASMR interactions. We find that ASMR videos are experiences of multimodal social connection, relaxing physical intimacy, and sensory-rich activity observation. Design implications are discussed to foster future ASMR-augmented video interactions.

As dating websites are becoming an essential part of how people meet intimate and romantic partners, it is vital to design these systems to be resistant to, or at least do not amplify, bias and discrimination. Instead, the results of our online experiment with a simulated dating website, demonstrate that popular dating website design choices, such as the user of the swipe interface (swiping in one direction to indicate a like and in the other direction to express a dislike) and match scores, resulted in people racially biases choices even when they explicitly claimed not to have considered race in their decision-making. This bias was significantly reduced when the order of information presentation was reversed such that people first saw substantive profile information related to their explicitly-stated preferences before seeing the profile name and photo. These results indicate that currently-popular design choices amplify people’s implicit biases in their choices of potential romantic partners, but the effects of the implicit biases can be reduced by carefully redesigning the dating website interfaces.

To promote engagement, recommendation algorithms on platforms like YouTube increasingly personalize users’ feeds, limiting users’ exposure to diverse content and depriving them of opportunities to reflect on their interests compared to others’. In this work, we investigate how exchanging recommendations with strangers can help users discover new content and reflect. We tested this idea by developing OtherTube—a browser extension for YouTube that displays strangers’ personalized YouTube recommendations. OtherTube allows users to (i) create an anonymized profile for social comparison, (ii) share their recommended videos with others, and (iii) browse strangers’ YouTube recommendations. We conducted a 10-day-long user study (n = 41) followed by a post-study interview (n = 11). Our results reveal that users discovered and developed new interests from seeing OtherTube recommendations. We identified user and content characteristics that affect interaction and engagement with exchanged recommendations; for example, younger users interacted more with OtherTube, while the perceived irrelevance of some content discouraged users from watching certain videos. Users reflected on their interests as well as others’, recognizing similarities and differences. Our work shows promise for designs leveraging the exchange of personalized recommendations with strangers.

Online social platforms centered around content creators often allow comments on content, where creators can then moderate the comments they receive. As creators can face overwhelming numbers of comments, with some of them harassing or hateful, platforms typically provide tools such as word filters for creators to automate aspects of moderation. From needfinding interviews with 19 creators about how they use existing tools, we found that they struggled with writing good filters as well as organizing and revising their filters, due to the difficulty of determining what the filters actually catch. To address these issues, we present FilterBuddy, a system that supports creators in authoring new filters or building from pre-made ones, as well as organizing their filters and visualizing what comments are captured by them over time. We conducted an early-stage evaluation of FilterBuddy with YouTube creators, finding that participants see FilterBuddy not just as a moderation tool, but also a means to organize their comments to better understand their audiences.

While many forms of financial support are currently available, there are still many complaints about inadequate financing from software maintainers. In May 2019, GitHub, the world’s most active social coding platform, launched the Sponsor mechanism as a step toward more deeply integrating open source development and financial support. This paper collects data on 8,028 maintainers, 13,555 sponsors, and 22,515 sponsorships and conducts a comprehensive analysis. We explore the relationship between the Sponsor mechanism and developers along four dimensions using a combination of qualitative and quantitative analysis, examining why developers participate, how the mechanism affects developer activity, who obtains more sponsorships, and what mechanism flaws developers have encountered in the process of using it. We find a long-tail effect in the act of sponsorship, with most maintainers’ expectations remaining unmet, and sponsorship has only a short-term, slightly positive impact on development activity but is not sustainable. While sponsors participate in this mechanism mainly as a means of thanking the developers of OSS that they use, in practice, the social status of developers is the primary influence on the number of sponsorships. We find that both the Sponsor mechanism and open source donations have certain shortcomings and need further improvements to attract more participants.

The HCI community has been advocating preregistration as a practice to improve the credibility of scientific research. However, it remains unclear how HCI researchers preregister studies and what preregistration users perceive as benefits and challenges. By systematically reviewing the past four CHI proceedings and surveying 11 researchers, we found that only 1.11% of papers presented preregistered studies, though both authors and reviewers of preregistered studies perceive it as beneficial. Our formative studies revealed key challenges ranging from a lack of detail about the study design, hindering comprehensibility, to inconsistencies between preregistrations and published papers. To explore ways for addressing these issues, we developed Apéritif, a research prototype that scaffolds the preregistration process and automatically generates analysis code and a methods description. In an evaluation with 17 HCI researchers, we found that Apéritif reduces the effort of preregistering a study, facilitates researchers’ workflows, and promotes consistency between research artifacts.

Using voice commands to automate smartphone tasks (e.g., making a video call) can effectively augment the interactivity of numerous mobile apps. However, creating voice command interfaces requires a tremendous amount of effort in labeling and compiling the graphical user interface (GUI) and the utterance data. In this paper, we propose AutoVCI, a novel approach to automatically generate voice command interface (VCI) from smartphone operation sequences. The generated voice command interface has two distinct features. First, it automatically maps a voice command to GUI operations and fills in parameters accordingly, leveraging the GUI data instead of corpus or hand-written rules. Second, it launches a complementary Q&A dialogue to confirm the intention in case of ambiguity. In addition, the generated voice command interface can learn and evolve from user interactions. It accumulates the history command understanding results to annotate the user’s input and improve its semantic understanding ability. We implemented this approach on Android devices and conducted a two-phase user study with 16 and 67 participants in each phase. Experimental results of the study demonstrated the practical feasibility of AutoVCI.

While advanced text generation algorithms (e.g., GPT-3) have enabled writers to co-create stories with an AI, guiding the narrative remains a challenge. Existing systems often leverage simple turn-taking between the writer and the AI in story development. However, writers remain unsupported in intuitively understanding the AI’s actions or steering the iterative generation. We introduce TaleBrush, a generative story ideation tool that uses line sketching interactions with a GPT-based language model for control and sensemaking of a protagonist’s fortune in co-created stories. Our empirical evaluation found our pipeline reliably controls story generation while maintaining the novelty of generated sentences. In a user study with 14 participants with diverse writing experiences, we found participants successfully leveraged sketching to iteratively explore and write stories according to their intentions about the character’s fortune while taking inspiration from generated stories. We conclude with a reflection on how sketching interactions can facilitate the iterative human-AI co-creation process.

Interfaces for machine learning (ML), information and visualizations about models or data, can help practitioners build robust and responsible ML systems. Despite their benefits, recent studies of ML teams and our interviews with practitioners (n=9) showed that ML interfaces have limited adoption in practice. While existing ML interfaces are effective for specific tasks, they are not designed to be reused, explored, and shared by multiple stakeholders in cross-functional teams. To enable analysis and communication between different ML practitioners, we designed and implemented Symphony, a framework for composing interactive ML interfaces with task-specific, data-driven components that can be used across platforms such as computational notebooks and web dashboards. We developed Symphony through participatory design sessions with 10 teams (n=31), and discuss our findings from deploying Symphony to 3 production ML projects at Apple. Symphony helped ML practitioners discover previously unknown issues like data duplicates and blind spots in models while enabling them to share insights with other stakeholders.

The effective width method of Fitts’ law can normalize speed-accuracy biases in 1D target pointing tasks. However, in graphical user interfaces, more meaningful target shapes are rectangular. To empirically determine the best way to normalize the subjective biases, we ran remote and crowdsourced user experiments with three speed-accuracy instructions. We propose to normalize the speed-accuracy biases by applying the effective sizes to existing Fitts’ law formulations including width W and height H. We call this target-size adjustment the bivariate effective width method. We found that, overall, Accot and Zhai’s weighted Euclidean model using the effective width and height independently showed the best fit to the data in which the three instruction conditions were mixed (i.e., the time data measured in all instructions were analyzed with a single regression expression). Our approach enables researchers to fairly compare two or more conditions (e.g., devices, input techniques, user groups) with the normalized throughputs.

We investigate performance characteristics when switching between four pointing methods: absolute touch, absolute pen, relative mouse, and relative trackpad. The established “subtraction method” protocol used in mode-switching studies is extended to test pairs of methods and accommodate switch direction, multiple baselines, and controlling relative cursor position. A first experiment examines method switching on and around the horizontal surface of a tablet. Results find switching between pen and touch is fastest, and switching between relative and absolute methods incurs additional time penalty. A second experiment expands the investigation to an emerging foldable all-screen laptop form factor where switching also occurs on an angled surface and along a smoothly curved hinge. Results find switching between trackpad and touch is fastest, with all switching times generally higher. Our work contributes missing empirical evidence for switching performance using modern input methods, and our results can inform interaction design for current and emerging device form factors.

In voice-based interfaces, non-verbal features represent a simple and underutilized design space for hands-free, language-agnostic interactions. We evaluate the performance of three fundamental types of voice-based musical interactions: pitch, interval, and melody. These interactions involve singing or humming a sequence of one or more notes. A 21-person study evaluates the feasibility and enjoyability of these interactions. The top performing participants were able to perform all interactions reasonably quickly (<5s) with average error rates between 1.3% and 8.6% after training. Others improved with training but still had error rates as high as 46% for pitch and melody interactions. The majority of participants found all tasks enjoyable. Using these results, we propose design considerations for using singing interactions as well as potential use cases for both standard computers and augmented reality glasses.

Textile interfaces enable designers to integrate unobtrusive media and smart home controls into furniture such as sofas. While the technical aspects of such controllers have been the subject of numerous research projects, the physical form factor of these controls has received little attention so far. This work investigates how general design properties, such as overall slider shape, raised vs. recessed sliders, and number and layout of tick marks, affect users’ preferences and performance. Our first user study identified a preference for certain design combinations, such as recessed, closed-shaped sliders. Our second user study included performance measurements on variations of the preferred designs from study 1, and took a closer look at tick marks. Tick marks supported orientation better than slider shape. Sliders with at least three tick marks were preferred, and performed well. Non-uniform, equally distributed tick marks reduced the movements users needed to orient themselves on the slider.

Electrotactile stimulation is a novel form of haptic feedback. There is little work investigating its basic design parameters and how they create effective tactile cues. This paper describes two experiments that extend our knowledge of two key parameters. The first investigated the combination of pulse width and amplitude (Intensity) on sensations of urgency, annoyance, valence and arousal. Results showed significant effects: increasing Intensity caused higher ratings of urgency, annoyance and arousal but reduced valence. We established clear levels for differentiating each sensation. A second study then investigated Intensity and Pulse Frequency to find out how many distinguishable levels could be perceived. Results showed that both Intensity and Pulse Frequency significantly affected perception, with four distinguishable levels of Intensity and two of Pulse Frequency. These results add significant new knowledge about the parameter space of electrotactile cue design and help designers select suitable properties to use when creating electrotactile cues.

Research reveals that managing mobile device use during family time can be a source of stress for parents. In particular, it can create conflict in their relationships. As such, there is a need to understand how these problematic experiences might be addressed by new approaches to technology design. This paper presents a study in which 14 parents were prompted to reflect on how their experiences and relationships could be improved by four design proposals. These proposals resulted from ideation workshops involving 12 professional designers, and were presented as scenario-based storyboards during interviews. Our interviews revealed three design approaches that appealed to parents. We describe seven benefits that parents imagined these approaches would have, and discuss ways in which they should be further explored. Thus, we contribute to a more complete understanding of how technology design might better support parents’ aspirations for how devices are used within the family.

Emotion-related parent-child interactions during early childhood play a crucial role in the development of emotion regulation, a fundamental life skill central to well-being. However, limited work in HCI has explored how technology could support parents in adopting supportive emotion socialisation practices. In this paper, we explore how an embodied, in-situ intervention in the form of a smart toy can impact emotion-related parent-child interactions in the home. We draw on (1) interviews with 29 parents of young children who had the smart toy for at least 1 month; (2) co-design workshops with 12 parents and 8 parenting course facilitators. We discuss how the smart toy impacted parent-child interactions around emotions for a subset of families, and draw on workshop data to explore how this could be designed for directly. Finally, we propose a set of design directions for technology-enabled systems aiming to elicit and scaffold specific parent-child interactions over time.

Despite its benefits for children’s skill development and parent-child bonding, many parents do not often engage in interactive storytelling by having story-related dialogues with their child due to limited availability or challenges in coming up with appropriate questions. While recent advances made AI generation of questions from stories possible, the fully-automated approach excludes parent involvement, disregards educational goals, and underoptimizes for child engagement. Informed by need-finding interviews and participatory design (PD) results, we developed StoryBuddy, an AI-enabled system for parents to create interactive storytelling experiences. StoryBuddy’s design highlighted the need for accommodating dynamic user needs between the desire for parent involvement and parent-child bonding and the goal of minimizing parent intervention when busy. The PD revealed varied assessment and educational goals of parents, which StoryBuddy addressed by supporting configuring question types and tracking child progress. A user study validated StoryBuddy’s usability and suggested design insights for future parent-AI collaboration systems.

To acquire language, children need rich language input. However, many parents find it difficult to provide children with sufficient language input, which risks delaying their language development. To aid these parents, we design Captivate!, the first system that provides contextual language guidance to parents during play. Our system tracks both visual and spoken language cues to infer targets of joint attention, enabling the real-time suggestion of situation-relevant phrases for the parent. We design our system through a user-centered process with immigrant families—a highly vulnerable yet understudied population—as well as professional speech language therapists. Next, we evaluate Captivate! on parents with children aged 1–3 to observe improvements in responsive language use. We share insights into developing contextual guidance technology for linguistically diverse families1.

The acoustic and visual experiences of musicians in the spaces they perform in are complex and organic in nature, entailing a continuous interaction with the environment. With this project, we leverage the power of Virtual Reality (VR) to support musicians in their creative practice by transporting them to novel sonic and visual worlds. For this, we developed a musician-centred VR system, featuring various acoustic and visual virtual environments, VR Rehearse & Perform, based on design requirements gathered with musicians and performance experts. To investigate how VR can be designed to support music-makers in their creative musical practice, we performed iterative tests with 19 musicians followed by semi-structured interviews. Our findings suggest that VR has the potential to support different aspects of the creative musical practice, such as rehearsing, performing and improvising. Our research provides insights and inspirations toward designing musician-centred VR experiences for various musical activities.

Digital interfaces are shrinking, driven by pressures of mass production and consumer culture, and often accompanied by a discourse of control, precision or convenience. Meanwhile, human bodies remain the same size, and the changing size of interfaces has implications for the formation of user identities. Drawing on embodied cognition, effort and entanglement theories of HCI, we explored the impact of interface size on the co-constitution of humans and technology. We designed an oversized digital musical instrument and invited musicians to use the instrument to create original performances. We found that both the performances and the musicians’ self-perception were influenced by the large size of the instrument, shining new light on the ways in which designing technology is designing humans and in turn culture.

Auditory interfaces increasingly support access to website content, through recent advances in voice interaction. Typically, however, these interfaces provide only limited audio styling, collapsing rich visual design into a static audio output style with a single synthesized voice. To explore the potential for more aesthetic and intuitive sound design for websites, we prompted 14 professional sound designers to create auditory website mockups and interviewed them about their designs and rationale. Our findings reveal their prioritized design considerations (aesthetics and emotion, user engagement, audio clarity, information dynamics, and interactivity), specific sound design ideas to support each consideration (e.g., replacing spoken labels with short, memorable audio expressions), and challenges with applying sound design practices to auditory websites. These findings provide promising direction for how to support designers in creating richer auditory website experiences.

Most current dance support technologies focus on dancers, teachers or choreographers who are engaged in a single activity. We are interested in creating tools that support professional dancers over longer periods of time, as their careers and personal practices evolve. We interviewed 12 professional and pre-professional dancers about a critical moment in their careers: the transition to a new dance style due to shifting interests, ageing or injury. We identify three key challenges—overcoming habits, learning new forms of movement, transitioning over time—and their strategies for addressing them. We argue that successful tools must help dancers change their mentality about new movement styles, rather than focusing solely on movement mechanics. We suggest three possible implications for design: develop “movement substrates” that handle multiple movement representations; integrate learning and reflection in a single session; and create movement definitions through movement. We conclude with a discussion of directions for future research.

In this paper, we explore how the physically embodied nature of robots can influence learning through non-verbal communication, such as gesturing. We take an embodied cognition perspective to examine student interactions with a NAO robot that uses gestures while reasoning about geometry conjectures. College aged students (N = 30) were randomly assigned to either a dynamic condition, where the robot uses dynamic gestures that represent and manipulate geometric shapes in the conjectures, or control condition, where the robot uses beat gestures that match the rhythm of speech. Students in the dynamic condition: (1) use more gestures when they reason about geometry conjectures, (2) look more at the robot as it speaks, (3) feel the robot is a better study partner and uses effective gestures, but (4) were not more successful in correctly reasoning about geometry conjectures. We discuss implications for socially supported and embodied learning with a physically present robot.

Many families engage daily with artificial intelligence (AI) applications, from conversations with a voice assistant to mobile navigation searches. While there are known ways for youth to learn about AI, we do not yet understand how to engage parents in this process. To explore parents’ roles in helping their children develop AI literacies, we designed 11 learning activities organized into four topics: image classification, object recognition, interaction with voice assistants, and unplugged AI co-design. We conducted a 5-week online in-home study with 18 children (5 to 11 years old) and 16 parents. We identify parents’ most common roles in supporting their children and consider the benefits of parent-child partnerships when learning AI literacies. Finally, we discuss how our different activities supported parents’ roles and present design recommendations for future family-centered AI literacies resources.

The unique role that AI plays in making decisions that affect human lives creates a need to foster improved public understanding of AI systems. Informal learning spaces are particularly important contexts for fostering AI literacy, as they have the potential to reach a broader audience and provide spaces for children and parents to learn together. This paper explores 1) what types of dialogue family groups engage in when learning about AI in an at-home learning environment in order to inform our understanding of 2) how to design AI literacy activities for informal learning contexts. We present an analysis of family group dialogue surrounding three different AI education activities and use our findings to reflect on, update, and add to existing principles for designing AI literacy educational interventions.

Advances in Natural Language Processing offer techniques to detect the empathy level in texts. To test if individual feedback on certain students’ empathy level in their peer review writing process will help them to write more empathic reviews, we developed ELEA, an adaptive writing support system that provides students with feedback on the cognitive and emotional empathy structures. We compared ELEA to a proven empathy support tool in a peer review setting with 119 students. We found students using ELEA wrote more empathic peer reviews with a higher level of emotional empathy compared to the control group. The high perceived skill learning, the technology acceptance, and the level of enjoyment provide promising results to use such an approach as a feedback application in traditional learning settings. Our results indicate that learning applications based on NLP are able to foster empathic writing skills of students in peer review scenarios.

Through a mixed-method analysis of data from Scratch, we examine how novices learn to program with simple data structures by using community-produced learning resources. First, we present a qualitative study that describes how community-produced learning resources create archetypes that shape exploration and may disadvantage some with less common interests. In a second quantitative study, we find broad support for this dynamic in several hypothesis tests. Our findings identify a social feedback loop that we argue could limit sources of inspiration, pose barriers to broadening participation, and confine learners’ understanding of general concepts. We conclude by suggesting several approaches that may mitigate these dynamics.

Designing solution plans before writing code is critical for successful algorithmic problem-solving. Novices, however, often plan on-the-fly during implementation, resulting in unsuccessful problem-solving due to lack of mental organization of the solution. Research shows that subgoal learning helps learners develop more complete solution plans by enhancing their understanding of the high-level solution structure. However, expert-created materials such as subgoal labels are necessary to provide learning benefits from subgoal learning, which are a scarce resource in self-learning due to limited availability and high cost. We propose a learnersourcing workflow that collects high-quality subgoal labels from learners by helping them improve their label quality. We implemented the workflow into AlgoSolve, a prototype interface that supports subgoal learning for algorithmic problems. A between-subjects study with 63 problem-solving novices revealed that AlgoSolve helped learners create higher-quality labels and more complete solution plans, compared to a baseline method known to be effective in subgoal learning.

This paper describes the design of an online learning platform that empowers musical creation and performance with Python code. For this platform we have developed an innovative computational notebook paradigm that we call TunePad playbooks. While playbooks borrow ideas from popular computational notebooks like Jupyter, we have designed them from the ground up to support creative musical expression including live performances. After discussing our design principles and features, we share findings from a series of artifact-centered interviews conducted with experienced TunePad users. Our results show how systems like ours might flexibly support a variety of creative workflows, while suggesting opportunities for future work in this area.

Undergraduate Teaching Assistants(TAs) in Computer Science courses are often the first and only point of contact when a student gets stuck on a programming problem. But these TAs are often relative beginners themselves, both in programming and in teaching. In this paper, we examine the impact of availability of corrected code on TAs’ ability to find, fix, and address bugs in student code. We found that seeing a corrected version of the student code helps TAs debug code 29% faster, and write more accurate and complete student-facing explanations of the bugs (30% more likely to correctly address a given bug). We also observed that TAs do not generally struggle with the conceptual understanding of the underlying material. Rather, their difficulties seem more related to issues with working memory, attention, and overall high cognitive load.

Making and executing physical activity plans can help people improve their physical activity levels. However, little is known about how people make physical activity plans in everyday settings and how people can be assisted in creating more successful plans. In this paper, we developed and deployed a mobile app as a probe to investigate the in-the-wild physical activity planning experience for 28 days with 17 participants. Additionally, we explored the impact of presenting successful and unsuccessful planning records on participants’ planning behaviors. Based on interviews before, during, and after the deployment, we offer a description of what factors participants considered to fit their exercise plans into their existing routines, as well as factors leading to plan failures and dissatisfaction with planned physical activity. With access to historical records, participants derived insights to improve their plans, including trends in successes and failures. Based on those findings, we discuss the implications for better supporting people to make and execute physical activity plans, including suggestions for incorporating historical records into planning tools.

User acceptance is key for the successful uptake and use of health technologies, but also impacted by numerous factors not always easily accessible nor operationalised by designers in practice. This work seeks to facilitate the application of acceptance theory in design practice through the Technology Acceptance (TAC) toolkit: a novel theory-based design tool and method comprising 16 cards, 3 personas, 3 scenarios, a virtual think-space, and a website, which we evaluated through workshops conducted with 21 designers of health technologies. Findings showed that the toolkit revised and extended designers’ knowledge of technology acceptance, fostered their appreciation, empathy and ethical values while designing for acceptance, and contributed towards shaping their future design practice. We discuss implications for considering user acceptance a dynamic, multi-stage process in design practice, and better supporting designers in imagining distant acceptance challenges. Finally, we examine the generative value of the TAC toolkit and its possible future evolution.

Sleep is a vital health issue. Continued sleep deficiency can increase the chance of stroke, cardiovascular disease, obesity, and diabetes. Previous studies have investigated sleep as an individual activity performed within bedrooms at night. In this study with twenty parents of young children, we identify sleep as a complex experience entangled with social dynamics between family members. For example, children's sleep means not just time for children to rest, but time for self-care for parents. This paper's contributions are twofold. First, we show how the boundaries that define sleep in terms of time (at night), space (in bedrooms), and unit of analysis (individual-focused) limit designers' opportunities to tackle the deeper sleep issues of families. Second, we suggest "division of labor" as an important but rarely discussed design concept to enhance family sleep, and as a design theme for home technologies that address issues emerging from social dynamics between householders.

With recent developments in medical and psychiatric research surrounding pupillary response, cheap and accessible pupillometers could enable medical benefits from early neurological disease detection to measurements of cognitive load. In this paper, we introduce a novel smartphone-based pupillometer to allow for future development in clinical research surrounding at-home pupil measurements. Our solution utilizes a NIR front-facing camera for facial recognition paired with the RGB selfie camera to perform tracking of absolute pupil dilation with sub-millimeter accuracy. In comparison to a gold standard pupillometer during a pupillary light reflex test, the smartphone-based system achieves a median MAE of 0.27mm for absolute pupil dilation tracking and a median error of 3.52% for pupil dilation change tracking. Additionally, we remotely deployed the system to older adults as part of a usability study that demonstrates promise for future smartphone deployments to remotely collect data in older, inexperienced adult users operating the system themselves.

While people’s identities can be marginalized through various forces, colonialism is one of the primary ways that continues to influence people’s lives and identities. Colonialism refers to the policies and practices where foreign powers migrate to other lands and alter the social structures, and thus identities, of local populations. What is less understood is how online spaces can support people in the aftermath of colonization in revising, repairing, and strengthening their identities—the process of identity decolonization work. Using trace ethnography beginning on 15 May, 2020 and ending on 15 July, 2020 and drawing on Poka Laenui’s framework of decolonization, we explore how South Asian Bengalis on the platform Bengali Quora (BnQuora) engage in collaborative identity decolonization work to reclaim narrative agency. We discuss how narratives serve to help people bounce back from threat or vulnerability—a concept we dub narrative resilience. We also describe potential implications for future scholarship focused on decolonization that extends multiple ongoing conversations around ICT for development, social justice, decolonial HCI, and identity research within the CHI community.

Recent HCI research has suggested a move from individualistic models of digital care and wellbeing to considering the family unit as a locus of support in this area; however, little work has examined the complex, granular everyday experience of such relationships, and the role of gender, class, and care is underexplored. This study focuses on women's familial relationships through interviews with 6 Irish women about their relationships with their mothers, as well as ways in which they maintain the care of themselves and others within these relationships. Our thematic analysis of this data generated four themes: self-other care, leaky boundaries, changes over the lifecourse, and space and conflict – from which we ideated a series of design concepts, six of which are presented here with critiques from our participants. From this exploratory work, we delineate four directions for future HCI research into women's close relationships.

This paper critically examines the impacts of social media-based business on urban residential architecture in Dhaka, Bangladesh and joins the growing body of work in critical HCI. Based on a seven-month-long qualitative empirical study in Dhaka, this paper reports how Facebook commerce (F-commerce) drives many local women to actively engage in home-based businesses, which in turn, challenges the inherent spatial regulations of modern residential architecture. This paper also documents how F-commerce mediated transformations in residential spaces are promoting heterogeneous functions, re-surfacing traditional values, and altering orders and rationales that define modern housing. Drawing from a rich body of literature in urban housing architecture, critical theories around modernism, South-Asian feminism, and postcolonial computing, we explain how these spatial transformations and alterations are “appropriating” architectural design vocabularies. Our findings further explain how negligence toward such emerging needs often marginalizes the women spatially and economically, who are involved in F-commerce. We conclude with design implications to architecture and HCI to address these issues, and connect our findings to the broader agendas of Postcolonial HCI around diversity, inclusion, and global development.

This project illuminates Black women and femme’s experiences with unwanted behavior and harassment on social media, and how they (re)claim and transform their experiences to cope, heal, and experience joy. This work situates Black women and femmes’ experiences within extant social media research, and examines how their unique identity creates multiple forms of interlocking oppression. In our focus groups, participants (N=49) described harms they experienced through racism, misogynoir, ableism, and sexual objectification, and their complex labor of protecting and transforming their experiences online. Despite the harmful effects of unwanted behavior online, participants described a Black feminist transformative politic, in which they cultivated healing and joy through various methods offline and online. Using a transformative justice lens, we discuss their experiences of harassment from white women and men, as well as the complexities of cultural betrayal when experiencing harassment from Black men.

During the COVID-19 pandemic, the World Health Organization provided a checklist to help people distinguish between accurate and misinformation. In controlled experiments in the United States and Germany, we investigated the utility of this ordered checklist and designed an interactive version to lower the cost of acting on checklist items. Across interventions, we observe non-trivial differences in participants’ performance in distinguishing accurate and misinformation between the two countries and discuss some possible reasons that may predict the future helpfulness of the checklist in different environments. The checklist item that provides source labels was most frequently followed and was considered most helpful. Based on our empirical findings, we recommend practitioners focus on providing source labels rather than interventions that support readers performing their own fact-checks, even though this recommendation may be influenced by the WHO’s chosen order. We discuss the complexity of providing such source labels and provide design recommendations.

Online harassment is a major societal challenge that impacts multiple communities. Some members of community, like female journalists and activists, bear significantly higher impacts since their profession requires easy accessibility, transparency about their identity, and involves highlighting stories of injustice. Through a multi-phased qualitative research study involving a focus group and interviews with 27 female journalists and activists, we mapped the journey of a target who goes through harassment. We introduce PMCR framework, as a way to focus on needs for Prevention, Monitoring, Crisis and Recovery. We focused on Crisis and Recovery, and designed a tool to satisfy a target’s needs related to documenting evidence of harassment during the crisis and creating reports that could be shared with support networks for recovery. Finally, we discuss users’ feedback to this tool, highlighting needs for targets as they face the burden and offer recommendations to future designers and scholars on how to develop tools that can help targets manage their harassment.

Implicit tendencies and cognitive biases play an important role in how information is perceived and processed, a fact that can be both utilised and exploited by computing systems. The Implicit Association Test (IAT) has been widely used to assess people’s associations of target concepts with qualitative attributes, such as the likelihood of being hired or convicted depending on race, gender, or age. The condensed version–the Brief IAT–aims to implicit biases by measuring the reaction time to concept classifications. To use this measure in HCI research, however, we need a way to construct and validate target concepts, which tend to quickly evolve and depend on geographical and cultural interpretations. In this paper, we introduce and evaluate a new method to appropriate the BIAT using crowdsourcing to measure people’s leanings on polarising topics. We present a web-based tool to test participants’ bias on custom themes, where self-assessments often fail. We validated our approach with 14 domain experts and assessed the fit of crowdsourced test construction. Our method allows researchers of different domains to create and validate bias tests that can be geographically tailored and updated over time. We discuss how our method can be applied to surface implicit user biases and run studies where cognitive biases may impede reliable results.

As misinformation, disinformation, and conspiracy theories increase online, so does journalism coverage of these topics. This reporting is challenging, and journalists fill gaps in their expertise by utilizing external resources, including academic researchers. This paper discusses how journalists work with researchers to report on online misinformation. Through an ethnographic study of thirty collaborations, including participant-observation and interviews with journalists and researchers, we identify five types of collaborations and describe what motivates journalists to reach out to researchers — from a lack of access to data to support for understanding misinformation context. We highlight challenges within these collaborations, including misalignment in professional work practices, ethical guidelines, and reward structures. We end with a call to action for CHI researchers to attend to this intersection, develop ethical guidelines around supporting journalists with data at speed, and offer practical approaches for researchers filling a “data mediator” role between social media and journalists.

There is a great deal of interest in the role that partisanship, and cross-party animosity in particular, plays in interactions on social media. Most prior research, however, must infer users’ judgments of others’ posts from engagement data. Here, we leverage data from Birdwatch, Twitter’s crowdsourced fact-checking pilot program, to directly measure judgments of whether other users’ tweets are misleading, and whether other users’ free-text evaluations of third-party tweets are helpful. For both sets of judgments, we find that contextual features – in particular, the partisanship of the users – are far more predictive of judgments than the content of the tweets and evaluations themselves. Specifically, users are more likely to write negative evaluations of tweets from counter-partisans; and are more likely to rate evaluations from counter-partisans as unhelpful. Our findings provide clear evidence that Birdwatch users preferentially challenge content from those with whom they disagree politically. While not necessarily indicating that Birdwatch is ineffective for identifying misleading content, these results demonstrate the important role that partisanship can play in content evaluation. Platform designers must consider the ramifications of partisanship when implementing crowdsourcing programs.

The uptake of open science resources needs knowledge construction on the side of the readers/receivers of scientific content. The design of technologies surrounding open science resources can facilitate such knowledge construction, but this has not been investigated yet. To do so, we first conducted a scoping review of literature, from which we draw design heuristics for knowledge construction in digital environments. Subsequently, we grouped the underlying technological functionalities into three design categories: i) structuring and supporting collaboration, ii) supporting the learning process, and iii) structuring, visualising and navigating (learning) content. Finally, we mapped the design categories and associated design heuristics to core components of popular open science platforms. This mapping constitutes a design space (design implications), which informs researchers and designers in the HCI community about suitable functionalities for supporting knowledge construction in existing or new digital open science platforms.

The shortage of high-quality teachers is one of the biggest educational problems faced by underdeveloped areas. With the development of information and communication technologies (ICTs), China has begun a remote co-teaching intervention program using ICTs for rural classes, forming a unique “co-teaching classroom”. We conducted semi-structured interviews with nine remote urban teachers and twelve local rural teachers. We identified the remote co-teaching classes’ standard practices and co-teachers’ collaborative work process. We also found that remote teachers’ high-quality class directly impacted local teachers and students. Furthermore, interestingly, local teachers were also actively involved in making indirect impacts on their students by deeply coordinating with remote teachers and adapting the resources offered by the remote teachers. We conclude by summarizing and discussing the challenges faced by teachers, lessons learned from the current program, and related design implications to achieve a more adaptive and sustainable ICT4D program design.

Leveraging social media as a domain of high relevance in the lives of most young adolescents, we led a synchronous virtual design workshop with 17 ethnically diverse, and geographically-dispersed middle school girls (aged 11-14) to co-create novel ICT experiences. Our participatory workshop centered on social media innovation, collaboration, and computational design. We present the culminating design ideas of novel online social spaces, focused on positive experiences for adolescent girls, produced in small-groups, and a thematic analysis of the idea generation and collaboration processes. We reflect on the strengths of utilizing social media as a domain for computing exploration with diverse adolescent girls, the role of facilitators in a synchronous virtual design workshop, and the technical infrastructure that can enable age-appropriate scaffolding for active participation and use of participatory design principles embedded within educational workshops with this population.

Instructors regularly learn and customize various feature-rich software applications to meet their unique classroom needs. Although instructors often prefer social help from colleagues to navigate this complex and time-consuming learning process, it can be difficult for them to locate relevant task-specific customizations, a challenge only exacerbated by the transition to online teaching due to COVID-19. To mitigate this, we explored how instructors could use an example-based customization sharing platform to discover, try, and appropriate their colleagues’ customizations within a learning management system (LMS). Our field deployment study revealed diverse ways that ten instructors from different backgrounds used customization sharing features to streamline their workflows, improve their LMS feature awareness, and explore new possibilities for designing their courses to match student expectations. Our findings provide new knowledge about customization sharing practices, highlighting the complex interplay of expertise, software learnability, domain-specific workflows, and social perceptions.

Autonomous vehicle (AV) technologies are rapidly evolving with the vision of having self-driving cars moving safely with no human input. However, it is clear that at least in the near and foreseeable future, AVs will not be able to resolve all road incidents and that in some situations remote human assistance will be required. However, remote driving is not trivial and introduces many challenges stemming mostly from the physical disconnect of the remote operator. In order to highlight these challenges and understand how to better design AV teleoperation interfaces, we conducted several observations of AV teleoperation sessions as well as in-depth interviews with 14 experts. Based on these interviews, we provide an investigation and analysis of the major AV teleoperation challenges. We follow this by providing design suggestions for the development of future teleoperation interfaces for assistance and driving of AVs.

Mobile Robotic Telepresence (MRP) systems are remotely controlled, mobile videoconferencing devices that allow the remote user to move independently and have a physical presence in the environment. This paper presents a longitudinal study of MRP use in the home, where the first author used an MRP to connect with family, her partner, and friends over a six-month period. Taking an ethnomethodological approach, we present video recorded fragments to explore the phenomenon of ‘visiting’ where MRP users drop into the home for a period of time. We unpack the more ‘procedural’ elements—arriving and departing—alongside ways of ‘dwelling’ together during a visit, and the qualities of mobility, autonomous presence and spontaneity that emerge.

Digital contact tracing is an ICT approach for controlling public health crises. It identifies users’ risk of infection based on their healthcare and travel information. In the COVID-19 pandemic, many countries implemented digital contact tracing to contain the coronavirus outbreak. However, the adoption rates vary significantly across different countries. In this study, we investigate Chinese people’s adoption of digital contact tracing. We aim at finding the influence of Chinese culture on people’s attitudes and behaviors toward the technology. We interviewed 26 Chinese participants and used thematic analysis to interpret the data. Our findings showed that Chinese culture shaped citizens’ interactions with the digital contact tracing at multiple levels; driven by the culture, Chinese citizens accepted digital contact tracing and contributed to making digital contact tracing a socio-technical infrastructure of people’s daily lives. We also discuss such cultural influences with the growing literature of human infrastructure and crisis informatics.

Whilst imbuing robots and voice assistants with personality has been found to positively impact user experience, little is known about user perceptions of personality in purely text-based chatbots. In a within-subjects study, we asked N=34 participants to interact with three chatbots with different levels of Extraversion (extraverted, average, introverted), each over the course of four days. We systematically varied the chatbots’ responses to manipulate Extraversion based on work in the psycholinguistics of human behaviour. Our results show that participants perceived the extraverted and average chatbots as such, whereas verbal cues transferred from human behaviour were insufficient to create an introverted chatbot. Whilst most participants preferred interacting with the extraverted chatbot, participants engaged significantly more with the introverted chatbot as indicated by the users’ average number of written words. We discuss implications for researchers and practitioners on how to design chatbot personalities that can adapt to user preferences.

In a study of everyday digital identity, a set of primary drawings were made by researchers in online focus group settings as a way to capture our participants’ spoken narratives of hyper-[in]security in the usages of digital identity. In a second stage of work, key extracts from the drawings were collaged using the method described in the paper, allowing an exploratory qualitative cartography of hyper-[in]security to be constructed. These secondary collages group the [in]securities thematically without losing essential contextual information. Samples of our data are given, to illustrate the contribution of the method to experience-centred design, with special reference to security from the perspective of marginalised and underserved communities. We discuss our method as a step towards multidimensional cognitive mapping of the salient features of our participants’ narratives about hyper-[in]security, potentially paving the way for further world building explorations of digital identity futures.

We present a qualitative study with 36 diverse social media users in India to critically examine how low-resource communities engage with fake videos, including cheapfakes and AI-generated deepfakes. We find that most users are unaware of digitally manipulated fake videos and perceive videos to be fake only when they present inaccurate information. Few users who know about doctored videos expect them to be of poor quality and know nothing about sophisticated deepfakes. Moreover, most users lack the skills and willingness to spot fake videos and some were oblivious to the risks and harms of fake videos. Even when users know a video to be fake, they prefer to take no action and sometimes willingly share fake videos that favor their worldview. Drawing on our findings, we discuss design recommendations for social media platforms to curb the spread of fake videos.

Social media has witnessed an unprecedented growth in users based in low-income communities in the Global South. However, much remains unknown about the drivers of misinformation in such communities. To fill this gap, we conducted an interview-based study to examine how rural and urban communities in India engage with misinformation on WhatsApp. We found that misinformation led to bitterness and conflict – rural users who had higher social status heavily influenced the perceptions and engagement of marginalized members. While urban users relied on the expertise of gatekeepers for verification, rural users engaged in collective deliberations in offline spaces. Both rural and urban users knowingly forwarded misinformation. However, rural users propagated hyperlocal misinformation, whereas urban users forwarded misinformation to reduce their efforts to assess information credibility. Using a public sphere lens, we propose that the reactions to misinformation provide a view of Indian society and its schisms around class, urbanity, and social interactions.

Technology has provided an environment for connecting indigenous community members and provide a means for them to seek and engage with their indigenous knowledge (IK). Emerging research has examined the effects of social media on specific IK, including the possibility of undermining community agency. In this work, we contrast how indigenous community members engage with IK offline, and in their own self-organized communities online. Through interviews with community members and a study of Facebook Pages and Facebook Groups, we seek to better understand these practices and elicit design recommendations. Our findings describe how community roles have shifted in the presence of technology, notably with absence of elders and the inclusion of “born towns”–community members who live in non-traditional settings. We also find that fluency in the indigenous language served both as a gatekeeper: guarding the community knowledge, while also facilitating discussion surrounding different aspects of IK.

The world population is projected to rapidly age over the next 30 years. Given the increasing digital technology adoption amongst older adults, researchers have investigated how technology can support aging populations. However, little work has examined how technology can support older adults during crises, despite increasingly common natural disasters, public health emergencies, and other crisis scenarios in which older adults are especially vulnerable. Addressing this gap, we conducted focus groups with older adults residing in coastal locations to examine to what extent they felt technology could support them during emergencies. Our findings characterize participants’ desire for tools that enhance community resilience-local knowledge, preparedness, community relationships, and communication, that help communities withstand disasters. Further, older adults’ crisis technology preferences were linked to their sense of control, social relationships, and digital readiness. We discuss how a focus on community resilience can yield crisis technologies that more effectively support older adults.

In this paper, we present a novel screen system employing polarizers that allow switching of the projection surface to the front, rear, or both sides using only two projectors on one side. In this system, we propose a method that employs two projectors equipped with polarizers and a multi-layered screen comprising an anti-reflective plate, transparent screen, and wire grid polarizer. The multi-layered screen changes whether the projected image is shown on the front or rear side of the screen depending on the polarization direction of the incident light. Hence, the proposed method can project images on the front, rear, or both sides of the screen by projecting images from either or both projectors using polarizers. In addition, the proposed method can be easily deployed by simply attaching multiple optical films. We implement a prototype and confirm that the proposed method can selectively switch the projection surface.

From transoceanic bridges to large-scale installations, truss structures have been known for their structural stability and shape complexity. In addition to the advantages of static trusses, truss structures have a large degree of freedom to change shape when equipped with rotatable joints and retractable beams. However, it is difficult to design a complex motion and build a control system for large numbers of trusses. In this paper, we present PneuMesh, a novel truss-based shape-changing system that is easy to design and build but still able to achieve a range of tasks. PneuMesh accomplishes this by introducing an air channel connection strategy and reconfigurable constraint design that drastically decreases the number of control units without losing the complexity of shape-changing. We develop a design tool with real-time simulation to assist users in designing the shape and motion of truss-based shape-changing robots and devices. A design session with seven participants demonstrates that PneuMesh empowers users to design and build truss structures with a wide range of shapes and various functional motions.

We describe how sheets of metalized mylar can be cut and then “inflated” into complex 3D forms with electrostatic charge for use in digitally-controlled, shape-changing displays. This is achieved by placing and nesting various cuts, slits and holes such that mylar elements repel from one another to reach an equilibrium state. Importantly, our technique is compatible with industrial and hobbyist cutting processes, from die and laser cutting to handheld exacto-knives and scissors. Given that mylar film costs <$1 per m2, we can create self-actuating 3D objects for just a few cents, opening new uses in low-cost consumer goods. We describe a design vocabulary, interactive simulation tool, fabrication guide, and proof-of-concept electrostatic actuation hardware. We detail our technique’s performance metrics along with qualitative feedback from a design study. We present numerous examples generated using our pipeline to illustrate the rich creative potential of our method.

We propose a novel interface concept in which interactive systems directly manipulate the user's head orientation. We implement this using electrical-muscle-stimulation (EMS) of the neck muscles, which turns the head around its yaw (left/right) and pitch (up/down) axis. As the first exploration of EMS for head actuation, we characterized which muscles can be robustly actuated. Second, we evaluated the accuracy of our system for actuating participants' head orientation towards static targets and trajectories. Third, we demonstrated how it enables interactions not possible before by building a range of applications, such as (1) synchronizing head orientations of two users, which enables a user to communicate head nods to another user while listening to music, and (2) directly changing the user's head orientation to locate objects in AR. Finally, in our second study, participants felt that our head actuation contributed positively to their experience in four distinct applications.

We present STRAIDE, a string-actuated interactive display environment that allows to explore the promising potential of shape-changing interfaces for casual visualizations. At the core, we envision a platform that spatially levitates elements to create dynamic visual shapes in space. We conceptualize this type of tangible mid-air display and discuss its multifaceted design dimensions. Through a design exploration, we realize a physical research platform with adjustable parameters and modular components. For conveniently designing and implementing novel applications, we provide developer tools ranging from graphical emulators to in-situ augmented reality representations. To demonstrate STRAIDE’s reconfigurability, we further introduce three representative physical setups as a basis for situated applications including ambient notifications, personal smart home controls, and entertainment. They serve as a technical validation, lay the foundations for a discussion with developers that provided valuable insights, and encourage ideas for future usage of this type of appealing interactive installation.

Aging in place technologies are designed to extend independence and autonomy in older adults, promoting quality of life and peace of mind. Prior work has described how adoption and continued use of these technologies are low when they are perceived as reinforcing aging stigma or reminding older adults of negative aspects of aging. Building on past research, we investigate older adults’ perceptions of four different kinds of aging in place technologies. Based on in-depth interviews with 18 older adults using a set of scenarios, we describe how different design characteristics contribute to perceived aging stigma. Additionally, we draw on Duner & Nordström’s classification of coping strategies to discuss how specific kinds of technology can best support older adults during different stages of the aging process.

Innovations in HCI research of health-related pervasive and ubiquitous technologies can potentially improve older adults’ access to healthcare resources and support long-term independence in the home. Despite efforts to include their voices in technology research and design, many older adults have yet to actualize these health benefits, with barriers of access and proficiency actually widening the gap of health inequities. We reviewed 174 HCI publications through a systematic review to examine who is engaged in the design of health technologies for older adults, methods used to engage them, and how different types of participation might impact design directions. Findings highlight that thus far, many identity dimensions have not been explored in HCI aging research. We identify research gaps and implications to promote expanding research engagement with these dimensions as a way to support the design of health technologies that see better adoption among marginalized populations.

Makeup and cosmetics offer the potential for self-expression and the reshaping of social roles for visually impaired people. However, there exist barriers to conducting a beauty regime because of the reliance on visual information and color variances in makeup. We present a content analysis of 145 YouTube videos to demonstrate visually impaired individuals’ unique practices before, during, and after doing makeup. Based on the makeup practices, we then conducted semi-structured interviews with 12 visually impaired people to discuss their perceptions of and challenges with the makeup process in more depth. Overall, through our findings and discussion, we present novel perceptions of makeup from visually impaired individuals (e.g., broader representations of blindness and beauty). The existing challenges provide opportunities for future research to address learning barriers, insufficient feedback, and physical and environmental barriers, making the experience of doing makeup more accessible to people with visual impairments.

Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.

With about 230 million packages delivered per day in 2020, fetching packages has become a routine for many city dwellers in China. When fetching packages, people usually need to go to collection sites of their apartment complexes or a KuaiDiGui, an increasingly popular type of self-service package pickup machine. However, little is known whether such processes are accessible to blind and low vision (BLV) city dwellers. We interviewed BLV people (N=20) living in a large metropolitan area in China to understand their practices and challenges of fetching packages. Our findings show that participants encountered difficulties in finding the collection site and localizing and recognizing their packages. When fetching packages from KuaiDiGuis, they had difficulty in identifying the correct KuaiDiGui, interacting with its touch screen, navigating the complex on-screen workflow, and opening the target compartment. We discuss design considerations to make the package fetching process more accessible to the BLV community.

Existing approaches for embedding unobtrusive tags inside 3D objects require either complex fabrication or high-cost imaging equipment. We present InfraredTags, which are 2D markers and barcodes imperceptible to the naked eye that can be 3D printed as part of objects, and detected rapidly by low-cost near-infrared cameras. We achieve this by printing objects from an infrared-transmitting filament, which infrared cameras can see through, and by having air gaps inside for the tag’s bits, which appear at a different intensity in the infrared image.

We built a user interface that facilitates the integration of common tags (QR codes, ArUco markers) with the object geometry to make them 3D printable as InfraredTags. We also developed a low-cost infrared imaging module that augments existing mobile devices and decodes tags using our image processing pipeline. Our evaluation shows that the tags can be detected with little near-infrared illumination (0.2lux) and from distances as far as 250cm. We demonstrate how our method enables various applications, such as object tracking and embedding metadata for augmented reality and tangible interactions.

We present Print-A-Sketch, an open-source handheld printer prototype for sketching circuits and sensors. Print-A-Sketch combines desirable properties from free-hand sketching and functional electronic printing. Manual human control of large strokes is augmented with computer control of fine detail. Shared control of Print-A-Sketch supports sketching interactive interfaces on everyday objects – including many objects with materials or sizes which otherwise are difficult to print on. We present an overview of challenges involved in such a system and show how these can be addressed using context-aware, dynamic printing. Continuous sensing ensures quality prints by adjusting inking-rate to hand movement and material properties. Continuous sensing also enables the print to adapt to previously printed traces to support incremental and iterative sketching. Results show good conductivity on many materials and high spatial precision, supporting on-the-fly creation of functional interfaces.

We present FoolProofJoint, a software tool that simplifies the assembly of laser-cut 3D models and reduces the risk of erroneous assembly. FoolProofJoint achieves this by modifying finger joint patterns. Wherever possible, FoolProofJoint makes similar looking pieces fully interchangeable, thereby speeding up the user's visual search for a matching piece. When that is not possible, FoolProofJoint gives finger joints a unique pattern of individual finger placements so as to fit only with the correct piece, thereby preventing erroneous assembly. In our benchmark set of 217 laser-cut 3D models downloaded from kyub.com, FoolProofJoint made groups of similar looking pieces fully interchangeable for 65% of all groups of similar pieces; FoolProofJoint fully prevented assembly mistakes for 97% of all models.

One important vision of robotics is to provide physical assistance by manipulating different everyday objects, e.g., hand tools, kitchen utensils. However, many objects designed for dexterous hand-control are not easily manipulable by a single robotic arm with a generic parallel gripper. Complementary to existing research on developing grippers and control algorithms, we present Roman, a suite of hardware design and software tool support for robotic engineers to create 3D printable mechanisms attached to everyday handheld objects, making them easier to be manipulated by conventional robotic arms. The Roman hardware comes with a versatile magnetic gripper that can snap on/off handheld objects and drive add-on mechanisms to perform tasks. Roman also provides software support to register and author control programs. To validate our approach, we designed and fabricated Roman mechanisms for 14 everyday objects/tasks presented within a design space and conducted expert interviews with robotic engineers indicating that Roman serves as a practical alternative for enabling robotic manipulation of everyday objects.

Recent advancements in personal fabrication have brought novices closer to a reality, where they can automate routine tasks with mobilized everyday objects. However, the overall process remains challenging- from capturing design requirements and motion planning to authoring them to creating 3D models of mechanical parts to programming electronics, as it demands expertise.

We introduce Mobiot, an end-user toolkit to help non-experts capture the design and motion requirements of legacy objects by demonstration. It then automatically generates 3D printable attachments, programs to operate assembled modules, a list of off-the-shelf electronics, and assembly tutorials. The authoring feature further assists users to fine-tune as well as to reuse existing motion libraries and 3D printed mechanisms to adapt to other real-world objects with different motions. We validate Mobiot through application examples with 8 everyday objects with various motions applied, and through technical evaluation to measure the accuracy of motion reconstruction.

Increased levels of interactivity and multi-sensory stimulation have been shown to enhance the immersion of Virtual Reality experiences. We present the AirRes mask that enables users to utilize their breathing for precise natural interactions with the virtual environment without suffering from limitations of the sensing equipment such as motion artifacts. Furthermore, the AirRes mask provides breathing resistance as novel output modality that can be adjusted in real-time by the application. In a user study, we demonstrate the mask’s precision measurements for interaction as well as its ability to use breathing resistance to communicate contextual information such as adverse environmental conditions that affect the user’s virtual avatar. Our results show that the AirRes mask enhances virtual experiences and has the potential to create more immersive scenarios for applications by enforcing the perception of danger or improving situational awareness in training simulations, or for psychotherapy by providing additional physical stimuli.

Today’s consumer virtual reality (VR) systems offer limited haptic feedback via vibration motors in handheld controllers. Rendering haptics to other parts of the body is an open challenge, especially in a practical and consumer-friendly manner. The mouth is of particular interest, as it is a close second in tactile sensitivity to the fingertips, offering a unique opportunity to add fine-grained haptic effects. In this research, we developed a thin, compact, beamforming array of ultrasonic transducers, which can render haptic effects onto the mouth. Importantly, all components are integrated into the headset, meaning the user does not need to wear an additional accessory, or place any external infrastructure in their room. We explored several effects, including point impulses, swipes, and persistent vibrations. Our haptic sensations can be felt on the lips, teeth and tongue, which can be incorporated into new and interesting VR experiences.

Voice user interfaces (VUIs) have made their way into people’s daily lives, from voice assistants to smart speakers. Although VUIs typically just react to direct user commands, increasingly, they incorporate elements of proactive behaviors. In particular, proactive smart speakers have the potential for many applications, ranging from healthcare to entertainment; however, their usability in everyday life is subject to interaction errors. To systematically investigate the nature of errors, we designed a voice-based Experience Sampling Method (ESM) application to run on proactive speakers. We captured 1,213 user interactions in a 3-week field deployment in 13 participants’ homes. Through auxiliary audio recordings and logs, we identify substantial interaction errors and strategies that users apply to overcome those errors. We further analyze the interaction timings and provide insights into the time cost of errors. We find that, even for answering simple ESMs, interaction errors occur frequently and can hamper the usability of proactive speakers and user experience. Our work also identifies multiple facets of VUIs that can be improved in terms of the timing of speech.

We present Lattice Menu, a gaze-based marking menu utilizing a lattice of visual anchors that helps perform accurate gaze pointing for menu item selection. Users who know the location of the desired item can leverage target-assisted gaze gestures for multilevel item selection by looking at visual anchors over the gaze trajectories. Our evaluation showed that Lattice Menu exhibits a considerably low error rate (~1%) and a quick menu selection time (1.3-1.6 s) for expert usage across various menu structures (4 × 4 × 4 and 6 × 6 × 6) and sizes (8, 10 and 12°). In comparison with a traditional gaze-based marking menu that does not utilize visual targets, Lattice Menu showed remarkably (~5 times) fewer menu selection errors for expert usage. In a post-interview, all 12 subjects preferred Lattice Menu, and most subjects (8 out of 12) commented that the provisioning of visual targets facilitated more stable menu selections with reduced eye fatigue.

Electronic health records in critical care medicine offer unprecedented opportunities for clinical reasoning and decision making. Paradoxically, these data-rich environments have also resulted in clinical decision support systems (CDSSs) that fit poorly into clinical contexts, and increase health workers cognitive load. In this paper, we introduce a novel approach to designing CDSSs that are embedded in clinical workflows, by presenting problem-based curated data views tailored for problem-driven discovery, team communication, and situational awareness. We describe the design and evaluation of one such CDSS, In-Sight, that embodies our approach and addresses the clinical problem of monitoring critically ill pediatric patients. Our work is the result of a co-design process, further informed by empirical data collected through formal usability testing, focus groups, and a simulation study with domain experts. We discuss the potential and limitations of our approach, and share lessons learned in our iterative co-design process.

Reflecting on stress-related data is critical in addressing one’s mental health. Personal Informatics (PI) systems augmented by algorithms and sensors have become popular ways to help users collect and reflect on data about stress. While prediction algorithms in the PI systems are mainly for diagnostic purposes, few studies examine how the explainability of algorithmic prediction can support user-driven self-insight. To this end, we developed MindScope, an algorithm-assisted stress management system that determines user stress levels and explains how the stress level was computed based on the user’s everyday activities captured by a smartphone. In a 25-day field study conducted with 36 college students, the prediction and explanation supported self-reflection, a process to re-establish preconceptions about stress by identifying stress patterns and recalling past stress levels and patterns that led to coping planning. We discuss the implications of exploiting prediction algorithms that facilitate user-driven retrospection in PI systems.

In interdisciplinary spaces such as digital health, datasets that are complex to collect, require specialist facilities, and/or are collected with specific populations have value in a range of different sectors. In this study we collected a simulated free-living dataset, in a smart home, with 12 participants (six people with Parkinson’s, six carers). We explored their initial perceptions of the sensors through interviews and then conducted two data exploration workshops, wherein we showed participants the collected data and discussed their views on how this data, and other data relating to their Parkinson’s symptoms, might be shared across different sectors. We provide recommendations around how participants might be better engaged in considering data sharing in the early stages of research, and guidance for how research might be configured to allow for more informed data sharing practices in the future.

We present the research area of personal dream informatics: studying the self-information systems that support dream engagement and communication between the dreaming self and the wakeful self. Through a survey study of 281 individuals primarily recruited from an online community dedicated to dreaming, we develop a dream-information systems view of dreaming and dream tracking as a type of self-information system. While dream-information systems are characterized by diverse tracking processes, motivations, and outcomes, they are universally constrained by the ephemeral dreamset—the short period of time between waking up and rapid memory loss of dream experiences. By developing a system dynamics model of dreaming we highlight feedback loops that serve as high leverage points for technology designers, and suggest a variety of design considerations for crafting technology that best supports dream recall, dream tracking, and dreamwork for nightmare relief and personal development.

We can now buy consumer brain-computer interface devices to help us meditate and focus, but what are we aiming to achieve? Mental workload (MWL) is an established concept, and as a form of personal data could be useful for making positive life changes. However, MWL is typically only studied for isolated tasks to avoid overload and underload. We investigated lived experiences of MWL, aiming to understand how tracking such data could implicate our everyday lives. 19 participants, that had previously experienced tracking their mental workload, took part in interviews and an Interpretive Phenomenological Analysis identified four superordinate themes. Results point towards mixed and changing perceptions of MWL and the importance of fluctuating between MWL levels in daily life in terms of performances, perceptions, and wellbeing. These findings are captured in an apparent Cycle, which outside factors can disrupt, and we discuss these cycles in terms of personal informatics and work performance.

Menstrual tracking is a mechanism widely engaged towards preserving menstrual dignity, as natural birth control, for ensuring adequate preparation for an upcoming cycle, among other motivations. We investigate the design of digital menstrual trackers towards enabling period-positive ecologies in otherwise stigmatized contexts. We examine menstrual tracking practices across ages (12–65 yrs.) using a combination of methods—3 surveys (450+ responses), a cultural probe (10 adolescents), interviews (16 adults), and a review of (9) mobile applications. Our analysis highlights the diversity across menstrual tracking practices and the role of relationships in influencing these practices throughout the menstrual journey. We also identify menstrual tracking as an avenue towards the emancipation of those who menstruate. Finally, we draw on Martha Nussbaum’s central human capabilities to discuss sociotechnical implications for redesigning digital menstrual trackers towards crafting just and period-positive futures.

The COVID-19 pandemic created unprecedented questions for touch-based public displays regarding hygiene, risks, and general awareness. We study how people perceive and consider hygiene on shared touchscreens, and how touchscreens could be improved through hygiene-related functions. First, we report the results from an online survey (n = 286). Second, we present a hygiene concept for touchscreens that visualizes prior touches and provides information about the cleaning of the display and number of prior users. Third, we report the feedback for our hygiene concept from 77 participants. We find that there is demand for improved awareness of public displays’ hygiene status, especially among those with stronger concerns about COVID-19. A particularly desired detail is when the display has been cleaned. For visualizing prior touches, fingerprints worked best. We present further considerations for designing for hygiene on public displays.

eHealth coaching is a promising approach for delivering effective, person-centered, digital health interventions. Yet, the way eHealth coaches ‘tailor’ interventions using technology is currently unclear. This study aims to understand how eHealth coaches tailor support to clients seeking to lose weight. Nine in-depth interviews with eHealth coaches were conducted. The data were analyzed using thematic analysis, deriving three themes: ‘Understanding the client,’ ‘Adapting Support,’ and ‘Navigating challenges.’ Coaches used both assessment tools and their growing relationship to understand client's needs. Coaches adapted the pace and nature of educational content, utilized their clients' preferred technologies, and adopted different personas to meet the needs of each individual. Providing tailored support also presented challenges for eHealth coaches, with technology eroding personal and professional boundaries. Our work interrogates what it means to ‘tailor’ support, through exploring coaching as a humanistic approach to behaviour change, with implications for the design of person-centered digital coaching interventions/systems.

Behavior change and improving health literacy based on normative ideals of motherhood is a dominant paradigm to address maternal health challenges. However, these ideals often remove women's control over their bodies overlooking how the bodily experiences of pregnancy are socially and culturally constructed. We report on 27 interviews with pregnant women and nursing mothers in rural and semi-urban areas of South India, and six focus groups with 23 frontline health workers as secondary data. We explore how the embodied pregnancy experiences are influenced and negotiated by the socio-cultural context and existing care infrastructures. Our findings highlight how the ways of seeing, knowing, and caring for a body of a pregnant woman through often conflicting norms, beliefs and practices of medicine, nourishment and care actively shape the experiences of pregnancy. We open up a space for novel opportunities for digital health technologies to enhance women's embodied experiences and pregnancy care infrastructures in the Global South.

The diminutive size of wrist wearables has prompted the design of many novel input techniques to increase expressivity. Finger identification, or assigning different functionality to different fingers, has been frequently proposed. However, while the value of the technique seems clear, its implementation remains challenging, often relying on external devices (e.g., worn magnets) or explicit instructions. Addressing these limitations, this paper explores a novel approach to natural and unencumbered finger identification on an unmodified smartwatch: sonar. To do this, we adapt an existing finger tracking smartphone sonar implementation—rather than extract finger motion, we process raw sonar fingerprints representing the complete sonar scene recorded during a touch. We capture data from 16 participants operating a smartwatch and use their sonar fingerprints to train a deep learning recognizer that identifies taps by the thumb, index, and middle fingers with an accuracy of up to 93.7%, sufficient to support meaningful application development.

Speech is inappropriate in many situations, limiting when voice control can be used. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. Using a dental retainer with capacitive touch sensors, SilentSpeller tracks tongue movement, enabling users to type by spelling words without voicing. SilentSpeller achieves an average 97% character accuracy in offline isolated word testing on a 1164-word dictionary. Walking has little effect on accuracy; average offline character accuracy was roughly equivalent on 107 phrases entered while walking (97.5%) or seated (96.5%). To demonstrate extensibility, the system was tested on 100 unseen words, leading to an average 94% accuracy. Live text entry speeds for seven participants averaged 37 words per minute at 87% accuracy. Comparing silent spelling to current practice suggests that SilentSpeller may be a viable alternative for silent mobile text entry.

By sensing how a user is holding a smartphone, adaptive user interfaces are possible such as those that automatically switch the displayed content and position of graphical user interface (GUI) components following how the phone is being held. We propose ReflecTouch, a novel method for detecting how a smartphone is being held by capturing images of the smartphone screen reflected on the cornea with a built-in front camera. In these images, the areas where the user places their fingers on the screen appear as shadows, which makes it possible to estimate the grasp posture. Since most smartphones have a front camera, this method can be used regardless of the device model; in addition, no additional sensor or hardware is required. We conducted data collection experiments to verify the classification accuracy of the proposed method for six different grasp postures, and the accuracy was 85%.

Face orientation can often indicate users’ intended interaction target. In this paper, we propose FaceOri, a novel face tracking technique based on acoustic ranging using earphones. FaceOri can leverage the speaker on a commodity device to emit an ultrasonic chirp, which is picked up by the set of microphones on the user’s earphone, and then processed to calculate the distance from each microphone to the device. These measurements are used to derive the user’s face orientation and distance with respect to the device. We conduct a ground truth comparison and user study to evaluate FaceOri’s performance. The results show that the system can determine whether the user orients to the device at a 93.5% accuracy within a 1.5 meters range. Furthermore, FaceOri can continuously track user’s head orientation with a median absolute error of 10.9 mm in the distance, 3.7° in yaw, and 5.8° in pitch. FaceOri can allow for convenient hands-free control of devices and produce more intelligent context-aware interactions.

Many interactive systems are susceptible to misinterpreting the user’s input actions or gestures. Interpretation errors are common when systems gather a series of signals from the user and then attempt to interpret the user’s intention based on those signals – e.g., gesture identification from a touchscreen, camera, or body-worn electrodes – and previous work has shown that interpretation error can cause significant problems for learning new input commands. Error-reduction strategies from telecommunications, such as repeating a command or increasing the length of the input while reducing its expressiveness, could improve these input mechanisms – but little is known about whether longer command sequences will cause problems for users (e.g., increased effort or reduced learning). We tested performance, learning, and perceived effort in a crowd-sourced study where participants learned and used input mechanisms with different error-reduction techniques. We found that error reduction techniques are feasible, can outperform error-prone ordinary input, and do not negatively affect learning or perceived effort.

Personal fabrication is made more accessible through repositories like Thingiverse, as they replace modeling with retrieval. However, they require users to translate spatial requirements to keywords, which paints an incomplete picture of physical artifacts: proportions or morphology are non-trivially encoded through text only. We explore a vision of in-situ spatial search for (future) physical artifacts, and present ShapeFindAR, a mixed-reality tool to search for 3D models using in-situ sketches blended with textual queries. With ShapeFindAR, users search for geometry, and not necessarily precise labels, while coupling the search process to the physical environment (e.g., by sketching in-situ, extracting search terms from objects present, or tracing them). We developed ShapeFindAR for HoloLens 2, connected to a database of 3D-printable artifacts. We specify in-situ spatial search, describe its advantages, and present walkthroughs using ShapeFindAR, which highlight novel ways for users to articulate their wishes, without requiring complex modeling tools or profound domain knowledge.

In this paper, we investigate the use of shape-change for interaction with sound zones. A core challenge to designing interaction with sound zone systems is to support users’ understanding of the unique spatial properties of sound zones. Shape-changing interfaces present new opportunities for addressing this. We present a structured investigation into this. We leveraged the knowledge of 12 sound experts to define a set of basic shapes and movements. Then, we constructed a prototype and conducted an elicitation study with 17 novice users, investigating the experience of these shapes and movements. Our findings show that physical visualizations of sound zones can be useful in supporting users’ experience of sound zones. We present a framework of 4 basic pattern categories that prompt different sound zone experiences and outline further research directions for shape-change in supporting sound zone interaction.

Sound zone technology allows multiple simultaneous sound experiences for multiple people in the same room without interference. However, given the inherent invisible and intangible nature of sound zones, it is unclear how to communicate the position and size of sound zones to users. This paper compares two visualisation techniques; absolute visualisation, relational visualisation, as well as a baseline condition without visualisations. In a within-subject experiment (N = 33), we evaluated these techniques for effectiveness and efficiency across four representative tasks. Our findings show that the absolute and relational visualisation techniques increase effectiveness in multi-user tasks but not in single-user tasks. The efficiency for all tasks was improved using visualisations. We discuss the potential of visualisations for sound zones and highlight future research opportunities for sound zone interaction.

Social Media (SM) has shown that we adapt our communication and disclosure behaviors to available technological opportunities. Head-mounted Augmented Reality (AR) will soon allow to effortlessly display the information we disclosed not isolated from our physical presence (e.g., on a smartphone) but visually attached to the human body. In this work, we explore how the medium (AR vs. Smartphone), our role (being augmented vs. augmenting), and characteristics of information types (e.g., level of intimacy, self-disclosed vs. non-self-disclosed) impact the users’ comfort when displaying personal information. Conducting an online survey (N=148), we found that AR technology and being augmented negatively impacted this comfort. Additionally, we report that AR mitigated the effects of information characteristics compared to those they had on smartphones. In light of our results, we discuss that information augmentation should be built on consent and openness, focusing more on the comfort of the augmented rather than the technological possibilities.

Limited interaction with professionals, infrastructural, and social constraints put barriers in providing holistic support to expectant and new mothers in low-resource settings. We examine the use of digital support groups facilitated through WhatsApp by a non-government organization in India. Complementing prior research, these digital peer support groups inform about an open public space created over a chat platform where rural communities and health professionals can engage. By qualitatively analyzing six months of interaction among 588 group members and collecting the experiences of the group moderators, we inform about how the support groups acted as an important source for compensating the gaps in the existing healthcare, providing reassurance support on routine health, explanation on test reports, validation and counseling support in ongoing treatments. We also derive implications for the future of digital support groups and the need for further research on the use of unplatformed design models in resource-constrained settings.

China has witnessed rapid growth in its e-commerce markets and livestreaming communities in recent years. The commercialization of livestreaming has led to the rise of e-commerce livestreamers, among which rural women constitute a substantial portion. To understand the motivations underlying these women's choices to engage in livestreaming activities and probe the extent to which they are empowered by this new form of entrepreneurship, we conducted an interview-based study with rural female livestreamers. We found that these women chose to be livestreamers for practical and self-presentation purposes and they gained a sense of self-empowerment through economic, social, intellectual, psychological, and spiritual dimensions. At the same time, however, they experienced and confronted social stigmas rooted in rural societies and the strategies they used to deal with these biases were vastly different. Our work contributes to the HCI community by providing a nuanced understanding of the motives and lived experiences of rural female livestreamers and offers design implications that could improve the everyday experiences of these livestreamers.

Women’s economic empowerment is central to gender equality. However, work opportunities available to low-income women in patriarchal societies are infrequent. While crowd work has the potential to increase labor participation of such women, much remains unknown about their engagement with crowd work and the resultant opportunities and tensions. To fill this gap, we critically examined the adoption and use of a crowd work platform by low-income women in India. Through a qualitative study, we found that women faced tremendous challenges, for example, in seeking permission from family members to do crowd work, lack of family support and encouragement, and often working in unfavorable environments where they had to hide their work lives. While crowd work took a toll on their physical and emotional wellbeing, it also led to increased confidence, agency, and autonomy. We discuss ways to reduce frictions and tensions in participation of low-income women on crowd work platforms.

Automatic Speech Recognition (ASR) researchers are turning their attention towards supporting low-resource languages, such as isiXhosa or Marathi, with only limited training resources. We report and reflect on collaborative research across ASR & HCI to situate ASR-enabled technologies to suit the needs and functions of two communities of low-resource language speakers, on the outskirts of Cape Town, South Africa and in Mumbai, India. We build on longstanding community partnerships and draw on linguistics, media studies and HCI scholarship to guide our research. We demonstrate diverse design methods to: remotely engage participants; collect speech data to test ASR models; and ultimately field-test models with users. Reflecting on the research, we identify opportunities, challenges, and use-cases of ASR, in particular to support pervasive use of WhatsApp voice messaging. Finally, we uncover implications for collaborations across ASR & HCI that advance important discussions at CHI surrounding data, ethics, and AI.

Volunteer moderators (mods) play significant roles in developing moderation standards and dealing with harmful content in their micro-communities. However, little work explores how volunteer mods work as a team. In line with prior work about understanding volunteer moderation, we interview 40 volunteer mods on Twitch — a leading live streaming platform. We identify how mods collaborate on tasks (off-streaming coordination and preparation, in-stream real-time collaboration, and relationship building both off-stream and in-stream to reinforce collaboration) and how mods contribute to moderation standards (collaboratively working on the community rulebook and individually shaping community norms). We uncover how volunteer mods work as an effective team. We also discuss how the affordances of multi-modal communication and informality of volunteer moderation contribute to task collaboration, standards development, and mod’s roles and responsibilities.

As an emerging interaction paradigm, physiological computing is increasingly being used to both measure and feed back information about our internal psychophysiological states. While most applications of physiological computing are designed for individual use, recent research has explored how biofeedback can be socially shared between multiple users to augment human-human communication. Reflecting on the empirical progress in this area of study, this paper presents a systematic review of 64 studies to characterize the interaction contexts and effects of social biofeedback systems. Our findings highlight the importance of physio-temporal and social contextual factors surrounding physiological data sharing as well as how it can promote social-emotional competences on three different levels: intrapersonal, interpersonal, and task-focused. We also present the Social Biofeedback Interactions framework to articulate the current physiological-social interaction space. We use this to frame our discussion of the implications and ethical considerations for future research and design of social biofeedback interfaces.

The ever-increasing pace of scientific publication necessitates methods for quickly identifying relevant papers. While neural recommenders trained on user interests can help, they still result in long, monotonous lists of suggested papers. To improve the discovery experience we introduce multiple new methods for augmenting recommendations with textual relevance messages that highlight knowledge-graph connections between recommended papers and a user’s publication and interaction history. We explore associations mediated by author entities and those using citations alone. In a large-scale, real-world study, we show how our approach significantly increases engagement—and future engagement when mediated by authors—without introducing bias towards highly-cited authors. To expand message coverage for users with less publication or interaction history, we develop a novel method that highlights connections with proxy authors of interest to users and evaluate it in a controlled lab study. Finally, we synthesize design implications for future graph-based messages.

We conduct the first comprehensive study of the behavioral factors which predict leader emergence within open source software (OSS) virtual teams. We leverage the full history of developers’ interactions with their teammates and projects at github.com between January 2010 and April 2017 (representing about 133 million interactions) to establish that – contrary to a common narrative describing open source as a pure “technical meritocracy” – developers’ communication abilities and community building skills are significant predictors of whether they emerge as team leaders. Inspirational communication therefore appears as central to the process of leader emergence in virtual teams, even in a setting like OSS, where technical contributions have often been conceptualized as the sole pathway to gaining community recognition. Those results should be of interest to researchers and practitioners theorizing about OSS in particular and, more generally, leadership in geographically dispersed virtual teams, as well as to online community managers.

Conversational agents (CAs) have the great potential in mitigating the clinicians’ burden in screening for neurocognitive disorders among older adults. It is important, therefore, to develop CAs that can be engaging, to elicit conversational speech input from older adult participants for supporting assessment of cognitive abilities. As an initial step, this paper presents research in developing the backchanneling ability in CAs in the form of a verbal response to engage the speaker. We analyzed 246 conversations of cognitive assessments between older adults and human assessors, and derived the categories of reactive backchannels (e.g. “hmm”) and proactive backchannels (e.g. “please keep going”). This is used in the development of TalkTive, a CA which can predict both timing and form of backchanneling during cognitive assessments. The study then invited 36 older adult participants to evaluate the backchanneling feature. Results show that proactive backchanneling is more appreciated by participants than reactive backchanneling.

Recent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We introduce ProtoSound, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fine-grained categories. ProtoSound is motivated by prior work examining sound awareness needs of DHH people and by a survey we conducted with 472 DHH participants. To evaluate ProtoSound, we characterized performance on two real-world sound datasets, showing significant improvement over state-of-the-art (e.g., +9.7% accuracy on the first dataset). We then deployed ProtoSound's end-user training and real-time recognition through a mobile application and recruited 19 hearing participants who listened to the real-world sounds and rated the accuracy across 56 locations (e.g., homes, restaurants, parks). Results show that ProtoSound personalized the model on-device in real-time and accurately learned sounds across diverse acoustic contexts. We close by discussing open challenges in personalizable sound recognition, including the need for better recording interfaces and algorithmic improvements.

As voice-based personal assistant technologies proliferate, e.g., smart speakers in homes, and more generally as voice-control of technology becomes increasingly ubiquitous, new accessibility barriers are emerging for many Deaf and Hard of Hearing (DHH) users. Progress in sign-language recognition may enable devices to respond to sign-language commands and potentially mitigate these barriers, but research is needed to understand how DHH users would interact with these devices and what commands they would issue. In this work, we directly engage with the DHH community, using a Wizard-of-Oz prototype that appears to understand American Sign Language (ASL) commands. Our analysis of video recordings of DHH participants revealed how they woke-up the device to initiate commands, structured commands in ASL, and responded to device errors, providing guidance to future designers and researchers. We share our dataset of over 1400 commands, which may be of interest to sign-language-recognition researchers.

Conversational interfaces increasingly rely on human-like dialogue to offer a natural experience. However, relying on dialogue involving multiple exchanges for even simple tasks can overburden users, particularly older adults. In this paper, we explored the use of politeness theory in conversation design to alleviate this burden and improve user experience. To achieve this goal, we categorized the voice interaction offered by a smart display application designed for older adults into seven major speech acts: request, suggest, instruct, comment, welcome, farewell, and repair. We identified face needs for each speech act, applied politeness strategies that best address these needs, and tested the ability of these strategies to shape the perceived politeness of a voice assistant in an online study (n = 64). Based on the findings of this study, we designed direct and polite versions of the system and conducted a field study (n = 15) in which participants used each of the versions for five days at their homes. Based on five factors merged from our qualitative findings, we identified four distinctive user personas—socially oriented follower, socially oriented leader, utility oriented follower, and utility oriented leader—that can inform personalized design of smart displays.

The ability to consider a wide range of solutions to a design problem is a crucial skill for designers, and is a major differentiator between experts and novices. One reason for this is that novices are unaware of the full extent of the design space in which solutions are situated. To support novice designers with design space exploration, we introduce Mixplorer, a system that allows designers to take an initial design and mix it with other designs. Mixplorer differs from existing tools by supporting the exploration of ill-defined design spaces through social design space exploration. To evaluate Mixplorer, we conducted (1) an interview study with design instructors who reported that Mixplorer would “help to open the minds” of novice designers and (2) a controlled experiment with novices, finding that the design-mixing functionality of Mixplorer provided significantly better support for creativity, and that participants who mixed designs produced more novel designs.

Isolated silos of scientific research and the growing challenge of information overload limit awareness across the literature and hinder innovation. Algorithmic curation and recommendation, which often prioritize relevance, can further reinforce these informational “filter bubbles.” In response, we describe Bridger, a system for facilitating discovery of scholars and their work. We construct a faceted representation of authors with information gleaned from their papers and inferred author personas, and use it to develop an approach that locates commonalities and contrasts between scientists to balance relevance and novelty. In studies with computer science researchers, this approach helps users discover authors considered useful for generating novel research directions. We also demonstrate an approach for displaying information about authors, boosting the ability to understand the work of new, unfamiliar scholars. Our analysis reveals that Bridger connects authors who have different citation profiles and publish in different venues, raising the prospect of bridging diverse scientific communities.

Current research in augmented, virtual, and mixed reality (XR) reveals a lack of tool support for designing and, in particular, prototyping XR applications. While recent tools research is often motivated by studying the requirements of non-technical designers and end-user developers, the perspective of industry practitioners is less well understood. In an interview study with 17 practitioners from different industry sectors working on professional XR projects, we establish the design practices in industry, from early project stages to the final product. To better understand XR design challenges, we characterize the different methods and tools used for prototyping and describe the role and use of key prototypes in the different projects. We extract common elements of XR prototyping, elaborating on the tools and materials used for prototyping and establishing different views on the notion of fidelity. Finally, we highlight key issues for future XR tools research.

Visual storytelling is a new approach to creative expression based on verbal and figural creativity. The keys to visual storytelling are narrating and drawing over a period of time, which can be beneficial but also demanding on creativity for children. Informed by need-finding investigations, we developed StoryDrawer, a co-creative system that supports visual storytelling for children aged 6–10 years through collaborative drawing between children and artificial intelligence (AI). The system includes a context-based voice agent and two AI-driven collaborative strategies: the real-time transformation of children's telling into drawings, and the generation of abstract sketches with semantic similarity to existing story content. We conducted a 2 × 2 study with 64 children to evaluate the efficacy of StoryDrawer by varying the two strategies in four conditions. The results suggest that StoryDrawer provoked participants’ creative and elaborate ideas and contributed to their creative outcomes during an engaging visual storytelling experience.

School-driven technological innovation has the potential to positively impact on classroom practice, yet it can also be disrupted by incompatibilities between the existing school ecology and new educational technologies. To help mitigate this disruption a particular staff member often takes on a facilitative leadership role to champion new technology initiatives. However little is known about how this technology leader role impacts on the adoption of new technologies in the classroom. Taking a situated lens, we embarked on a multiple case study of four schools who were aiming to adopt a new literacy game in the classroom. Through interviews with technology leaders and fieldnotes from our site observations, we systematically analysed their actions and concerns over two academic terms. This highlighted an overwhelming concern with managing the material dimension of the technology, teacher agency and division of labour and mechanisms for communication and monitoring. Our findings raise important considerations for HCI researchers seeking to embed their technologies into practice alongside recommendations for supporting leaders tasked with coordinating this process.

Synchronous online learning has become a trend in recent years. However, instructors often face the challenge of inferring audiences’ reactions and learning status without seeing their faces in video feeds, which prevents instructors from establishing connections with students. To solve this problem, based on a need-finding survey with 67 college instructors, we propose Glancee, a real-time interactive system with adaptable configurations, sidebar-based visual displays, and comprehensive learning status detection algorithms. Then, we conduct a within-subject user study in which 18 college instructors deliver lectures online with Glancee and two baselines, EngageClass and ZoomOnly. Results show that Glancee can effectively support online teaching and is perceived to be significantly more helpful than the baselines. We further investigate how instructors’ emotions, behaviors, attention, cognitive load, and trust are affected during the class. Finally, we offer design recommendations for future online teaching assistant systems.

The COVID-19 global pandemic has ignited lightning-fast adoption of digital tools in our communities, organizations, and systems of governance. It also inspired an unprecedented level of providing access to digital devices to communities and individuals lacking prior access. The situation and circumstances provide a unique opportunity to understand digital divides through a new lens. In this work, we contribute a contemporaneous understanding of digital divides beyond access by qualitatively analyzing over 300 calls made to a volunteer-based community IT help desk. We highlight the intertwined network of challenges leading to ecosystem digital divides and contribute new insights into how the complex socio-technical systems of practice, and the tools to support them, must adapt to bridge digital divides more effectively.

Sustainable Development Goal 4 promotes inclusive and equitable quality education and lifelong learning opportunities for all. However, regions with ongoing socio-political conflict suffer disruption to education and learning. We situate our work in Kashmir, India, affected by socio-political conflict for more than three decades. We did multiple field visits and conducted 21 semi-structured interviews with parents, teachers, students, and members of a non-government organization that runs Community Learning Centers in Kashmir. Our findings present the barriers in education caused by disruption and the role of community learning centers in overcoming the barriers within these contextual constraints. Further, we discuss engaging researchers and policymakers to leverage human infrastructure, embedding uncertainty into the design, infrastructuring trust, and content usability to develop solutions to make education more accessible. Despite significant research in HCI and Education, research in this particular context is under-explored, and our work contributes to filling this gap.

Effective data literacy instruction requires that learners move beyond understanding statistics to being able to humanize data through a contextual understanding of argumentation and reasoning in the real-world. In this paper, we explore the implementation of a co-designed data comic unit about adolescent friendships. The 7th grade unit involved students analyzing data graphs about adolescent friendships and crafting comic narratives to convey perspectives on that data. Findings from our analysis of 33 student comics, and interviews with two teachers and four students, show that students engaged in various forms of data reasoning and social-emotional reasoning. These findings contribute an understanding of how students make sense of data about personal, everyday experiences; and how an arts-integrated curriculum can be designed to support their mutual engagement in both data and social-emotional reasoning.

This paper introduces the concept of “news informatics” to refer to journalistic presentation of big data in online sites. For users to be engaged with such data-driven public information, it is important to incorporate interactive tools so that each person can extract personally relevant information. Drawing upon a communication model of interactivity, we designed a data-rich site with three different types of interactive features—namely, modality interactivity, message interactivity, and source interactivity—and empirically tested their relative and combined effects on user engagement and user experience with a 2 (modality) × 3 (source) × 2 (message) field experiment (N =166). Findings shed light on how interface designers, online news editors and journalists can maximize user engagement with data-rich news content. Certain interactivity combinations are found to be better than others, with a structural equation model (SEM) revealing the underlying theoretical mechanisms and providing implications for the design of news informatics.

Analysis is a key part of usability testing where UX practitioners seek to identify usability problems and generate redesign suggestions. Although previous research reported how analysis was conducted, the findings were typically focused on individual analysis or based on a small number of professionals in specific geographic regions. We conducted an online international survey of 279 UX practitioners on their practices and challenges while collaborating during data analysis. We found that UX practitioners were often under time pressure to conduct analysis and adopted three modes of collaboration: independently analyze different portions of the data and then collaborate, collaboratively analyze the session with little or no independent analysis, and independently analyze the same set of data and then collaborate. Moreover, most encountered challenges related to lack of resources, disagreements with colleagues regarding usability problems, and difficulty merging analysis from multiple practitioners. We discuss design implications to better support collaborative data analysis.

Sentence completion, originally a semi-projective psychological technique, has been used as an effective and lightweight user research method in UX design. More information is yet still needed to understand how different sentence stems probe users’ insights, thereby providing recommendations for effective sentence completion surveys. We used the completion method on a large-scale sample to explore (e-)readers’ experiences and needs. Depending on their reading habits, participants (N=1880) were asked to complete a set of sentences, as part of a web survey. With 14143 user ideas collected in two weeks, our results confirm that remote online sentence completion is a cost-effective data collection method able to uncover feelings, attitudes, motivations, needs, or frustrations. Variation in sentence stems affected collected data in terms of item response rate, idea quantity as well as variety and originality. Building on previous research, this paper delivers actionable insights to optimize the richness of sentence completion outputs.

For many students, attending college is a dramatic but necessary change. To gain a better understanding of experiences that are unique to autistic college students, we conducted a mixed-method study with 20 students (10 autistic and 10 neurotypical). We collected physiological, contextual, experience, and environmental data from their natural environment using Fitbit and smartphones. We found that stress patterns, emotional states, and physical states are similar for both groups. Our autistic participants prioritized academic success over everything else, often intentionally confining their movements among academic, resident, and work locations to engage themselves with academic work as much as possible. They had a small number of friends, always preferring quality over quantity and sometimes regarding friends as close as family members. To maintain a better social life, they extensively used social media. They slept more than neurotypical participants per day; however, they experienced lower sleep quality.

Voice assistants (VAs) are present in homes, smartphones, and cars. They allow users to perform tasks without graphical or tactile user interfaces, as they are designed for natural language interaction. However, we found that currently, VAs are emulating human behavior by responding in complete sentences, limiting the design options, and preventing VAs from meeting their full potential as a utilitarian tool. We implemented a VA that handles requests in three response styles: two differing short keyword-based response styles and a full-sentence baseline. In a user study, 72 participants interacted with our VA by issuing eight requests. Results show that the short responses were perceived similarly useful and likable while being perceived as more efficient, especially for commands, and sometimes better to comprehend than the baseline. To achieve widespread adoption, we argue that VAs should be customizable and adapt to users instead of always responding in full sentences.

Data-driven approaches that form the foundation of advancements in machine learning (ML) are powered in large part by human infrastructures that enable the collection of large datasets. We study the movement of data through multiple stages of data processing in the context of public health in India, examining the data work performed by frontline health workers, data stewards, and ML developers. We conducted interviews with these stakeholders to understand their varied perspectives on valuing data across stages, working with data to attain this value, and challenges arising throughout. We discuss the tensions in valuing and how they might be addressed, as we emphasize the need for improved transparency and accountability when data are transformed from one stage of processing to the next.

HCI engages with data science through many topics and themes. Researchers have addressed biased dataset problems, arguing that bad data can cause innocent software to produce bad outcomes. But what if our software is not so innocent? What if the human decisions that shape our data-processing software, inadvertently contribute their own sources of bias? And what if our data-work technology causes us to forget those decisions and operations? Based in feminisms and critical computing, we analyze forgetting practices in data work practices. We describe diverse beneficial and harmful motivations for forgetting. We contribute: (1) a taxonomy of data silences in data work, which we use to analyze how data workers forget, erase, and unknow aspects of data; (2) a detailed analysis of forgetting practices in machine learning; and (3) an analytic vocabulary for future work in remembering, forgetting, and erasing in HCI and the data sciences.

Design has been used to contest existing socio-technical arrangements, provoke conversations around matters of concern, and operationalize radical theories such as agonism, which embraces difference and contention. However, the focus is usually on creating something new: a product, interface or artifact. In this paper, we investigate what happens when critical unmaking is deployed as a deliberate design strategy in an intergenerational, agonistic urban context. Specifically, we report on how youth in a six-week design internship used unmaking as a design move to subvert conventional narratives about their surrounding urban context. We analyze how this led to conflictual encounters at the local senior center, and compare it to the other, making-centric proposals which received favorable feedback but failed to raise the same important discussions. Through this ethnographic account, we argue that critical unmaking is important yet overlooked, and should be in the repertoire of design moves available for agonism and provocation.

Communicating risk to the public in the lead-up to tropical storms has the potential to significantly reduce the impacts on both livelihood and property. While significant research has been conducted in the storm risk community on how people receive, seek, and utilize risk information, given the importance of computing technologies and social media in these activities, human-centered design stands to make important contributions to this area. Drawing on an extensive literature review and 48 interviews with hurricane experts and members of the public, this paper makes three contributions. First, we provide a broad overview of hurricane risk communication. We then offer a set of guiding insights to inform HCI research work in this domain. Finally, we identify 6 opportunities that future human centered design work might pursue. In sum, this paper offers an invitation and a starting point for HCI to take up the problem of hurricane risk communication.

Data collection is often a laborious enterprise that forms part of the wider craft skill of doing research. In this essay, I try to understand whether parts of research processes in Human-Centred Computing (HCC) have been commodified, with a particular focus on data collection. If data collection has been commodified, do researchers act as producers or consumers in the process? And if researchers are consumers, has data collection become a consumption experience? If so, what are the implications of this? I explore these questions by considering the status of craft and consumption in the research process and by developing examples of consumption experiences. I note the benefits of commodity research artefacts, while highlighting the potentially deleterious effects consumption experiences could have on our ability to generate insights into the relations between people and technology. I finish the paper by relating consumption experiences to contemporary issues in HCC and lay out a programme of empirical work that would help answer some of the questions this paper raises.

Workplace stress-reduction interventions have produced mixed results due to engagement and adherence barriers. Leveraging technology to integrate such interventions into the workday may address these barriers and help mitigate the mental, physical, and monetary effects of workplace stress. To inform the design of a workplace stress-reduction intervention system, we conducted a four-week longitudinal study with 86 participants, examining the effects of intervention type and timing on usage, stress reduction impact, and user preferences. We compared three intervention types and two delivery timing conditions: Pre-scheduled (PS) by users and Just-in-time (JIT) prompted by the system-identified user stress-levels. We found JIT participants completed significantly more interventions than PS participants, but post-intervention and study-long stress reduction was not significantly different between conditions. Participants rated low-effort interventions highest, but high-effort interventions reduced the most stress. Participants felt JIT provided accountability but desired partial agency over timing. We present type and timing implications.

Young adults have high rates of mental health conditions, yet they are the age group least likely to seek traditional treatment. They do, however, seek information about their mental health online, including by filling out online mental health screeners. To better understand online self-screening, and its role in help-seeking, we conducted focus groups with 50 young adults who voluntarily completed a mental health screener hosted on an advocacy website. We explored (1) catalysts for taking the screener, (2) anticipated outcomes, (3) reactions to the results, and (4) desired next steps. For many participants, the screener results validated their lived experiences of symptoms, but they were nevertheless unsure how to use the information to improve their mental health moving forward. Our findings suggest that online screeners can serve as a transition point in young people’s mental health journeys. We discuss design implications for online screeners, post-screener feedback, and digital interventions broadly.

Young adults have high rates of mental health conditions, but most do not want or cannot access formal treatment. We therefore recruited young adults with depression or anxiety symptoms to co-design a digital tool for self-managing their mental health concerns. Through study activities—consisting of an online discussion group and a series of design workshops—participants highlighted the importance of easy-to-use digital tools that allow them to exercise independence in their self-management. They described ways that an automated messaging tool might benefit them by: facilitating experimentation with diverse concepts and experiences; allowing variable depth of engagement based on preferences, availability, and mood; and collecting feedback to personalize the tool. While participants wanted to feel supported by an automated tool, they cautioned against incorporating an overtly human-like motivational tone. We discuss ways to apply these findings to improve the design and dissemination of digital mental health tools for young adults.

Amidst increasing reports of mental health problems in Canadian university students, those of Asian descent have particularly struggled to seek out services due to cultural barriers. Counselling practices have long noted that culture influences how mental health is perceived and treated, yet the design of mental health technologies is limited with respect to how users’ backgrounds influence usability and adoption. To identify inclusive design opportunities, we interviewed 20 East Asian university students in Canada. We found that they struggle to engage with technologies for mental health due to cultural stigma which have led them to prefer apps that support self-help though still valuing social help. We present inclusive design opportunities for mental health technologies that sensitively consider these challenges, including supporting learning opportunities with peers through storytelling and skill-sharing to promote literacy, empowerment, and advocacy for their own health. We conclude by discussing how universities can promote mental wellbeing more inclusively.

The ability to manage emotions effectively is critical to healthy psychological and social development in youth. Prior work has focused on investigating the design of mental health technologies for this population, yet it is still unclear how to help them cope with emotionally difficult situations in-the-moment. In this paper, we aim to explore the appropriation, naturally emerging engagement patterns, and perceived psychological impact of an exemplar interactive tangible device intervention designed to provide in-situ support, when deployed with n=109 youth for 1.5 months. Our findings from semi-structured interviews and co-design workshops with a subset of participants (n=44 and n=25, respectively) suggest the potential of using technology-enabled objects to aid with down-regulation and self-compassion in moments of heightened emotion, to facilitate the practice of cognitive strategies, and to act as emotional companions. Lastly, we discuss design opportunities for integrating situated and embodied support in mental health interventions for youth.

Sustainable Development (SD) in its dimensions – environment, economy, and society – is a growing area of concern within the HCI community. This paper advances a systematic literature review on sustainability across the Sustainable Human-Computer Interaction (SHCI) body of work. The papers were classified according to the Triple Bottom Line (TBL) framework to understand how the pillars of SD play into the HCI discourse on sustainability. The economic angle was identified as a gap in SHCI literature. To meet the TBL of SD, however, a balance needs to be sought across all ‘lines’. In this paper, we propose that HCI can advance the discussion and the understanding of the economic concepts around sustainability through taking a sociology perspective on the economic angle of the TBL. We sustain this claim by discussing economic concepts and the role that digital can play in redefining the established foundations of our economic system.

A transition to renewable energy increases the risks of disruptions when electricity supply does not meet demand. HCI has explored how digital technologies can mitigate such problems in households through support for reducing or shifting electricity use. However, faster transitions may be possible if some disturbances can be acceptable and households are supported in adapting to them. In this paper, we present a study of 21 Swedish households and their experiences of and ideas on how to manage disruptions in electricity supply. We call this perspective household energy resilience and identify three strategies for resilience: (1) response diversity, i.e., diversity in ways of carrying out normally electricity-dependent practices, (2) creating opportunities to develop resilience, and (3) building community energy resilience. Furthermore, we suggest how HCI can support these strategies, both by providing tools to increase resilience and by carefully designing technology and services to be more resilient in themselves. 

This paper responds to sustainable HCI’s call to design long-term participatory projects with grassroots communities to counter the local effects of climate change and support more viable practices. We contribute a methodological approach to participatory speculative design as a series of interrelated experiments in living, working in symbiosis with a food-growing community moving towards collective resilience and food sovereignty. As an example of sustainability research within HCI, community food-growing has predominantly focused on collaborative acts of growing rather than disagreements, divergences and frictions. Limited attention has been paid to the challenges of effectively negotiating collaborative, sustainable speculative futures in this context. This paper reports on a workshop series on sustainable community food-growing using situated participatory speculation to address potential tensions when working collaboratively towards socio-technical alternatives.

This paper discusses Digital Environmental Stewardship as an analytical framework that can help HCI scholarship to understand, design, and assess sociotechnical interventions concerned with sustainable waste management practices. Drawing on environmental studies, we outline key concepts of environmental stewardship – namely actors, capacity, and motivations – to unpack how different initiatives for handling waste are organised, both through grassroots and top-down interventions, and through varying sociotechnical configurations. We use these dimensions to analyse three different cases of waste management that illustrate how actions of care for the environment are ecologically organised, and what challenges might hinder them beyond –or besides– behavioural motivations. We conclude with a discussion on the orientation to action that the suggested framework provides, and its role in understanding, designing and assessing digital technologies in this domain. We argue that examining how stewardship actions fold into each other helps design sociotechnical interventions for managing waste from within a relational perspective.

Structured note-taking forms such as sketchnoting, self-tracking journals, and bullet journaling go beyond immediate capture of information scraps. Instead, hand-drawn pride-in-craftmanship increases perceived value for sharing and display. But hand-crafting lists, tables, and calendars is tedious and repetitive. To support these practices digitally, Style Blink (“Style-Blocks+Ink”) explores handcrafted styling as a first-class object. Style-blocks encapsulate digital ink, enabling people to craft, modify, and reuse embellishments and decorations for larger structures, and apply custom layouts. For example, we provide interaction instruments that style ink for personal expression, inking palettes that afford creative experimentation, fillable pens that can be “loaded” with commands and actions to replace menu selections, techniques to customize inked structures post-creation by modifying the underlying handcrafted style-blocks and to re-layout the overall structure to match users’ preferred template. In effect, any ink stroke, notation, or sketch can be encapsulated as a style-object and re-purposed as a tool. Feedback from 13 users show the potential of style adaptation and re-use in individual sketching practices.

While using VR, efficient text entry is a challenge: users cannot easily locate standard physical keyboards, and keys are often out of reach, e.g. when standing. We present TapGazer, a text entry system where users type by tapping their fingers in place. Users can tap anywhere as long as the identity of each tapping finger can be detected with sensors. Ambiguity between different possible input words is resolved by selecting target words with gaze. If gaze tracking is unavailable, ambiguity is resolved by selecting target words with additional taps. We evaluated TapGazer for seated and standing VR: seated novice users using touchpads as tap surfaces reached 44.81 words per minute (WPM), 79.17% of their QWERTY typing speed. Standing novice users tapped on their thighs with touch-sensitive gloves, reaching 45.26 WPM (71.91%). We analyze TapGazer with a theoretical performance model and discuss its potential for text input in future AR scenarios.

A key aspect of knowledge work is the analysis and manipulation of sets of related documents. We conducted interviews with 12 patent examiners and 12 scientists and found that all face difficulties using specialized tools for managing text from multiple documents across interconnected activities, including searching, collecting, annotating, organizing, writing and reviewing, while manually tracking their provenance. We introduce Passages, interactive objects that reify text selections and can then be manipulated, reused, and shared across multiple tools. Passages directly supports the above-listed activities as well as fluid transitions among them. Two user studies show that participants found Passages both elegant and powerful, facilitating their work practices and enabling greater reuse and novel strategies for analyzing and composing documents. We argue that Passages offers a general approach applicable to a wide variety of text-based interactions.

We present a QWERTY-based text entry system, TypeAnywhere, for use in off-desktop computing environments. Using a wearable device that can detect finger taps, users can leverage their touch-typing skills from physical keyboards to perform text entry on any surface. TypeAnywhere decodes typing sequences based only on finger-tap sequences without relying on tap locations. To achieve optimal decoding performance, we trained a neural language model and achieved a 1.6% character error rate (CER) in an offline evaluation, compared to a 5.3% CER from a traditional n-gram language model. Our user study showed that participants achieved an average performance of 70.6 WPM, or 80.4% of their physical keyboard speed, and 1.50% CER after 2.5 hours of practice over five days on a table surface. They also achieved 43.9 WPM and 1.37% CER when typing on their laps. Our results demonstrate the strong potential of QWERTY typing as a ubiquitous text entry solution.

The need to generate convincing simulation of voices often arises in the context of avatar therapy, a treatment approach for disorders such as schizophrenia. This treatment involves patients interacting with simulations of the entity they imagine to be responsible for the voices they hear, for which there is often no external reference available. However, in such scenarios, there is little knowledge of how to design and reproduce these voices in a convincing manner. Existing voice manipulation interfaces are often complex to use, and highly limited in their ability to modify vocal characteristics beyond small adjustments. To address these challenges, we designed a framework that allows users to explore and select from a large set of voices, and thereafter manipulate the voice(s) to converge towards an effective match for one they have in mind. We demonstrated both the usability and superior performance of this system compared to existing voice manipulation interfaces.

Various automated eating detection wearables have been proposed to monitor food intakes. While these systems overcome the forgetfulness of manual user journaling, they typically show low accuracy at outside-the-lab environments or have intrusive form-factors (e.g., headgear). Eyeglasses are emerging as a socially-acceptable eating detection wearable, but existing approaches require custom-built frames and consume large power. We propose MyDJ, an eating detection system that could be attached to any eyeglass frame. MyDJ achieves accurate and energy-efficient eating detection by capturing complementary chewing signals on a piezoelectric sensor and an accelerometer. We evaluated the accuracy and wearability of MyDJ with 30 subjects in uncontrolled environments, where six subjects attached MyDJ on their own eyeglasses for a week. Our study shows that MyDJ achieves 0.919 F1-score in eating episode coverage, with 4.03 × battery time over the state-of-the-art systems. In addition, participants reported wearing MyDJ was almost as comfortable (94.95%) as wearing regular eyeglasses.

Data Videos (DVs), or animated infographics that tell stories with data, are becoming increasingly popular. Despite their potential to induce attitude change, little is explored about how to produce effective DVs. This paper describes two studies that explored factors linked to the potential of health DVs to improve viewers’ behavioural change intentions. We investigated: 1) how viewers’ affect is linked to their behavioural change intentions; 2) how these affect are linked to the viewers’ personality traits; 3) which attributes of DVs are linked to their persuasive potential. Results from both studies indicated that viewers’ negative affect lowered their behavioural change intentions. Individuals with higher neuroticism exhibited higher negative affect and were harder to convince. Finally, Study 2 proved that providing any solutions to the health problem, presented in the DV, made the viewers perceive the videos as more actionable while lowering their negative affect, and importantly, induced higher behavioural change intentions.

Individuals and communities around the world are increasingly exposed to extreme heat as a result of climate change. Urban residents are particularly vulnerable to extreme heat due to the urban heat island effect. However, understanding an individual’s heat exposure and risk is difficult to assess due to variations in temperature within an urban environment. In this paper, we examine the potential for wearable temperature sensors to accurately measure personal heat exposure. We synthesize literature from fields spanning urban planning to public health and present the results of a user study validating a set of four commonly used off-the-shelf temperature sensors in two different urban settings across five on-body locations. Our investigation found that wearable temperature sensors are less reliable in highly urban areas and when worn in direct sunlight. We discuss important design considerations for wearable temperature sensors and identify actionable ways to improve future studies.

Image-based sexual abuse (IBSA) is a severe social problem that causes survivors tremendous pain. IBSA survivors may encounter a lack of information and victim blame when seeking online and offline assistance. While institutions support survivors, they cannot be available 24 hours a day. Because the immediate reaction to IBSA is crucial to remove intimate images and prevent further distribution, survivors need first responders who are always accessible and do not blame them. Chatbots are constantly available, do not judge the conversation partner, and may deliver structured information and words of comfort. Therefore, we developed a chatbot to provide information and emotional support to IBSA survivors in dealing with their abuse. We analyzed nine chatbots for sexual violence survivors to identify common design elements. In addition, we sought advice from five professional counselors about the challenges survivors have while responding to their harm. We conducted a user study with 25 participants to determine the chatbot’s effectiveness in providing information and emotional support compared to internet search. The chatbot was better than the internet search regarding information organization, accessibility, and conciseness. Furthermore, the chatbot excels in providing emotional support to survivors. We discuss the survivor-centered information structure and design consideration of emotionally supportive conversation.

Existing conversational approaches for Internet of Things (IoT) service mashup do not support modification because of the usability challenge, although it is common for users to modify the service mashups in IoT environments. To support the modification of IoT service mashups through conversational interfaces in a usable manner, we propose the conversational mashup modification agent (CoMMA). Users can modify IoT service mashups using CoMMA through natural language conversations. CoMMA has a two-step mashup modification interaction, an implicature-based localization step, and a modification step with a disambiguation strategy. The localization step allows users to easily search for a mashup by vocalizing their expressions in the environment. The modification step supports users to modify mashups by speaking simple modification commands. We conducted a user study and the results show that CoMMA is as effective as visual approaches in terms of task completion time and perceived task workload for modifying IoT service mashups.

We examine mid-air typing data collected from touch typists to evaluate the features and classification models for recognizing finger stroke. A large number of finger movement traces have been collected using finger motion capture systems, labeled into individual finger strokes, and classified into several key features. We test finger kinematic features, including 3D position, velocity, acceleration, and temporal features, including previous fingers and keys. Based on this analysis, we assess the performance of various classifiers, including Naive Bayes, Random Forest, Support Vector Machines, and Deep Neural Networks, in terms of the accuracy for correctly classifying the keystroke. We finally incorporate a linguistic heuristic to explore the effectiveness of the character prediction model and improve the total accuracy.

Touchscreen tracking latency, often 80ms or more, creates a rubber-banding effect in everyday direct manipulation tasks such as dragging, scrolling, and drawing. This has been shown to decrease system preference, user performance, and overall realism of these interfaces. In this research, we demonstrate how the addition of a thin, 2D micro-patterned surface with 5 micron spaced features can be used to reduce motor-visual touchscreen latency. When a finger, stylus, or tangible is translated across this textured surface frictional forces induce acoustic vibrations which naturally encode sliding velocity. This acoustic signal is sampled at 192kHz using a conventional audio interface pipeline with an average latency of 28ms. When fused with conventional low-speed, but high-spatial-accuracy 2D touch position data, our machine learning model can make accurate predictions of real time touch location.

Full-body tracking in virtual reality improves presence, allows interaction via body postures, and facilitates better social expression among users. However, full-body tracking systems today require a complex setup fixed to the environment (e.g., multiple lighthouses/cameras) and a laborious calibration process, which goes against the desire to make VR systems more portable and integrated. We present HybridTrak, which provides accurate, real-time full-body tracking by augmenting inside-out1 upper-body VR tracking systems with a single external off-the-shelf RGB web camera. HybridTrak uses a full-neural solution to convert and transform users’ 2D full-body poses from the webcam to 3D poses leveraging the inside-out upper-body tracking data. We showed HybridTrak is more accurate than RGB or depth-based tracking methods on the MPI-INF-3DHP dataset. We also tested HybridTrak in the popular VRChat app and showed that body postures presented by HybridTrak are more distinguishable and more natural than a solution using an RGBD camera.

Gaze and speech are rich contextual sources of information that, when combined, can result in effective and rich multimodal interactions. This paper proposes a machine learning-based pipeline that leverages and combines users’ natural gaze activity, the semantic knowledge from their vocal utterances and the synchronicity between gaze and speech data to facilitate users’ interaction. We evaluated our proposed approach on an existing dataset, which involved 32 participants recording voice notes while reading an academic paper. Using a Logistic Regression classifier, we demonstrate that our proposed multimodal approach maps voice notes with accurate text passages with an average F1-Score of 0.90. Our proposed pipeline motivates the design of multimodal interfaces that combines natural gaze and speech patterns to enable robust interactions.

Walking impairment is a debilitating symptom of Multiple Sclerosis (MS), a disease affecting 2.8 million people worldwide. While clinicians’ in-person observational gait assessments are important, research suggests that data from wearable sensors can indicate early onset of gait impairment, track patients’ responses to treatment, and support remote and longitudinal assessment. We present an inquiry into supporting the transition from research to clinical practice. Co-design by HCI, biomedical, neurology and rehabilitation researchers resulted in a data-rich interface prototype for augmented gait analysis based on visualized sensor data. We used this as a prompt in interviews with ten experienced clinicians from a range of MS rehabilitation roles. We find that clinicians value quantitative sensor data within a whole patient narrative, to help track specific rehabilitation goals, but identify a tension between grasping critical information quickly and more detailed understanding. Based on the findings we make design recommendations for data-rich remote rehabilitation interfaces.

Touchless input could transform clinical activity by allowing health professionals direct control over medical imaging systems in a sterile manner. Currently, users face the issues of being unable to directly manipulate imaging in aseptic environments, as well as needing to touch shared surfaces in other hospital areas. Unintended input is a key challenge for touchless interaction and could be especially disruptive in medical contexts. We evaluated four clutching techniques with 34 health professionals, measuring interaction performance and interviewing them to obtain insight into their views on clutching, and touchless control of medical imaging. As well as exploring the performance of the different clutching techniques, our analysis revealed an appetite for reliable touchless interfaces, a strong desire to reduce shared surface contact, and suggested potential improvements such as combined authentication and touchless control. Our findings can inform the development of novel touchless medical systems and identify challenges for future research.

Clinical decision support tools have typically focused on one-time support for diagnosis or prognosis, but have the ability to support providers in longitudinal planning of patient care regimens amidst infrastructural challenges. We explore an opportunity for technology support for discontinuing antidepressants, where clinical guidelines increasingly recommend gradual discontinuation over abruptly stopping to avoid withdrawal symptoms, but providers have varying levels of experience and diverse strategies for supporting patients through discontinuation. We conducted two studies with 12 providers, identifying providers’ needs in developing discontinuation plans and deriving design guidelines. We then iteratively designed and implemented AT Planner, instantiating the guidelines by projecting taper schedules and providing flexibility for adjustment. Provider feedback on AT Planner highlighted that discontinuation plans required balancing interpersonal and infrastructural constraints and surfaced the need for different technological support based on clinical experience. We discuss the benefits and challenges of incorporating flexibility and advice into clinical planning tools.

Forgetfulness is a primary factor of medication nonadherence, a problem that contributes to worse health outcomes and increased mortality among people with chronic conditions. Common strategies to address forgetfulness, such as timed reminders, have limited effectiveness. However, there is limited information about why these strategies fail. To address this gap, we conducted interviews with people who take medications daily and miss doses at least twice a month. We contribute a state-based Medication Routine Framework composed of four states (Wellness, New Task, Erratic, and Disruption) in two axes (regularity and time scale). Because most nonadherence due to forgetfulness occurs in nonroutine states (i.e., Erratic and Disruption state), we argue that improving technology for medication adherence requires designing for these states. In this paper, we describe each state in detail and discuss opportunities for adapting medication reminder strategies to overcome the challenges of nonroutine states.

Virtual Reality (VR) has potential for productive knowledge work, however, midair pointing with controllers or hand gestures does not offer the precision and comfort of traditional 2D mice. Directly integrating mice into VR is difficult as selecting targets in a 3D space is negatively impacted by binocular rivalry, perspective mismatch, and improperly calibrated control-display (CD) gain. To address these issues, we developed Depth-Adaptive Cursor , a 2D-mouse driven pointing technique for 3D selection with depth-adaptation that continuously interpolates the cursor depth by inferring what users intend to select based on the cursor position, the viewpoint, and the selectable objects. Depth-Adaptive Cursor uses a novel CD gain tool to compute a usable range of CD gains for general mouse-based pointing in VR. A user study demonstrated that Depth-Adaptive Cursor significantly improved performance compared with an existing mouse-based pointing technique without depth-adaption in terms of time (21.2%), error (48.3%), perceived workload, and user satisfaction.

We introduce OVRlap, a VR interaction technique that lets the user perceive multiple places at the same time from a first-person perspective. OVRlap achieves this by overlapping viewpoints. At any time, only one viewpoint is active, meaning that the user may interact with objects therein. Objects seen from the active viewpoint are opaque, whereas objects seen from passive viewpoints are transparent. This allows users to perceive multiple locations at once and easily switch to the one in which they want to interact. We compare OVRlap and a single-viewpoint technique in a study where 20 participants complete object-collection and monitoring tasks. We find that in both tasks, participants are significantly faster and move their head significantly less with OVRlap. We propose how the technique might be improved through automated switching of the active viewpoint and intelligent viewpoint rendering.

Rendering instant and intense impact feedback on users’ hands, limbs and head to enhance realism in virtual reality (VR) has been proposed in previous works, but impact on the body is still less discussed. With the body’s large surface area to utilize, numerous impact patterns can be rendered in versatile VR applications, e.g., being shot, blasted, punched or slashed on body in VR games. Herein we propose ImpactVest to render spatio-temporal multilevel impact force feedback on body. By independently controlling nine impactors in a 3 × 3 layout using elastic force, impact is generated at different levels, positions and time sequences for versatile spatial and temporal combinations. We conducted a just-noticeable difference (JND) study to understand users’ impact level distinguishability on the body. A time interval threshold study was then performed to ascertain what time interval thresholds between two impact stimuli should be used to distinguish from simultaneous impact, a continuous impact stroke and two discrete impact stimuli. Based on the results, we conducted a VR experience study to verify that impact feedback from ImpactVest enhances VR realism.

The maximum physical range of horizontal human eye movement is approximately 45°. However, in a natural gaze shift, the difference in the direction of the gaze relative to the frontal direction of the head rarely exceeds 25°. We name this region of 25° − 45° the “Kuiper Belt” in the eye-gaze interaction. We try to utilize this region to solve the Midas touch problem to enable a search task while reducing false input in the Virtual Reality environment. In this work, we conduct two studies to figure out the design principle of how we place menu items in the Kuiper Belt as an “out-of-natural angle” region of the eye-gaze movement, and determine the effectiveness and workload of the Kuiper Belt-based method. The results indicate that the Kuiper Belt-based method facilitated the visual search task while reducing false input. Finally, we present example applications utilizing the findings of these studies.

We can create Virtual Reality (VR) interactions that have no equivalent in the real world by remapping spacetime or altering users’ body representation, such as stretching the user’s virtual arm for manipulation of distant objects or scaling up the user’s avatar to enable rapid locomotion. Prior research has leveraged such approaches, what we call beyond-real techniques, to make interactions in VR more practical, efficient, ergonomic, and accessible. We present a survey categorizing prior movement-based VR interaction literature as reality-based, illusory, or beyond-real interactions. We survey relevant conferences (CHI, IEEE VR, VRST, UIST, and DIS) while focusing on selection, manipulation, locomotion, and navigation in VR. For beyond-real interactions, we describe the transformations that have been used by prior works to create novel remappings. We discuss open research questions through the lens of the human sensorimotor control system and highlight challenges that need to be addressed for effective utilization of beyond-real interactions in future VR applications, including plausibility, control, long-term adaptation, and individual differences.

How do people interact with computers? This fundamental question was asked by Card, Moran, and Newell in 1983 with a proposition to frame it as a question about human cognition – in other words, as a matter of how information is processed in the mind. Recently, the question has been reframed as one of adaptation: how do people adapt their interaction to the limits imposed by cognition, device design, and environment? The paper synthesizes advances toward an answer within the theoretical framework of computational rationality. The core assumption is that users act in accordance with what is best for them, given the limits imposed by their cognitive architecture and their experience of the task environment. This theory can be expressed in computational models that explain and predict interaction. The paper reviews the theoretical commitments and emerging applications in HCI, and it concludes by outlining a research agenda for future work.

Modern games make creative use of First- and Third-person perspectives (FPP and TPP) to allow the player to explore virtual worlds. Traditionally, FPP and TPP perspectives are seen as distinct concepts. Yet, Virtual Reality (VR) allows for flexibility in choosing perspectives. We introduce the notion of a perspective continuum in VR, which is technically related to the camera position and conceptually to how users perceive their environment in VR. A perspective continuum enables adapting and manipulating the sense of agency and involvement in the virtual world. This flexibility of perspectives broadens the design space of VR experiences through deliberately manipulating perception. In a study, we explore users’ attitudes, experiences and perceptions while controlling a virtual character from the two known perspectives. Statistical analysis of the empirical results shows the existence of a perspective continuum in VR. Our findings can be used to design experiences based on shifts of perception.

Online medical crowdfunding campaigns (OMCCs) help patients seek financial support. First impressions (FIs) of an OMCC, including perceived empathy, credibility, justice, impact, and attractiveness, could affect viewers’ donation decisions. Images play a crucial role in manifesting FIs, and it is beneficial for fundraisers to understand how viewers may judge their selected images for OMCCs beforehand. This work proposes a data-driven approach to assessing whether an OMCC image conveys appropriate FIs. We first crowdsource viewers’ perception of OMCC images. Statistical analysis confirms that agreement on all five dimensions of FIs exists, and these FIs positively correlate with donation intention. We compute image content, color, texture, and composition features, then analyze the correlation between these visual features and FIs. We further predict FIs based on these features, and the best model achieves an overall F1-score of 0.727. Finally, we discuss how our insights could benefit fundraisers and possible ethical concerns.

Affordance refers to the perception of possible actions allowed by an object. Despite its relevance to human–computer interaction, no existing theory explains the mechanisms that underpin affordance-formation; that is, how affordances are discovered and adapted via interaction. We propose an integrative theory of affordance-formation based on the theory of reinforcement learning in cognitive sciences. The key assumption is that users learn to associate promising motor actions to percepts via experience when reinforcement signals (success/failure) are present. They also learn to categorize actions (e.g., “rotating” a dial), giving them the ability to name and reason about affordance. Upon encountering novel widgets, their ability to generalize these actions determines their ability to perceive affordances. We implement this theory in a virtual robot model, which demonstrates human-like adaptation of affordance in interactive widgets tasks. While its predictions align with trends in human data, humans are able to adapt affordances faster, suggesting the existence of additional mechanisms.

Understanding decision-making in dynamic and complex settings is a challenge yet essential for preventing, mitigating, and responding to adverse events (e.g., disasters, financial crises). Simulation games have shown promise to advance our understanding of decision-making in such settings. However, an open question remains on how we extract useful information from these games. We contribute an approach to model human-simulation interaction by leveraging existing methods to characterize: (1) system states of dynamic simulation environments (with Principal Component Analysis), (2) behavioral responses from human interaction with simulation (with Hidden Markov Models), and (3) behavioral responses across system states (with Sequence Analysis). We demonstrate this approach with our game simulating drug shortages in a supply chain context. Results from our experimental study with 135 participants show different player types (hoarders, reactors, followers), how behavior changes in different system states, and how sharing information impacts behavior. We discuss how our findings challenge existing literature.

Social wearables promise to augment and enhance social interactions. However, despite two decades of HCI research on wearables, we are yet to see widespread adoption of social wearables into everyday life. More in-situ investigations into the social dynamics and cultural practices afforded by wearing interactive technology are needed to understand the drivers and barriers to adoption. To this end, we study social wearables in the context of Nordic student culture and the students’ practice of adorning boiler suits. Through a co-creation process, we designed Digi Merkki, a personalised interactive clothing patch. In a two-week elicitation diary study, we captured how 16 students adopted Digi Merkki into their social practices. We found that Digi Merkki afforded a variety of social interaction strategies, including sharing, spamming, and stealing pictures, which supported meaning-making and community-building. Based on our findings, we articulate “Memetic Expression” as a strong concept for designing social wearables.

Through collaborative playlists (CPs), streaming platform users have co-curated music together for various purposes for over a decade. As the COVID-19 pandemic has transformed how people come together through technology and engage with music, CPs have also taken on new roles and value. To understand how CP usage and perception have evolved since the onset of COVID-19, we conducted a mixed-methods investigation of CPs in the United States. Survey results from primarily CP users (N=142) revealed that interest in and usage of CPs have mostly increased since the pandemic, and that the role of music in connecting with others is positively correlated with the perceived impact of COVID-19. Follow-up interviews (N=9) provided additional insights into changing perceptions and usage patterns of CPs during COVID-19; for instance, fewer collaborators per playlist reflects users’ greater focus on strengthening social connections and relationships. Taken together, findings and design implications on digitally mediated co-curation further elucidate the necessity for social and collaborative experiences with music supported by CPs during COVID-19.

Cycling has become increasingly popular as a means of transportation. However, cyclists remain a highly vulnerable group of road users. According to accident reports, one of the most dangerous situations for cyclists are uncontrolled intersections, where cars approach from both directions. To address this issue and assist cyclists in crossing decision-making at uncontrolled intersections, we designed two visualizations that: (1) highlight occluded cars through an X-ray vision and (2) depict the remaining time the intersection is safe to cross via a Countdown. To investigate the efficiency of these visualizations, we proposed an Augmented Reality simulation as a novel evaluation method, in which the above visualizations are represented as AR, and conducted a controlled experiment with 24 participants indoors. We found that the X-ray ensures a fast selection of shorter gaps between cars, while the Countdown facilitates a feeling of safety and provides a better intersection overview.

Automated vehicles are expected to substitute driver-pedestrian communication via LED strips or displays. This communication is expected to improve trust and the crossing process in general. However, numerous factors such as other pedestrians’ behavior, perceived time pressure, or previous experience influence crossing decisions. Therefore, we report the results of a triply subdivided Virtual Reality study (N=18) evaluating these. Results show that external communication was perceived as hedonically pleasing, increased perceived safety and trust, and also that pedestrians’ behavior affected participants’ behavior. A timer did not alter crossing behavior, however, repeated exposure increased trust and reduced crossing times, showing a habituation effect. Our work helps better to integrate research on external communication in ecologically valid settings.

Runners want to actively explore unknown environments without the fear of getting lost. We conducted a survey to better understand runners’ needs and practices in this context. The survey results emphasized the interest in flexible exploration. To address this interest, we designed FlexNav, a system that supports exploratory running through flexible tours that link the best runnable zones in a neighbourhood. FlexNav provides adaptive navigation support enabling runners to follow such tours without continually getting disruptive directions. We tested it with runners to assess its usability. The results confirm the usefulness of the system and the users' preference for flexible tours over fully specified tours with turn-by-turn guidance. Our study highlights the subjective nature of runnable zones and the subtle balance between guidance and exploration.

The disadvantaged population is often underserved and marginalized in technology engagement: prior works show they are generally more reluctant and experience more barriers in adopting and engaging with mainstream technology. Here, we contribute to the HCI4D and ICTD literature through a novel “counter” case study on Chinese social commerce (e.g., Pinduoduo), which 1) first prospers among the traditionally underserved community from developing regions ahead of the more technologically advantaged communities, and 2) has been heavily engaged by this community. Through 12 in-depth interviews with social commerce users from the traditionally underserved community in Chinese developing regions, we demonstrate how social commerce, acting as a “virtual bazaar”, brings online the traditional offline socioeconomic lives the community has lived for ages, fits into the community’s social, cultural, and economic context, and thus effectively promotes technology inclusivity. Our work provides novel insights and implications for building inclusive technology for the “next billion” population.

The current mechanisms that drive the development of AI technologies are widely criticized for being tech-oriented and market-led instead of stemming from societal challenges. In Human-Centered AI discourses, and more broadly in Human-Computer Interaction research, initiatives have been proposed to engage experts from various domains of social science in determining how AI should reach our societies, predominantly through informing the adoption policies. Our contribution, however, seeks a more essential role for social sciences, namely to introduce discursive standpoints around what we need AI to be. With a focus on the domain of urbanism, the specific goal has been to elicit – from interviews with 16 urban experts – the imaginaries of how AI can and should impact future cities. Drawing on the social science literature, we present how the notion of “imaginary” has essentially framed this research and how it could reveal an alternative vision of non-human intelligent actors in future cities.

As interest within the HCI community expands beyond urban settings, novel tools and devices are being developed to support more sustainable interactions with natural environments and inform conservation action. Yet little is known about the users of these devices, and how their requirements and priorities might affect the usability or operationalization of the devices in the real world. Using the ‘e-seed’, a biomimetic self-drilling interface as a ‘research probe’, we conducted a qualitative user study with 14 subject matter experts in areas like forestry and agriculture to understand the value and limits for devices and systems in ecological restoration and monitoring. We highlight unique challenges in existing ecological practices, opportunities for technological interventions, and the policies and economic constraints affecting the feasibility of such interventions. We present a set of critical design considerations for building and deploying novel devices in natural and semi-natural ecosystems and discuss implications for future research.

Ridesharing services do not make data of their availability (supply, utilization, idle time, and idle distance) and surge pricing publicly available. It limits the opportunities to study the spatiotemporal trends of the availability and surge pricing of these services. Only a few research studies conducted in North America analyzed these features for only Uber and Lyft. Despite the interesting observations, the results of prior works are not generalizable or reproducible because: i) the datasets collected in previous publications are spatiotemporally sensitive, i.e., previous works do not represent the current availability and surge pricing of ridesharing services in different parts of the world; and ii) the analyses presented in previous works are limited in scope (in terms of countries and ridesharing services they studied). Hence, prior works are not generally applicable to ridesharing services operating in different countries.

This paper addresses the issue of ridesharing-data unavailability by presenting Ridesharing Measurement Suite (RMS). RMS removes the barrier of entry for analyzing the availability and surge pricing of ridesharing services for ridesharing users, researchers from various scientific domains, and regulators. RMS continuously collects the data of the availability and surge pricing of ridesharing services. It exposes real-time data of these services through i) graphical user interfaces and ii) public APIs to assist various stakeholders of these services and simplify the data collection and analysis process for future ridesharing research studies. To signify the utility of RMS, we deployed RMS to collect and analyze the availability and surge pricing data of 10 ridesharing services operating in nine countries for eight weeks in pre and during pandemic periods. Using the data collected and analyzed by RMS, we identify that previous articles miscalculated the utilization of ridesharing services as they did not count in the vehicles driving in multiple categories of the same service. We observe that during COVID-19, the supply of ridesharing services decreased by 54%, utilization of available vehicles increased by 6%, and a 5 × increase in the surge frequency of services. We also find that surge occurs in a small geographical region, and its intensity reduces by 50% in about 0.5 miles away from the location of a surge. We present several other interesting observations on ridesharing services’ availability and surge pricing.

We found significant gaps in the climates and built environments used as settings for studies of HCI outdoors. The experience of using a computer outdoors varies widely depending on location-specific factors such as weather and the availability of electricity. We surveyed 699 papers from CHI venues and found 101 studies involving a person and a computer interacting outdoors for which we could determine the study location. We categorized each study location by climate using the Köppen-Geiger scheme and by built environment using the Recreation Opportunity Spectrum. 91 of 101 studies took place in temperate or continental climates and 82 took place in urban settings. Emerging understanding of the ongoing impacts of climate change increases the importance of investigating HCI outdoors in a wider range of weather conditions. While some primitive natural settings have been preserved against development at great cost, we found no studies of HCI outdoors in those settings.

When modeling passive data to infer individual mental wellbeing, a common source of ground truth is self-reports. But these tend to represent the psychological facet of mental states, which might not align with the physiological facet of that state. Our paper demonstrates that when what people “feel” differs from what people “say they feel”, we witness a semantic gap that limits predictions. We show that predicting mental wellbeing with passive data (offline sensors or online social media) is related to how the ground-truth is measured (objective arousal or self-report). Features with psycho-social signals (e.g., language) were better at predicting self-reported anxiety and stress. Conversely, features with behavioral signals (e.g., sleep), were better at predicting stressful arousal. Regardless of the source of ground truth, integrating both signals boosted prediction. To reduce the semantic gap, we provide recommendations to evaluate ground truth measures and adopt parsimonious sensing.

Scientists and journalists strive to report numbers with high precision to keep readers well-informed. Our work investigates whether this practice can backfire due to the cognitive costs of processing multi-digit precise numbers. In a pre-registered randomized experiment, we presented readers with several news stories containing numbers in either precise or round versions. We then measured their ability to approximately recall these numbers and make estimates based on what they read. Our results revealed a counter-intuitive effect where reading round numbers helped people better approximate the precise values, while seeing precise numbers made them worse. We also conducted two surveys to elicit individual preferences for the ideal degree of rounding for numbers spanning seven orders of magnitude in various contexts. From the surveys, we found that people tended to prefer more precision when the rounding options contained only digits (e.g., ”2,500,000”) than when they contained modifier terms (e.g., ”2.5 million”). We conclude with a discussion of how these findings can be leveraged to enhance numeracy in digital content consumption.

The effective use of assistive interfaces (i.e. those that offer suggestions or reform the user’s input to match inferred intentions) depends on users making good decisions about whether and when to engage or ignore assistive features. However, prior work from economics and psychology shows systematic decision-making biases in which people overreact to low probability events and underreact to high probability events – modelled using a probability weighting function. We examine the theoretical implications of this probability weighting for interaction, including its suggestion that users will overuse inaccurate interface assistance and underuse accurate assistance. We then conduct a new analysis of data from a previously published study, quantifying the degree of bias users exhibited, and demonstrating conformance with these predictions. We discuss implications for design, including strategies that could be used to mitigate the deleterious effects of the observed biases.

The philosophical construct readiness-to-hand describes focused, intuitive, tool use, and has been linked to tool-embodiment and immersion. The construct has been influential in HCI and design for decades, but researchers currently lack appropriate measures and tools to investigate it empirically. To support such empirical work we investigate the possibility of operationalising readiness-to-hand in measurements of multfractality in movement, building on recent work in cognitive science. We conduct two experiments (N=44, N=30) investigating multifractality in mouse movements during a computer game, replicating prior results and contributing new findings. Our results show that multifractality correlates with dimensions associated with readiness-to-hand, including skill and task-engagement, during tool breakdown, task learning and normal play. We describe future possibilities for the application of these methods in HCI, supporting such work by sharing scripts and data (https://osf.io/2hm9u/), and introducing a new data-driven approach to parameter selection.

Typing on mobile devices is a common and complex task. The act of typing itself thereby encodes rich information, such as the typing method, the context it is performed in, and individual traits of the person typing. Researchers are increasingly using a selection or combination of experience sampling and passive sensing methods in real-world settings to examine typing behaviours. However, there is limited understanding of the effects these methods have on measures of input speed, typing behaviours, compliance, perceived trust and privacy. In this paper, we investigate the tradeoffs of everyday data collection methods. We contribute empirical results from a four-week field study (N=26). Here, participants contributed by transcribing, composing, passively having sentences analyzed and reflecting on their contributions. We present a tradeoff analysis of these data collection methods, discuss their impact on text-entry applications, and contribute a flexible research platform for in the wild text-entry studies.

With the rapid development of creativity support tools, creative practitioners (e.g., designers, artists, architects) have to constantly explore and adopt new tools into their practice. While HCI research has focused on developing novel creativity support tools, little is known about creative practitioner’s values when exploring and adopting these tools. We collect and analyze 23 videos, 13 interviews, and 105 survey responses of creative practitioners reflecting on their values to derive a value framework. We find that practitioners value the tools’ functionality, integration into their current workflow, performance, user interface and experience, learning support, costs and emotional connection, in that order. They largely discover tools through personal recommendations. To help unify and encourage reflection from the wider community of CST stakeholders (e.g., systems creators, researchers, marketers, educators), we situate the framework within existing research on systems, creativity support tools and technology adoption.

Creating digital comics involves multiple stages, some creative and some menial. For example, coloring a comic requires a labor-intensive stage known as ‘flatting,’ or masking segments of continuous color, as well as creative shading, lighting, and stylization stages. The use of AI can automate the colorization process, but early efforts have revealed limitations—technical and UX—to full automation. Via a formative study of professionals, we identify flatting as a bottleneck and key target of opportunity for human-guided AI-driven automation. Based on this insight, we built FlatMagic, an interactive, AI-driven flat colorization support tool for Photoshop. Our user studies found that using FlatMagic significantly reduced professionals’ real and perceived effort versus their current practice. While participants effectively used FlatMagic, we also identified potential constraints in interactions with AI and partially automated workflows. We reflect on implications for comic-focused tools and the benefits and pitfalls of intermediate representations and partial automation in designing human-AI collaboration tools for professionals.

Despite an abundance of carefully-crafted tutorials, trial-and-error remains many people’s preferred way to learn complex software. Yet, approaches to facilitate trial-and-error (such as tooltips) have evolved very little since the 1980s. While existing mechanisms work well for simple software, they scale poorly to large feature-rich applications. In this paper, we explore new techniques to support trial-and-error in complex applications. We identify key benefits and challenges of trial-and-error, and introduce a framework with a conceptual model and design space. Using this framework, we developed three techniques: ToolTrack to keep track of trial-and-error progress; ToolTrip to go beyond trial-and-error of single commands by highlighting related commands that are frequently used together; and ToolTaste to quickly and safely try commands. We demonstrate how these techniques facilitate trial-and-error, as illustrated through a proof-of-concept implementation in the CAD software Fusion 360. We conclude by discussing possible scenarios and outline directions for future research on trial-and-error.

Prior work has demonstrated augmented reality’s benefits to education, but current tools are difficult to integrate with traditional instructional methods. We present Paper Trail, an immersive authoring system designed to explore how to enable instructors to create AR educational experiences, leaving paper at the core of the interaction and enhancing it with various forms of digital media, animations for dynamic illustrations, and clipping masks to guide learning. To inform the system design, we developed five scenarios exploring the benefits that hand-held and head-worn AR can bring to STEM instruction and developed a design space of AR interactions enhancing paper based on these scenarios and prior work. Using the example of an AR physics handout, we assessed the system’s potential with PhD-level instructors and its usability with XR design experts. In an elicitation study with high-school teachers, we study how Paper Trail could be used and extended to enable flexible use cases across various domains. We discuss benefits of immersive paper for supporting diverse student needs and challenges for making effective use of AR for learning.

Guided by the 12 principles of animation, stylization is a core 2D animation feature but has been utilized mainly by experienced animators. Although there are tools for stylizing 2D animations, creating stylized 3D animations remains a challenging problem due to the additional spatial dimension and the need for responsive actions like contact and collision. We propose a system that helps users create stylized casual 3D animations. A layered authoring interface is employed to balance between ease of use and expressiveness. Our surface level UI is a timeline sequencer that lets users add preset stylization effects such as squash and stretch and follow through to plain motions. Users can adjust spatial and temporal parameters to fine-tune these stylizations. These edits are propagated to our node-graph-based second level UI, in which the users can create custom stylizations after they are comfortable with the surface level UI. Our system also enables the stylization of interactions among multiple objects like force, energy, and collision. A pilot user study has shown that our fluid layered UI design allows for both ease of use and expressiveness better than existing tools.

Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concrete subjects as well as 51 abstract and figurative styles. From this evaluation, we present design guidelines that can help people produce better outcomes from text-to-image generative models.

Although large language models (LLMs) have demonstrated impressive potential on simple tasks, their breadth of scope, lack of transparency, and insufficient controllability can make them less effective when assisting humans on more complex tasks. In response, we introduce the concept of Chaining LLM steps together, where the output of one step becomes the input for the next, thus aggregating the gains per step. We first define a set of LLM primitive operations useful for Chain construction, then present an interactive system where users can modify these Chains, along with their intermediate results, in a modular way. In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration. Additionally, we saw that users developed new ways of interacting with LLMs through Chains: they leveraged sub-tasks to calibrate model expectations, compared and contrasted alternative strategies by observing parallel downstream effects, and debugged unexpected model outputs by “unit-testing” sub-components of a Chain. In two case studies, we further explore how LLM Chains may be used in future applications.

In this paper, we present a natural language code synthesis tool, GenLine, backed by 1) a large generative language model and 2) a set of task-specific prompts that create or change code. To understand the user experience of natural language code synthesis with these new types of models, we conducted a user study in which participants applied GenLine to two programming tasks. Our results indicate that while natural language code synthesis can sometimes provide a magical experience, participants still faced challenges. In particular, participants felt that they needed to learn the model’s “syntax,” despite their input being natural language. Participants also struggled to form an accurate mental model of the types of requests the model can reliably translate and developed a set of strategies to debug model input. From these findings, we discuss design implications for future natural language code synthesis tools built using large generative language models.

Accurately recognizing icon types in mobile applications is integral to many tasks, including accessibility improvement, UI design search, and conversational agents. Existing research focuses on recognizing the most frequent icon types, but these technologies fail when encountering an unrecognized low-frequency icon. In this paper, we work towards complete coverage of icons in the wild. After annotating a large-scale icon dataset (327,879 icons) from iPhone apps, we found a highly uneven distribution: 98 common icon types covered 92.8% of icons, while 7.2% of icons were covered by more than 331 long-tail icon types. In order to label icons with widely varying occurrences in apps, our system uses an image classification model to recognize common icon types with an average of 3,000 examples each (96.3% accuracy) and applies a few-shot learning model to classify long-tail icon types with an average of 67 examples each (78.6% accuracy). Our system also detects contextual information that helps characterize icon semantics, including nearby text (95.3% accuracy) and modifier symbols added to the icon (87.4% accuracy). In a validation study with workers (n = 23), we verified the usefulness of our generated icon labels. The icon types supported by our work cover 99.5% of collected icons, improving on the previously highest 78% coverage in icon classification work.

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

Skin is a promising interaction medium and has been widely explored for mobile, and expressive interaction. Recent research in HCI has seen the development of Epidermal Computing Devices: ultra-thin and non-invasive devices which reside on the user’s skin, offering intimate integration with the curved surfaces of the body, while having physical and mechanical properties that are akin to skin, expanding the horizon of on-body interaction. However, with rapid technological advancements in multiple disciplines, we see a need to synthesize the main open research questions and opportunities for the HCI community to advance future research in this area. By systematically analyzing Epidermal Devices contributed in the HCI community, physical sciences research and from our experiences in designing and building Epidermal Devices, we identify opportunities and challenges for advancing research across five themes. This multi-disciplinary synthesis enables multiple research communities to facilitate progression towards more coordinated endeavors for advancing Epidermal Computing.

Wearable devices for life-logging and healthcare have been studied, but the need for frequent charging imposes inconvenience for long-term use. Integrating textile-based wireless chargers (i.e., coil) into clothing enables sustainable wearable computing by charging the on-body devices in use. However, the electromagnetic field generated by conventional coil chargers strongly interferes with human body, and the high resistance of conductive threads leads to inefficient power delivery. This paper presents Meander Coil++, enabling safe, energy-efficient, and body-scale wireless power delivery. Meander Coil++ uses a wiring pattern that suppresses electromagnetic exposure to the human body without compromising power delivery performance and a liquid-metal-based low-loss conductive cord. With these advancements, Meander Coil++ transmits a few watts of power to on-body devices at 25% DC-to-DC efficiency while complying with international safety guidelines regarding electromagnetic exposure. We envision Meander Coil++ can maintain multiple devices on body for weeks beyond the confines of their small battery capacity.

We present KnitSkin, a bio-inspired sleeve that can traverse diverse cylindrical terrains, ranging from a user’s forearm at a wearable scale, to pipes and tree branches at an environmental scale. Fabricated with a machine knitted substrate, the sleeve configures a stepped array of knitted scales that exhibit anisotropic friction. Coupled with the extension of actuators enclosed in the sleeve, the scales enable effective directional locomotion on cylindrical surfaces with varying slopes, textures, and curvatures. KnitSkin’s substrates are characterized by scales whose geometries and materials can be fine-tuned and channels that can accommodate diverse actuators. We introduce the design elements of KnitSkin in which we characterize a series of substrate parameters and their resulting anisotropic behaviors. In evaluating the locomotion, we examine the variables associated with the surface and actuator characteristics. KnitSkin obtains diverse applications across different scales, including wearable interfaces, industrial pipe-monitoring, to agricultural robots.

Wearable biosignal monitoring systems are becoming increasingly ubiquitous as tools for autonomous health monitoring and other real-world applications. Despite the continual advancements in this field, anthropometric considerations for women’s form are often overlooked in the design process, making systems ill fit and less effective. In this paper, we present a full garment assembly, ARGONAUT, with integrated textile electrocardiogram (ECG) electrodes in a 3-lead configuration that is designed specifically for women’s form. Through the exploration of materials, anthropometry, and garment assembly, we designed and tested ARGONAUT against the laboratory standard to determine performance through R-peak detection and noise interference. We investigated common issues faced when designing a wearable ECG garment, such as fit, motion artifact mitigation, and social wearability, to develop a dynamic design process that can be used to expand the advancing technology sensor integrated garments to all individuals in order to allow for equal access to potential health benefits.

In recent years, actuators that handle fluids such as gases and liquids have been attracting attention for their applications in soft robots and shape-changing interfaces. In the field of HCI, there have been various inflatable prototyping tools that utilize air control, however, very few tools for liquid control have been developed. In this study, we propose HydroMod, new constructive modules that can easily generate liquid flow and programmatically control liquid flow, with the aim of lowering the barrier to entry for prototyping with liquids. HydroMod consists of palm-sized small modules, which can generate liquid flow with the electrohydrodynamics (EHD) phenomenon by simply connecting the modules. Moreover, users can configure and control the flow path by simply recombining the modules. In this paper, we propose the design of the modules, evaluate the performance of HydroMod as a fluid system, and also show the possible application scenarios of fluid prototyping using this system.

Sexual-minority women (SMWs) in China are often subject to strong stigmatization and tend to have limited opportunities to connect with other SMWs in offline contexts. Although dating apps help them connect and seek social support, little is known about SMWs’ practices of self-disclosure and connection-building through those apps. To address this gap, we interviewed 43 SMW dating-app users in China. We found that these SMWs developed distinctive self-disclosure strategies, such as posting non-facial photos and implicitly disclosing their whereabouts by blending location information into photos that only those in the know could understand, to avoid interference from aggressive acquaintances and other risks of unintentional disclosure of their SMW identities. Moreover, they used dating apps not only to recognize other SMWs offline and build relationships with them, but to exchange emotional support in the process of SMW identity development. Our findings have design implications for supporting SMWs and improving their online dating experiences.

This paper investigates how the conceptions of gender in memes are central to socializing at hackathons. Drawing on a multi-sited ethnography of seven hackathons, I provide insight into how references to memes and informal technology culture shape interaction in local manifestations of this culture. The contribution of this paper is twofold. First, I show how vocabularies and artifacts of technology culture move between on and offline spaces. These findings have implications for HCI research that investigates questions of materiality in computer-mediated communication. Second, I show how even the mundane memes of technology culture can reveal the toxic masculinity and ideology of Incels. By tying these internet memes to a physical context, I unpack how humor can reveal and perpetuate the enduring masculine dominance of technology. I end with recommendations for increasing inclusivity at hackathons, based on how HCI is uniquely positioned to understand how Internet symbols and interactions manifest offline.

Conference authorship, attendance and presentation is a key measure of quality in the HCI academic, yet we know conferences are not equally accessible for reasons that have nothing to do with quality. In this paper we examine two axes of diversity: gender, and geographic location (of both authors and conferences) and examine how they affect participation at major HCI conferences. Diversity in a group is associated with better outcomes from its work, and HCI has made numerous contributions to increasing representation in other communities. Reflecting on our own situation can produce recommendations for the planning of future HCI conferences, and identify challenges of representation that the HCI community should endeavor to address.

Women make up approximately half of the workforce in ride-hailing, food delivery, and home service platforms in North America. While studies have reported that gig workers face bias, harassment, and a gender pay gap, we have limited understanding of women’s perspectives of these issues and their coping mechanisms. We interviewed 20 women gig workers to hear their unique experiences with these challenges. We found that gig platforms are gender-agnostic, meaning they do not acknowledge women’s experiences and the value they bring. By not enforcing anti-harassment policies in design, gig platforms also leave women workers vulnerable to bias and harassment. Due to the lack of support for immediate actions and in fear of losing access to work, women workers “brush off” harassment. In addition, the platforms’ dispatching and recommendation mechanisms do not acknowledge women’s contributions in perceived safety for customers and social support for peer workers.

People write personalized greeting cards on various occasions. While prior work has studied gender roles in greeting card messages, systematic analysis at scale and tools for raising the awareness of gender stereotyping remain under-investigated. To this end, we collect a large greeting card message corpus covering three different occasions (birthday, Valentine’s Day and wedding) from three sources (exemplars from greeting message websites, real-life greetings from social media and language model generated ones). We uncover a wide range of gender stereotypes in this corpus via topic modeling, odds ratio and Word Embedding Association Test (WEAT). We further conduct a survey to understand people’s perception of gender roles in messages from this corpus and if gender stereotyping is a concern. The results show that people want to be aware of gender roles in the messages, but remain unconcerned unless the perceived gender roles conflict with the recipient’s true personality. In response, we developed GreetA, an interactive visualization and writing assistant tool to visualize fine-grained topics in greeting card messages drafted by the users and the associated gender perception scores, but without suggesting text changes as an intervention.

Haptic Feedback is essential for lifelike Virtual Reality (VR) experiences. To provide a wide range of matching sensations of being touched or stroked, current approaches typically need large numbers of different physical textures. However, even advanced devices can only accommodate a limited number of textures to remain wearable. Therefore, a better understanding is necessary of how expectations elicited by different visualizations affect haptic perception, to achieve a balance between physical constraints and great variety of matching physical textures.

In this work, we conducted an experiment (N=31) assessing how the perception of roughness is affected within VR. We designed a prototype for arm stroking and compared the effects of different visualizations on the perception of physical textures with distinct roughnesses. Additionally, we used the visualizations’ real-world materials, no-haptics and vibrotactile feedback as baselines. As one result, we found that two levels of roughness can be sufficient to convey a realistic illusion.

A significant drawback of text passwords for end-user authentication is password reuse. We propose a novel approach to detect password reuse by leveraging gaze as well as typing behavior and study its accuracy. We collected gaze and typing behavior from 49 users while creating accounts for 1) a webmail client and 2) a news website. While most participants came up with a new password, 32% reported having reused an old password when setting up their accounts. We then compared different ML models to detect password reuse from the collected data. Our models achieve an accuracy of up to 87.7% in detecting password reuse from gaze, 75.8% accuracy from typing, and 88.75% when considering both types of behavior. We demonstrate that using gaze, password reuse can already be detected during the registration process, before users entered their password. Our work paves the road for developing novel interventions to prevent password reuse.

What do pedestrian crossings, ATMs, elevators and ticket machines have in common? These are just a few of the ubiquitous yet essential elements of public-space infrastructure that rely on physical buttons or touchscreens; common interactions that, until recently, were considered perfectly safe to perform. This work investigates how we might integrate touchless technologies into public-space infrastructure in order to minimise physical interaction with shared devices in light of the ongoing COVID-19 pandemic. Drawing on an ethnographic exploration into how public utilities are being used, adapted or avoided, we developed and evaluated a suite of technology probes that can be either retrofitted into, or replace, these services. In-situ community deployments of our probes demonstrate strong uptake and provide insight into how hands-free technologies can be adapted and utilised for the public domain; and, in turn, used to inform the future of walk-up-and use public technologies.

Advanced technologies are increasingly enabling the creation of interactive devices with non-rectangular form-factors but it is currently unclear what alternative form-factors are desirable for end-users. We contribute an understanding of the interplay between the rationale for the form factors of such devices and their interactive content through think-aloud design sessions in which participants could mold devices as they wished using clay. We analysed their qualitative reflections on how the shapes affected interaction. Using thematic analysis, we identified shape features desirable on handheld freeform devices and discuss the particularity of three themes central to the choice of form factors: freeform dexterity, shape features discoverability and shape adaptability (to the task and context). In a second study following the same experimental set-up, we focused on the trade off between dexterity and discoverability and the relation to the concept of affordance. Our work reveals the shape features that impact the most the choice of grasps on freeform devices from which we derive design guidelines for the design of such devices.

Knowledge of users’ affective states can improve their interaction with smartphones by providing more personalized experiences (e.g., search results and news articles). We present an affective state classification model based on data gathered on smartphones in real-world environments. From touch events during keystrokes and the signals from the inertial sensors, we extracted two-dimensional heat maps as input into a convolutional neural network to predict the affective states of smartphone users. For evaluation, we conducted a data collection in the wild with 82 participants over 10 weeks. Our model accurately predicts three levels (low, medium, high) of valence (AUC up to 0.83), arousal (AUC up to 0.85), and dominance (AUC up to 0.84). We also show that using the inertial sensor data alone, our model achieves a similar performance (AUC up to 0.83), making our approach less privacy-invasive. By personalizing our model to the user, we show that performance increases by an additional 0.07 AUC.

The opaque data practices in smart home devices have raised significant privacy concerns for smart home users and bystanders. One way to learn about the data practices is through privacy-related notifications. However, how to deliver these notifications to users and bystanders and increase their awareness of data practices is not clear. We surveyed 136 users and 123 bystanders to understand their preferences of receiving privacy-related notifications in smart homes. We further collected their responses to four mechanisms that improve privacy awareness (e.g., Data Dashboard) as well as their selections of mechanisms in four different scenarios (e.g., friend visiting). Our results showed the pros and cons of each privacy awareness mechanism, e.g., Data Dashboard can help reduce bystanders’ dependence on users. We also found some unique benefits of each mechanism (e.g., Ambient Light could provide unobtrusive privacy awareness). We summarized four key design dimensions for future privacy awareness mechanisms design.

Home energy management systems (HEMS) offer control and the ability to manage energy, generating and collecting energy consumption data at the most detailed level. However, data at this level poses various privacy concerns, including, for instance, profiling consumer behaviors and large-scale surveillance. The question of how utility providers can get value from such data without infringing consumers’ privacy has remained under-investigated. We address this gap by exploring the pro-sharing attitudes and privacy perceptions of 30 HEMS users and non-users through an interview study. While participants are concerned about data misuse and stigmatization, our analysis also reveals that incentives, altruism, trust, security and privacy, transparency and accountability encourage data sharing. From this analysis, we derive privacy design strategies for HEMS that can both improve privacy and engender adoption.

We conducted a user study with 380 Android users, profiling them according to two key privacy behaviors: the number of apps installed and the Dangerous permissions granted to those apps. We identified four unique privacy profiles: 1) Privacy Balancers (49.74% of participants), 2) Permission Limiters (28.68%), 3) App Limiters (14.74%), and 4) the Privacy Unconcerned (6.84%). App and Permission Limiters were significantly more concerned about perceived surveillance than Privacy Balancers and the Privacy Unconcerned. App Limiters had the lowest number of apps installed on their devices with the lowest intention of using apps and sharing information with them, compared to Permission Limiters who had the highest number of apps installed and reported higher intention to share information with apps. The four profiles reflect the differing privacy management strategies, perceptions, and intentions of Android users that go beyond the binary decision to share or withhold information via mobile apps.

With the growing smartphone penetration rate, smartphone settings remain one of the main models for information privacy and security controls. Yet, their usability is largely understudied, especially with respect to the usability impact on underrepresented socio-economic and low-tech groups. In an online survey with 178 users, we find that many people are not aware of smartphone privacy and security settings, their defaults, and have not configured them in the past, but are willing to do it in the future. Some participants perceive low self-efficacy and expect difficulties and usability issues with configuring those settings. Finally, we find that certain socio-demographic groups are more vulnerable to risks and feel less prepared to use smartphone settings to protect their online privacy and security.

The confusion matrix, a ubiquitous visualization for helping people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. We conduct formative research with machine learning practitioners at Apple and find that conventional confusion matrices do not support more complex data-structures found in modern-day applications, such as hierarchical and multi-output labels. To express such variations of confusion matrices, we design an algebra that models confusion matrices as probability distributions. Based on this algebra, we develop Neo, a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications. Finally, we demonstrate Neo’s utility with three model evaluation scenarios that help people better understand model performance and reveal hidden confusions.

With the wide usage of data visualizations, a huge number of Scalable Vector Graphic (SVG)-based visualizations have been created and shared online. Accordingly, there has been an increasing interest in exploring how to retrieve perceptually similar visualizations from a large corpus, since it can benefit various downstream applications such as visualization recommendation. Existing methods mainly focus on the visual appearance of visualizations by regarding them as bitmap images. However, the structural information intrinsically existing in SVG-based visualizations is ignored. Such structural information can delineate the spatial and hierarchical relationship among visual elements, and characterize visualizations thoroughly from a new perspective. This paper presents a structure-aware method to advance the performance of visualization retrieval by collectively considering both the visual and structural information. We extensively evaluated our approach through quantitative comparisons, a user study and case studies. The results demonstrate the effectiveness of our approach and its advantages over existing methods.

Data visualizations are created and shared on the web at an unprecedented speed, raising new needs and questions for processing and analyzing visualizations after they have been generated and digitized. However, existing formalisms focus on operating on a single visualization instead of multiple visualizations, making it challenging to perform analysis tasks such as sorting and clustering visualizations. Through a systematic analysis of previous work, we abstract visualization-related tasks into mathematical operators such as union and propose a design space of visualization operations. We realize the design by developing ComputableViz, a library that supports operations on multiple visualization specifications. To demonstrate its usefulness and extensibility, we present multiple usage scenarios concerning processing and analyzing visualization, such as generating visualization embeddings and automatically making visualizations accessible. We conclude by discussing research opportunities and challenges for managing and exploiting the massive visualizations on the web.

The promise of visualization recommendation systems is that analysts will be automatically provided with relevant and high-quality visualizations that will reduce the work of manual exploration or chart creation. However, little research to date has focused on what analysts value in the design of visualization recommendations. We interviewed 18 analysts in the public health sector and explored how they made sense of a popular in-domain dataset1 in service of generating visualizations to recommend to others. We also explored how they interacted with a corpus of both automatically- and manually-generated visualization recommendations, with the goal of uncovering how the design values of these analysts are reflected in current visualization recommendation systems. We find that analysts champion simple charts with clear takeaways that are nonetheless connected with existing semantic information or domain hypotheses. We conclude by recommending that visualization recommendation designers explore ways of integrating context and expectation into their systems.

Data exploration systems have become popular tools with which data analysts and others can explore raw data and organize their observations. However, users of such systems who are unfamiliar with their datasets face several challenges when trying to extract data events of interest to them. Those challenges include progressively discovering informative charts, organizing them into a logical order to depict a meaningful fact, and arranging one or more facts to illustrate a data event. To alleviate them, we propose VisGuide—a data exploration system that generates personalized recommendations to aid users’ discovery of data events in breadth and depth by incrementally learning their data exploration preferences and recommending meaningful charts tailored to them. As well as user preferences, VisGuide’s recommendations simultaneously consider sequence organization and chart presentation. We conducted two user studies to evaluate 1) the usability of VisGuide and 2) user satisfaction with its recommendation system. The results of those studies indicate that VisGuide can effectively help users create coherent and user-oriented visualization trees that represent meaningful data events.

Urban accessibility assessments are challenging: they involve varied stakeholders across decision-making contexts while serving a diverse population of people with disabilities. To better support urban accessibility assessment using data visualizations, we conducted a three-part interview study with 25 participants across five stakeholder groups using map visualization probes. We present a multi-stakeholder analysis of visualization needs and sensemaking processes to explore how interactive visualizations can support stakeholder decision making. In particular, we elaborate how stakeholders’ varying levels of familiarity with accessibility, geospatial analysis, and specific geographic locations influences their sensemaking needs. We then contribute 10 design considerations for geovisual analytic tools for urban accessibility communication, planning, policymaking, and advocacy.

Immersive virtual reality (VR) is being used as an enriching experience for people living in residential aged care, or nursing homes, where care staff play a critical role supporting clients to use VR. In HCI research concerned with technology use in aged care, however, the role of formal caregivers has received limited attention. We conducted interviews with 11 caregivers working in care homes that have implemented VR as part of the social program offered to residents. Our findings highlight tensions between the opportunities created by the immersive VR experience and the risks and challenges full immersion presents for people in aged care. In this paper, we draw on an ethics of care framework to make visible the care practices involved in facilitating VR in aged care homes, highlighting the care required to ensure that older adults experience benefits when using immersive VR, while risks and challenges are carefully managed.

The ability to cross the street at intersections is an essential skill, often taught to people who are blind or have low vision (BLV) with the aid of tactile maps and kits or toys. However, each of the existing mapping tools has shortcomings. We investigated whether co-designed 3D printed components can offer benefits. Guided by consultation with 11 Orientation and Mobility (O&M) professionals, we co-designed a series of 3D printed kits that they then used in their practice with BLV children who showed high levels of engagement and learning. The 3D materials were found to demonstrate the key concepts for street crossings in a portable, engaging and professional manner. They will be released for free download, enabling O&M professionals to access or modify the materials as required. We hope that use of our co-designed 3D printed tools will contribute to the safety, independence and inclusion of BLV people.

Current activity tracking technologies are largely trained on younger adults’ data, which can lead to solutions that are not well-suited for older adults. To build activity trackers for older adults, it is crucial to collect training data with them. To this end, we examine the feasibility and challenges with older adults in collecting activity labels by leveraging speech. Specifically, we built MyMove, a speech-based smartwatch app to facilitate the in-situ labeling with a low capture burden. We conducted a 7-day deployment study, where 13 older adults collected their activity labels and smartwatch sensor data, while wearing a thigh-worn activity monitor. Participants were highly engaged, capturing 1,224 verbal reports in total. We extracted 1,885 activities with corresponding effort level and timespan, and examined the usefulness of these reports as activity labels. We discuss the implications of our approach and the collected dataset in supporting older adults through personalized activity tracking technologies.

As artificial agents proliferate, there will be more and more situations in which they must communicate their capabilities to humans, including what they can “see.” Artificial agents have existed for decades in the form of computer-controlled agents in videogames. We analyze videogames in order to not only inspire the design of better agents, but to stop agent designers from replicating research that has already been theorized, designed, and tested in-depth. We present a qualitative thematic analysis of sight cues in videogames and develop a framework to support human-agent interaction design. The framework identifies the different locations and stimulus types – both visualizations and sonifications – available to designers and the types of information they can convey as sight cues. Insights from several other cue properties are also presented. We close with suggestions for implementing such cues with existing technologies to improve the safety, privacy, and efficiency of human-agent interactions.

The lack of reliable, personalized information often complicates sexual violence survivors’ support-seeking. Recently, there is an emerging approach to conversational information systems for support-seeking of sexual violence survivors, featuring personalization with wide availability and anonymity. However, a single best solution might not exist as sexual violence survivors have different needs and purposes in seeking support channels. To better envision conversational support-seeking systems for sexual violence survivors, we explore personalization trade-offs in designing such information systems. We implement a high-fidelity prototype dialogue-based information system through four design workshop sessions with three professional caregivers and interviewed with four self-identified survivors using our prototype. We then identify two forms of personalization trade-offs for conversational support-seeking systems: (1) specificity and sensitivity in understanding users and (2) relevancy and inclusiveness in providing information. To handle these trade-offs, we propose a reversed approach that starts from designing information and inclusive tailoring that considers unspecified needs, respectively.

We present a case-study of using Ethnographic Experiential Futures (EXF) to surface underlying divergences, tensions and dilemmas implicit in views of the futures of ”social agents” among professional researchers familiar with the state of the art. Based on expert interviews, we designed three ”letters from the future,” research probes that were mailed to 15 participants working in the field, to encounter and respond to. We lay out the elements and design choices that shaped these probes, present our remote and asynchronous study design, and discuss lessons learned about the use of EXF. We find that this form of hybrid design/futures intervention has the potential to provide professional communities with opportunities to grapple with potential ethical dilemmas early on. However, the knowledge and tools for doing so are still in the making. Our contribution is a step towards advancing the potential benefits of experiential futures for technology designers and researchers.

With the advancements in AI, agents (i.e., smart products, robots, software agents) are increasingly capable of working closely together with humans in a variety of ways while benefiting from each other. These human-agent collaborations have gained growing attention in the HCI community; however, the field lacks clear guidelines on how to design the agents’ behaviors in collaborations. In this paper, the qualities that are relevant for designers to create robust and pleasant human-agent collaborations were investigated. Bratman's Shared Cooperative Activity framework was used to identify the core characteristics of collaborations and survey the most important issues in the design of human-agent collaborations, namely code-of-conduct, task delegation, autonomy and control, intelligibility, common ground, offering help and requesting help. The aim of this work is to add structure to this growing and important facet of HCI research and operationalize the concept of human-agent collaboration with concrete design considerations.

Augmented Reality (AR) embeds virtual content in physical spaces, including virtual agents that are known to exert a social presence on users. Existing design guidelines for AR rarely consider the social implications of an agent’s personal space (PS) and that it can impact user behavior and arousal. We report an experiment (N=54) where participants interacted with agents in an AR art gallery scenario. When participants approached six virtual agents (i.e., two males, two females, a humanoid robot, and a pillar) to ask for directions, we found that participants respected the agents’ PS and modulated interpersonal distances according to the human-like agents’ perceived gender. When participants were instructed to walk through the agents, we observed heightened skin-conductance levels that indicate physiological arousal. These results are discussed in terms of proxemic theory that result in design recommendations for implementing pervasive AR experiences with virtual agents.

Research on intelligent virtual agents (IVAs) often concerns the implementation of human-like behavior by integrating artificial intelligence algorithms. Thus far, few studies focused on mimicry of cognitive imperfections inherent to humans in IVAs. Neglecting to implement such imperfect behavior in IVAs might result in less believable or engaging human-agent interactions. In this paper, we simulate human imprecision in conversational IVAs’ temporal statements. We conducted a survey to identify temporal statement patterns, transferred them to a conversational IVA, and conducted a user study evaluating the effects of time precision on perceived anthropomorphism and usefulness. Statistical analyses reveal significant interaction between time precision and agents’ use of memory aids, indicating that (i) imprecise agents are perceived as more human-like than precise agents when responding immediately, and (ii) unnaturally high levels of temporal precision can be compensated for by memory aid use. Further findings underscore the value of continued inquiry into cultural variations.

Behaviour in virtual environments might be informed by our experiences in physical environments, but virtual environments are not constrained by the same physical, perceptual, or social cues. Instead of replicating the properties of physical spaces, one can create virtual experiences that diverge from reality by dynamically manipulating environmental, aural, and social properties. This paper explores digital proxemics, which describe how we use space in virtual environments and how the presence of others influences our behaviours, interactions, and movements. First, we frame the open challenges of digital proxemics in terms of activity, social signals, audio design, and environment. We explore a subset of these challenges through an evaluation that compares two audio designs and two displays with different social signal affordances: head-mounted display (HMD) versus desktop PC. We use quantitative methods using instrumented tracking to analyse behaviour, demonstrating how personal space, proximity, and attention compare between desktop PC and HMDs.

Virtual humans can be used to deliver persuasive arguments; yet, those with synthetic text-to-speech (TTS) have been perceived less favorably than those with recorded human speech. In this paper, we investigate standard concatenative TTS and more advanced neural TTS. We conducted a 3x2 between-subjects experiment (n=79) to evaluate the effect of a virtual human’s speech fidelity at three levels (Standard TTS, Neural TTS, and Human speech) and the listener’s gender (male or female) on perceptions and persuasion. We found that the virtual human was perceived as significantly less trustworthy by both genders, if they used neural TTS compared to human speech, while male listeners (but not females) also perceived standard TTS as less trustworthy than human speech. Our findings indicate that neural TTS may not be an effective choice for persuasive virtual humans and that gender of the listener plays a role in how virtual humans are perceived.

Visualizing biosignals can be important for social Virtual Reality (VR), where avatar non-verbal cues are missing. While several biosignal representations exist, designing effective visualizations and understanding user perceptions within social VR entertainment remains unclear. We adopt a mixed-methods approach to design biosignals for social VR entertainment. Using survey (N=54), context-mapping (N=6), and co-design (N=6) methods, we derive four visualizations. We then ran a within-subjects study (N=32) in a virtual jazz-bar to investigate how heart rate (HR) and breathing rate (BR) visualizations, and signal rate, influence perceived avatar arousal, user distraction, and preferences. Findings show that skeuomorphic visualizations for both biosignals allow differentiable arousal inference; skeuomorphic and particles were least distracting for HR, whereas all were similarly distracting for BR; biosignal perceptions often depend on avatar relations, entertainment type, and emotion inference of avatars versus spaces. We contribute HR and BR visualizations, and considerations for designing social VR entertainment biosignal visualizations.

Recent work has investigated the construction of touch-sensitive knitted fabrics, capable of being manufactured at scale, and having only two connections to external hardware. Additionally, several sensor design patterns and application prototypes have been introduced. Our aim is to start shaping the future of this technology according to user expectations. Through a formative focus group study, we explore users’ views of using these fabrics in different contexts and discuss potential concerns and application areas. Subsequently, we take steps toward addressing relevant questions, by first providing design guidelines for application designers. Furthermore, in one user study, we demonstrate that it is possible to distinguish different swipe gestures and identify accidental contact with the sensor, a common occurrence in everyday life. We then present experiments investigating the effect of stretching and laundering of the sensors on their resistance, providing insights about considerations necessary to include in computational models.

Moving a slider to set the music volume or control the air conditioning is a familiar task that requires little attention. However, adjusting a virtual slider on a featureless touchscreen is much more demanding and can be dangerous in situations such as driving. Here, we study how a gradual tactile feedback, provided by a haptic touchscreen, can replace visual cues. As users adjust a setting with their finger, they feel a continuously changing texture, which spatial frequency correlates to the value of the setting. We demonstrate that, after training with visual and auditory feedback, users are able to adjust a setting on a haptic touchscreen without looking at the screen, thereby reducing visual distraction. Every learning strategy yielded similar performance, suggesting an amodal integration. This study shows that surface haptics can provide intuitive and precise tuning possibilities for tangible interfaces on touchscreens.

With the proliferation of shape-change research in affective computing, there is a need to deepen understandings of affective responses to shape-change display. Little research has focused on affective reactions to tactile experiences in shape-change, particularly in the absence of visual information. It is also rare to study response to the shape-change as it unfolds, isolated from a final shape-change outcome. We report on two studies on touch-affect associations, using the crossmodal “Bouba-Kiki” paradigm, to understand affective responses to shape-change as it unfolds. We investigate experiences with a shape-change gadget, as it moves between rounded (“Bouba”) and spiky (“Kiki”) forms. We capture affective responses via the circumplex model, and use a motion analysis approach to understand the certainty of these responses. We find that touch-affect associations are influenced by both the size and the frequency of the shape-change and may be modality-dependent, and that certainty in affective associations is influenced by association-consistency.

Augmented reality (AR) and virtual reality (VR) technologies create exciting new opportunities for people to interact with computing resources and information. Less exciting is the need for holding hand controllers, which limits applications that demand expressive, readily available interactions. Prior research investigated freehand AR/VR input by transforming the user’s body into an interaction medium. In contrast to previous work that has users’ hands grasp virtual objects, we propose a new interaction technique that lets users’ hands become virtual objects by imitating the objects themselves. For example, a thumbs-up hand pose is used to mimic a joystick. We created a wide array of interaction designs around this idea to demonstrate its applicability in object retrieval and interactive control tasks. Collectively, we call these interaction designs Hand Interfaces. From a series of user studies comparing Hand Interfaces against various baseline techniques, we collected quantitative and qualitative feedback, which indicates that Hand Interfaces are effective, expressive, and fun to use.

Interacting with not only virtual but also real objects, or even virtual objects augmented by real objects becomes a trend of virtual reality (VR) interactions and is common in augmented reality (AR). However, current haptic shape rendering devices generally focus on feedback of virtual objects, and require the users to put down or take off those devices to perceive real objects. Therefore, we propose FingerX to render haptic shapes and enable users to touch, grasp and interact with virtual and real objects simultaneously. An extender on the fingertip extends to a corresponding height to support between the fingertip and the real objects or the hand, to render virtual shapes. A ring rotates and withdraws the extender behind the fingertip when touching real objects. By independently controlling four extenders and rings on each finger with the exception of the pinky finger, FingerX renders feedback in three common scenarios, including touching virtual objects augmented by real environments (e.g., a desk), grasping virtual objects augmented by real objects (e.g., a bottle) and grasping virtual objects in the hand. We conducted a shape recognition study to evaluate the recognition rates for these three scenarios and obtained an average recognition rate of 76.59% with shape visual feedback. We then performed a VR study to observe how users interact with virtual and real objects simultaneously and verify that FingerX significantly enhances VR realism, compared to current vibrotactile methods.

Providing haptic feedback in virtual reality to make the experience more realistic has become a strong focus of research in recent years. The resulting haptic feedback systems differ greatly in their technologies, feedback possibilities, and overall realism making it challenging to compare different systems. We propose the Haptic Fidelity Framework providing the means to describe, understand and compare haptic feedback systems. The framework locates a system in the spectrum of providing realistic or abstract haptic feedback using the Haptic Fidelity dimension. It comprises 14 criteria that either describe foundational or limiting factors. A second Versatility dimension captures the current trade-off between highly realistic but application-specific and more abstract but widely applicable feedback. To validate the framework, we compared the Haptic Fidelity score to the perceived feedback realism of evaluations from 38 papers and found a strong correlation suggesting the framework accurately describes the realism of haptic feedback.

Voice interactive devices often use keyword spotting for device activation. However, this approach suffers from misrecognition of keywords and can respond to keywords not intended for calling the device (e.g., ”You can ask Alexa about it.”), causing accidental device activations. We propose a method that leverages prosodic features to differentiate calling/not-calling voices (F1 score: 0.869), allowing devices to respond only when called upon to avoid misactivation. As a proof of concept, we built a prototype smart speaker called Aware that allows users to control the device activation by speaking the keyword in specific prosody patterns. These patterns are chosen to represent people’s natural calling/not-calling voices, which are uncovered in a study to collect such voices and investigate their prosodic difference. A user study comparing Aware with Amazon Echo shows Aware can activate more correctly (F1 score 0.93 vs. 0.56) and is easy to learn and use.

This paper’s goal is to understand the haptic-visual congruency perception of skin-slip on the fingertips given visual cues in Virtual Reality (VR). We developed SpinOcchio (Spin for the spinning mechanism used, Occhio for the Italian word “eye”), a handheld haptic controller capable of rendering the thickness and slipping of a virtual object pinched between two fingers. This is achieved using a mechanism with spinning and pivoting disks that apply a tangential skin-slip movement to the fingertips. With SpinOcchio, we determined the baseline haptic discrimination threshold for skin-slip, and, using these results, we tested how haptic realism of motion and thickness is perceived with varying visual cues in VR. Surprisingly, the results show that in all cases, visual cues dominate over haptic perception. Based on these results, we suggest applications that leverage skin-slip and grip interaction, contributing further to realistic experiences in VR.

Multimodal Interfaces (MMIs) combining speech and spatial input have the potential to elicit minimal cognitive load. Low cognitive load increases effectiveness as well as user satisfaction and is regarded as an important aspect of intuitive use. While this potential has been extensively theorized in the research community, experiments that provide supporting observations based on functional interfaces are still scarce. In particular, there is a lack of studies comparing the commonly used Unimodal Interfaces (UMIs) with theoretically superior synergistic MMI alternatives. Yet, these studies are an essential prerequisite for generalizing results, developing practice-oriented guidelines, and ultimately exploiting the potential of MMIs in a broader range of applications. This work contributes a novel observation towards the resolution of this shortcoming in the context of the following combination of applied interaction techniques, tasks, application domain, and technology: We present a comprehensive evaluation of a synergistic speech & touch MMI and a touch-only menu-based UMI (interaction techniques) for selection and system control tasks in a digital tabletop game (application domain) on an interactive surface (technology). Cognitive load, user experience, and intuitive use are evaluated, with the former being assessed by means of the dual-task paradigm. Our experiment shows that the implemented MMI causes significantly less cognitive load and is perceived significantly more usable and intuitive than the UMI. Based on our results, we derive recommendations for the interface design of digital tabletop games on interactive surfaces. Further, we argue that our results and design recommendations are suitable to be generalized to other application domains on interactive surfaces for selection and system control tasks.

One can embed a vibration actuator to a physical button and augment the physical button’s original kinesthetic response with a programmable vibration generated by the actuator. Such vibration-augmented buttons inherit the advantages of both physical and virtual buttons. This paper reports the information transmission capacity of vibration-augmented buttons. It was obtained by conducting a series of absolute identification experiments while increasing the number of augmented buttons. The information transmission capacity found was 2.6 bits, and vibration-augmented and physical buttons showed similar abilities in rendering easily recognizable haptic responses. In addition, we showcase a VR text entry application that utilizes vibration-augmented buttons. Our method provides several error messages to the user during text entry using a VR controller that includes an augmented button. We validate that the variable haptic feedback improves task performance, cognitive workload, and user experience for a transcription task.

We describe an automated, multimodal embodied conversational agent that plays the role of a genetic counselor, designed to communicate breast cancer risk and the recommended medical guidelines to women. The counselor's dialogue is driven by intelligent tutoring systems techniques, risk communication principles, and information processing theories. The virtual counselor dynamically tailors its counseling based on user traits, preferred information processing methods, and dynamic comprehension assessments. We conducted a between-subject evaluation study with 30 women, comparing the adaptive counselor to a non-adaptive version of the counselor and a control condition. Women in the adaptive condition demonstrated a significantly greater increase in breast cancer genetics knowledge compared to women in the other conditions. Our results demonstrate the effectiveness of the multidimensional adaptation mechanisms for improving cancer genetic risk communication.

Human-Computer Interaction (HCI) research on menstrual tracking has emphasized the need for more inclusive design of mechanisms for tracking and sharing information on menstruation. We investigate menstrual tracking and data-sharing attitudes and practices in educated, young (20-30 years old) menstruating individuals based in the United States, with self-identified minimal menstrual education backgrounds. Using interviews (N=18), a survey (N=62), and participatory design (N=7), we find that existing mechanisms for tracking and sharing data on menstruation are not adequately responsive to the needs of those who seek relevant menstrual education, are not in the sexual majority, and/or wish to customize what menstrual data they share and with whom. Our analysis highlights a design gap for participants with minimal sexual education backgrounds who wish to better understand their cycles. We also contribute a deepened understanding of structural health inequities that impact menstrual tracking and sharing practices, making recommendations for technology-mediated menstrual care.

Breastfeeding can be challenging, but it is difficult for antenatal education to convey issues associated with the lived experience of breastfeeding. In our work, we explore the potential of interactive simulations to support antenatal education, and present Virtual Feed, a Virtual Reality breastfeeding simulation for parents-to-be developed following a three-step process. (1) We created an experience prototype that features basic VR scenarios and a tangible baby, (2) we engaged in design sessions with 19 parents and parents-to-be to derive design implications to further refine the simulation, and (3) we evaluated the system through case studies to examine the perspectives of parents and parents-to-be on the simulation. Our results show that the simulation successfully engaged users and sparked curiosity, while also encouraging reflection about the challenges of breastfeeding. On this basis, we discuss challenges for the design of simulations with the purpose of supplementing antenatal education.

Recently, chatbots have been deployed in health care in various ways such as providing educational information, and monitoring and triaging symptoms. However, they can be ineffective when they are designed without a careful consideration of the cultural context of the users, especially for marginalized groups. Chatbots designed without cultural understanding may result in loss of trust and disengagement of the user. In this paper, through an interview study, we attempt to understand how chatbots can be better designed for Black American communities within the context of COVID-19. Along with the interviews, we performed design activities with 18 Black Americans that allowed them to envision and design their own chatbot to address their needs and challenges during the pandemic. We report our findings on our participants’ needs for chatbots’ roles and features, and their challenges in using chatbots. We then present design implications for future chatbot design for the Black American population.

Conversational interaction, for example through chatbots, is well-suited to enable automated health coaching tools to support self-management and prevention of chronic diseases. However, chatbots in health are predominantly scripted or rule-based, which can result in a stagnant and repetitive user experience in contrast with more dynamic, data-driven chatbots in other domains. Consequently, little is known about the tradeoffs of pursuing data-driven approaches for health chatbots. We examined multiple artificial intelligence (AI) approaches to enable micro-coaching dialogs in nutrition — brief coaching conversations related to specific meals, to support achievement of nutrition goals — and compared, reinforcement learning (RL), rule-based, and scripted approaches for dialog management. While the data-driven RL chatbot succeeded in shorter, more efficient dialogs, surprisingly the simplest, scripted chatbot was rated as higher quality, despite not fulfilling its task as consistently. These results highlight tensions between scripted and more complex, data-driven approaches for chatbots in health.

Depression and anxiety among college students have been on the rise globally. Cognitive Behavioural Therapy has emerged as an empirically reinforced and effective treatment. However, factors like cost, lack of resources, misguided prioritization and stigmatization of mental health issues in the Global South limit students’ access to psychotherapy. While technology can bridge this gap, research shows current self-guided mHealth apps for CBT are not always evidence-based and have limited efficacy compared to therapist-guided alternatives. In this paper, we explore whether interactive storytelling and other gamification mechanisms can increase the efficacy of a self-guided mHealth app, while drawing from empirically supported CBT protocols. We designed an mHealth application with contextualised storylines to help students learn psychological concepts and better identify the negative patterns in their thoughts. We present the results of a 3-arm randomized controlled trial conducted to assess the effect of this application compared to active and inactive control conditions.

Smartphone overuse is related to a variety of issues such as lack of sleep and anxiety. We explore the application of Self-Affirmation Theory on smartphone overuse intervention in a just-in-time manner. We present TypeOut, a just-in-time intervention technique that integrates two components: an in-situ typing-based unlock process to improve user engagement, and self-affirmation-based typing content to enhance effectiveness. We hypothesize that the integration of typing and self-affirmation content can better reduce smartphone overuse. We conducted a 10-week within-subject field experiment (N=54) and compared TypeOut against two baselines: one only showing the self-affirmation content (a common notification-based intervention), and one only requiring typing non-semantic content (a state-of-the-art method). TypeOut reduces app usage by over 50%, and both app opening frequency and usage duration by over 25%, all significantly outperforming baselines. TypeOut can potentially be used in other domains where an intervention may benefit from integrating self-affirmation exercises with an engaging just-in-time mechanism.

Past research has demonstrated that accounts of trusted others can provide additional context into real world behavior relevant to clinical decision-making and patient engagement. Our research investigates the Social Sensing System, a concept which leverages trusted other feedback for veterans in therapy for PTSD. In our two phase study, we work with 10 clinicians to develop text-message queries and realistic scenarios to present to patients and trusted others. We then present the results in the form of a storyboard to 10 veterans with PTSD and 10 trusted others and gather feedback via semi-structured interview and survey. We find that while trusted other feedback may provide a unique and useful perspective, key design features and considerations of underlying relationships must be considered. We present our findings and utilize the mechanisms and conditions framework to assess the power dynamics of systems such as social sensing in the mental health realm.

Two decades of focus on User Experience has yielded an array of digital technologies that help people experience, understand and share emotions. Although the effects of specific technologies upon emotion have been well studied, less is known about how people actively appropriate and combine the full range of devices, apps and services at their disposal to deliberately manage emotions in everyday life. We conducted a one-week diary study in which 23 adults recorded interactions between their emotions and technology use. They reported using a diverse range of emotion-shaping tools and strategies as part of coping with daily challenges, managing routines, and pursuing work and social goals. We analyse these data in the light of psychological theories of emotion. Our findings point to the significance of digital emotion regulation as a powerful perspective to inform wider debates about the impacts of technology on social and emotional well-being.

Traditionally, game designers have driven the development process of psychotherapeutic games with psychotherapists playing a consultant role. In contrast, our study explores psychotherapists independently designing digitally gamified psychotherapy. Our workshop participants consisted of six psychotherapists who created gamified digital therapies with the design prompt of anxiety disorder. We analyze the resulting six prototypes from a game design and psychotherapy perspective. We present insights into strategies for digitally gamifying therapy and identify challenges and opportunities for the future of gamified psychotherapy, grounded in the experiences of psychotherapists designing such therapies. This study also reflects on the use of user-friendly development tools for independently curating gamified digital therapies, especially by non-technical users such as psychotherapists.

Smart homes continue to raise concerns about privacy and encroachment of businesses into intimate spaces. Prior research has focused on families and device owners in western contexts (Europe and North America), and has identified the importance of bystanders: individuals who are subjected to smart device use of others. Given the cultural and contextual aspects of accommodating bystanders, we identify a gap where bystanders in non-western societies have been insufficiently researched. To address this we conduct 20 interviews with domestic workers and household employers in Jordan, exploring privacy attitudes and practices. Our analysis uncovers a complex interplay between religious and social norms; legal and regulatory perspectives on privacy; and tensions between households and their domestic workers. We explore issues arising from smart homes coexisting as a residence and workplace, and highlight how workplace protections are ill-suited. We structure our findings to help inform public awareness, policy makers, manufacturers, and future research.

The Amazon Alexa voice assistant provides convenience through automation and control of smart home appliances using voice commands. Amazon allows third-party applications known as skills to run on top of Alexa to further extend Alexa’s capability. However, as multiple skills can share the same invocation phrase and request access to sensitive user data, growing security and privacy concerns surround third-party skills. In this paper, we study the availability and effectiveness of existing security indicators or a lack thereof to help users properly comprehend the risk of interacting with different types of skills. We conduct an interactive user study (inviting active users of Amazon Alexa) where participants listen to and interact with real-world skills using the official Alexa app. We find that most participants fail to identify the skill developer correctly (i.e., they assume Amazon also develops the third-party skills) and cannot correctly determine which skills will be automatically activated through the voice interface. We also propose and evaluate a few voice-based skill type indicators, showcasing how users would benefit from such voice-based indicators.

Fitness tracking applications allow athletes to record and share their exercises online, including GPS routes of their activities. However, sharing mobility data potentially raises real-world privacy and safety risks. One strategy to mitigate that risk is a “Privacy Zone,” which conceals portions of the exercise routes that fall within a certain radius of a user-designated sensitive location. A pressing concern is whether privacy zones are an effective deterrent against common attackers, such as a bike thief that carefully scrutinizes online exercise activities in search of their next target. Further, little is known about user perceptions of privacy zones or how they fit into the broader landscape of available privacy precautions.

This work presents an online user study (N=603) that investigates the privacy concerns of fitness tracking users and evaluates the efficacy of privacy zones. Participants were first asked about their privacy behaviors with respect to fitness tracking applications. Next, participants completed an interactive task in which they attempted to deduce hidden locations protected by a privacy zone; we manipulated the number of displayed exercise activities that interacted with the privacy zone, as well as its size. Finally, participants were asked further questions about their impressions of privacy zones and use of other privacy precautions. We found that participants successfully inferred protected locations; for the most common privacy zone size, 68% of guesses fell within 50 meters of the hidden location when participants were shown just 3 activities. Further, we found that participants who viewed 3 activities were more confident about their success in the task compared to participants who viewed 1 activity. Combined, these results indicate that users’ privacy-sensitive locations are at risk even when using a privacy zone. We conclude by considering the implications of our findings on related privacy features and discuss recommendations to fitness tracking users and services to improve the privacy and safety of fitness trackers.

In this paper, we studied people’s smart home privacy-protective behaviors (SH-PPBs), to gain a better understanding of their privacy management do’s and don’ts in this context. We first surveyed 159 participants and elicited 33 unique SH-PPB practices, revealing that users heavily rely on ad hoc approaches at the physical layer (e.g., physical blocking, manual powering off). We also characterized the types of privacy concerns users wanted to address through SH-PPBs, the reasons preventing users from doing SH-PPBs, and privacy features they wished they had to support SH-PPBs. We then storyboarded 11 privacy protection concepts to explore opportunities to better support users’ needs, and asked another 227 participants to criticize and rank these design concepts. Among the 11 concepts, Privacy Diagnostics, which is similar to security diagnostics in anti-virus software, was far preferred over the rest. We also witnessed rich evidence of four important factors in designing SH-PPB tools, as users prefer (1) simple, (2) proactive, (3) preventative solutions that can (4) offer more control.

Traditional approaches to technology design have historically ignored Blackness in both who engages and conceptualizes future technologies. Design contributions of groups marginalized along race and class are often othered, and rarely considered the design standard. While frameworks have emerged to encourage attention to gender and social justice in design, little work has acknowledged evidence of the Black imaginary in this process. The current canon of design defines futuring and speculation as stemming from a narrow view of science fiction, one which does not include Black futurist perspectives. In this essay, we expand the canon of design by arguing that frameworks such as Afrofuturism, Afrofuturist feminism, and Black feminism be considered instrumental in design’s imagining of our future technological landscape. We contribute to the larger conversation of who gets to future in design, suggesting a dialogic relationship between those who conceptualize design and those who consider design’s societal impact.

Rural infrastructure is known to be more prone to breakdown than urban infrastructure. This paper explores how the fragility of rural infrastructure is reproduced through the process of engineering design. Building on values in design, we examine how eventual use is anticipated by engineering researchers building on emerging infrastructure for digital agriculture (DA). Our approach combines critically reflective technical systems-building with interviews with other practitioners to understand and address moments early in the design process where the eventual effects of DA systems may be being built-in. Our findings contrast researchers’ visions of seamless farming technologies with the seamful realities of their work to produce them. We trace how, when anticipating future use, the seams that researchers themselves experience disappear, other seams are hidden from view by institutional support, and seams end users may face are too distant to be in sight. We develop suggestions for the design of these technologies grounded in a more artful management of seamfulness and seamlessness during the process of design and development.

When considering the democratic intentions of co-design, designers and design researchers must evaluate the impact of power imbalances embedded in common design and research dynamics. This holds particularly true in work with and for marginalized communities, who are frequently excluded in design processes. To address this issue, we examine how existing design tools and methods are used to support communities in processes of community building or reimagining, considering the influence of race and identity. This paper describes our findings from 27 interviews with community design practitioners conducted to evaluate the Building Utopia toolkit, which employs an Afrofuturist lens for speculative design processes. Our research findings support the importance of design tools that prompt conversations on race in design, and tensions between the desire for imaginative design practice and the immediacy of social issues, particularly when designing with Black and brown communities.

Reducing uncertainty around the nature of racist interactions is one of the key motivations driving individual behaviors for coping with those incidents. However, there are few appropriate technologies to support BIPOC (Black, Indigenous, People of Color) in engaging in social uncertainty reduction around this vulnerable, sensitive topic. This paper reports on an exploratory design study investigating how social technology might facilitate uncertainty reduction through three “provotypes” - provocative prototypes of user-generated speculative design concepts. U.S.-based participants engaged with the provotypes through an interactive fiction to explore their usefulness in the context of a racist microaggression. Results showed that engaging the provotypes through interactive fiction facilitated complex and productive interactions and critiques. This work contributes a novel method for conducting exploratory design, remote user studies using interactive fiction as well as priorities, tensions, and further information what role, if any, technology might play in managing racist interactions.

In this work, we use composted food waste to create ReClaym: a personal biomaterial, made at home, that reflects the makers’ relationship with food. We combine our personal compost with non-toxic binders to create a biomaterial that can completely biodegrade. We propose Intimate Making as an approach for working with ReClaym that leverages familiar, hands-on techniques to enhance maker-material communication, which in turn leads to a deeper material understanding. We explore methods to customize ReClaym via color, texture, sensing, and conductivity. We apply manual fabrication techniques (sculpting, molding, and hand-held extruding) and design explorations to create a collection of applications that include garden paraphernalia, games, and personal items found in the home. We then examine the entire life cycle of ReClaym, which both begins and ends with composting—giving food waste a second life and providing a sustainable end of life for ReClaym artifacts, thus making the process truly circular.

Museums are interested in designing emotional visitor experiences to complement traditional interpretations. HCI is interested in the relationship between Affective Computing and Affective Interaction. We describe Sensitive Pictures, an emotional visitor experience co-created with the Munch art museum. Visitors choose emotions, locate associated paintings in the museum, experience an emotional story while viewing them, and self-report their response. A subsequent interview with a portrayal of the artist employs computer vision to estimate emotional responses from facial expressions. Visitors are given a souvenir postcard visualizing their emotional data. A study of 132 members of the public (39 interviewed) illuminates key themes: designing emotional provocations; capturing emotional responses; engaging visitors with their data; a tendency for them to align their views with the system's interpretation; and integrating these elements into emotional trajectories. We consider how Affective Computing can hold up a mirror to our emotions during Affective Interaction

Our eating practices are increasingly overshadowed by the presence of screen-based media technologies that conflict with the ideologies of mindful eating. However, little is known about whether and how screens influence our eating behaviors. To contribute to this understanding, we present a rich account of dining practices of ten participants with and without screen. Our study revealed that eating with screens was found more enjoyable than eating alone. Screens can influence one's awareness of hunger and other behaviors like chewing rate and food gaze, whereas screen-media did not trigger any judgements for food. Drawing on the study insights, we highlight the role of technology to support bodily awareness, savoring, a non-judgmental attitude to eating and on rethinking distractions as companions. The outlined considerations encourage a creative yet careful take on making mindful eating more accessible within the realities of screen-based dining cultures.

As parts of our planet continue to experience extreme heat waves, it is more urgent than ever for human-food interaction research to examine climate-resilient and sustainable food practices. Our work, conducted in the hottest city in the USA, focuses on solar cooking as a set of creative DIY activities that use extreme heat and mitigate human impact on the environment. We report on a summer-long study whereby 7 enthusiasts built solar ovens from scratch and experimented with solar recipes ranging from slow-cooked pork and chicken to bread, kale chips, brownies, jerky, and fruit rollups. Our findings depict solar cooking as a form of iterative DIY, which, through its challenges and creative workarounds, serves as a point of engagement with both food and extreme heat. We reflect on solar cooking as a climate-resilient food practice and conclude with design considerations for HCI to support solar cooking as a habitual community practice.

Yo–Yo Machines are playful communication devices designed to help people feel socially connected while physically separated. We designed them to reach as many people as possible, both to make a positive impact during the COVID-19 pandemic and to assess a self-build approach to circulating research products and the appeal of peripheral and expressive communication devices. A portfolio of four distinct designs, based on over 30 years of research, were made available for people to make by following simple online instructions (yoyomachines.io). Each involves connecting a pair of identical devices over the internet to allow simple communication at a distance. This paper describes our motivation for the project, previous work in the area, the design of the devices, supporting website and publicity, and how users have made and used Yo-Yo Machines. Finally, we reflect on what we learned about peripheral and expressive communication devices and implications for the self-build approach.

Television captions blocking visual information causes dissatisfaction among Deaf and Hard of Hearing (DHH) viewers, yet existing caption evaluation metrics do not consider occlusion. To create such a metric, DHH participants in a recent study imagined how bad it would be if captions blocked various on-screen text or visual content. To gather more ecologically valid data for creating an improved metric, we asked 24 DHH participants to give subjective judgments of caption quality after actually watching videos, and a regression analysis revealed which on-screen contents’ occlusion related to users’ judgments. For several video genres, a metric based on our new dataset out-performed the prior state-of-the-art metric for predicting the severity of captions occluding content during videos, which had been based on that prior study. We contribute empirical findings for improving DHH viewers’ experience, guiding the placement of captions to minimize occlusions, and automated evaluation of captioning quality in television broadcasts.

Deaf and Hard-of-Hearing (DHH) users face accessibility challenges during in-person and remote meetings. While emerging use of applications incorporating automatic speech recognition (ASR) is promising, more user-interface and user-experience research is needed. While co-design methods could elucidate designs for such applications, COVID-19 has interrupted in-person research. This study describes a novel methodology for conducting online co-design workshops with 18 DHH and hearing participant pairs to investigate ASR-supported mobile and videoconferencing technologies along two design dimensions: Correcting errors in ASR output and implementing notification systems for influencing speaker behaviors. Our methodological findings include an analysis of communication modalities and strategies participants used, use of an online collaborative whiteboarding tool, and how participants reconciled differences in ideas. Finally, we present guidelines for researchers interested in online DHH co-design methodologies, enabling greater geographically diversity among study participants even beyond the current pandemic.

We present the first large-scale longitudinal analysis of missing label accessibility failures in Android apps. We developed a crawler and collected monthly snapshots of 312 apps over 16 months. We use this unique dataset in empirical examinations of accessibility not possible in prior datasets. Key large-scale findings include missing label failures in 55.6% of unique image-based elements, longitudinal improvement in ImageButton elements but not in more prevalent ImageView elements, that 8.8% of unique screens are unreachable without navigating at least one missing label failure, that app failure rate does not improve with number of downloads, and that effective labeling is neither limited to nor guaranteed by large software organizations. We then examine longitudinal data in individual apps, presenting illustrative examples of accessibility impacts of systematic improvements, incomplete improvements, interface redesigns, and accessibility regressions. We discuss these findings and potential opportunities for tools and practices to improve label-based accessibility.

Blind users rely on alternative text (alt-text) to understand an image; however, alt-text is often missing. AI-generated captions are a more scalable alternative, but they often miss crucial details or are completely incorrect, which users may still falsely trust. In this work, we sought to determine how additional information could help users better judge the correctness of AI-generated captions. We developed ImageExplorer, a touch-based multi-layered image exploration system that allows users to explore the spatial layout and information hierarchies of images, and compared it with popular text-based (Facebook) and touch-based (Seeing AI) image exploration systems in a study with 12 blind participants. We found that exploration was generally successful in encouraging skepticism towards imperfect captions. Moreover, many participants preferred ImageExplorer for its multi-layered and spatial information presentation, and Facebook for its summary and ease of use. Finally, we identify design improvements for effective and explainable image exploration systems for blind users.

Interacting with flying objects has fueled people’s imagination throughout history. Over the past decade, the Human-Drone Interaction (HDI) community has been working towards making this dream a reality. Despite notable findings, we lack a high-level perspective on the current and future use cases for interacting with drones. We present a holistic view of domains and applications of use that are described, studied, and envisioned in the HDI body of work. To map the extent and nature of the prior research, we performed a scoping review (N=217). We identified 16 domains and over 100 applications where drones and people interact together. We then describe in depth the main domains and applications reported in the literature and further present under-explored use cases with great potential. We conclude with fundamental challenges and opportunities for future research in the field. This work contributes a systematic step towards increased replicability and generalizability of HDI research.

Online platforms commonly collect and display user-generated information to support subsequent users’ decision-making. However, studies have noticed that presenting collective information can pose social influences on individuals’ opinions and alter their preferences accordingly. It is essential to deepen understanding of people’s preferences when exposed to others’ opinions and the underlying cognitive mechanisms to address potential biases. Hence, we conducted a laboratory study to investigate how products’ ratings and reviews influence participants’ stated preferences and cognitive responses assessed by their Electroencephalography (EEG) signals. The results showed that social ratings and reviews could alter participants’ preferences and affect their status of attention, working memory, and emotion. We further conducted predictive analyses to show that participants’ Electroencephalography-based measures can achieve higher power than behavioral measures to discriminate how collective information is displayed to users. We discuss the design implications informed by the results to shed light on the design of collective rating systems.

There is a growing interest in extending crowdwork beyond traditional desktop-centric design to include mobile devices (e.g., smartphones). However, mobilizing crowdwork remains significantly tedious due to a lack of understanding about the mobile usability requirements of human intelligence tasks (HITs). We present a taxonomy of characteristics that defines the mobile usability of HITs for smartphone devices. The taxonomy is developed based on findings from a study of three consecutive steps. In Step 1, we establish an initial design of our taxonomy through a targeted literature analysis. In Step 2, we verify and extend the taxonomy through an online survey with Amazon Mechanical Turk crowdworkers. Finally, in Step 3 we demonstrate the taxonomy’s utility by applying it to analyze the mobile usability of a dataset of scraped HITs. In this paper, we present the iterative development of the taxonomy, highlighting the observed practices and preferences around mobile crowdwork. We conclude with the implications of our taxonomy for accessibly and ethically mobilizing crowdwork not only within the context of smartphone devices, but beyond them.

User experience (UX) summarizes user perceptions and responses resulting from the interaction with a product, system, or service. The User Experience Questionnaire (UEQ) is one standardized instrument for measuring UX. With six scales, it identifies areas in which product improvements will have the highest impact. In this paper, we evaluate the reliability and validity of this questionnaire. The data of N = 1, 121 participants who interacted with one of 23 products indicated an acceptable to good reliability of all scales. The results show, however, that the scales were not independent of each other. Combining perspicuity, efficiency, and dependability to pragmatic aspects as well as novelty and stimulation to hedonic aspects of UX improved the model fit significantly. The systematic variations of product properties and correlations with the System Usability Scale (SUS) in a second experiment with N=499 participants supported the validity of these two factors. Practical implications of the results are discussed.

In the last decade, interest in accessible and eyes-free text entry has continued to grow. However, little research has been done to explore the feasibility of using audibly distinct phrases for text entry tasks. To better understand whether preexisting phrases used in text entry research are sufficiently distinct for eyes-free text entry tasks, we used Microsoft’s and Apple’s desktop text-to-speech systems to generate all 500 phrases from MacKenzie and Soukoreff’s set [32] using the default male and female voices. We then asked 392 participants recruited through Amazon’s Mechanical Turk to transcribe the generated audio clips. We report participant transcription errors and present the 96 phrases that were observed with no comprehension errors. These phrases were further tested with 80 participants who identified as low-vision and/or blind recruited through Twitter. We contribute the 92 phrases that were observed to maintain no comprehension errors across both experiments.

Social interactions are an essential part of many digital games, and provide benefits to players; however, problematic social interactions also lead to harm. To inform our understanding of the origins of harmful social behaviours in gaming contexts, we examine how trait psychopathy influences player perceptions and behaviours within a gaming task. After measuring participants’ (n=385) trait-level boldness, meanness, and disinhibition, we expose them to neutral and angry social interactions with a non-player character (NPC) in a gaming task and assess their perceptions, verbal responses, and movement behaviours. Our findings demonstrate that the traits significantly influence interpretation of NPC emotion, verbal responses to the NPC, and movement behaviours around the NPC. These insights can inform the design of social games and communities and can help designers and researchers better understand how social functioning translates into gaming contexts.

In interactive story games, players make decisions that advance and modify the unfolding story. In many cases, these decisions have a moral component. Examining decision-making in these games illuminates whether players mobilize their real-life morality to make in-game decisions and what impact this has in both the game world and real life. Using mixed-methods consisting of semi-structured interviews and the Moral Foundations Questionnaire (MFQ30), we collected data from 19 participants who played the game Detroit: Become Human. We analyzed how participants applied their real-life morals toward in-game decisions using thematic analysis and statistical analysis of the MFQ30 results. Qualitative findings indicate that participants mobilize their moral intuitions to make in-game decisions and how much participants cared about their game characters influenced their choices. We contribute a better understanding of how players react to moral dilemmas in interactive story games for game designers to help them improve player experience.

Research on the emotional experience of playing videogames has increased in recent years, yet much of this work is focused on the hedonistic player experience (PX) commonly associated with the nebulous concept of ‘fun’ and positive affect. Researchers are increasingly paying more attention to the eudaimonic PX commonly associated with ‘appreciation’, mixed-affect and reflection. To further investigate eudaimonic PX we interviewed 24 games players about ‘significant or memorable emotional experiences’ from their games playing and used grounded theory to analyse their responses. This led to the construction of the concept of ‘emotional exploration’ which is used to help explain (i) why players would seek out a eudaimonic PX, (ii) how eudaimonic PX is constituted and (iii) how developers can design for a eudaimonic PX. We further make the case for the ‘eudaimonic gameplay experience’ to be realised as different and separate to pre-existing notions of eudaimonic entertainment.

Do people use games to cope with adverse life events and crises? Research informed by self-determination theory proposes that people might compensate for thwarted basic psychological needs in daily life by seeking out games that satisfy those lacking needs. To test this, we conducted a preregistered mixed-method survey study (n = 285) on people’s gaming behaviours and need states during early stages of the COVID-19 pandemic (May 2020). We found qualitative evidence that gaming was an often actively sought out and successful means of replenishing particular needs, but one that could ‘backfire’ for some through an appraisal process discounting gaming as ‘unreal’. Meanwhile, contrary to our predictions, the quantitative data showed a “rich get richer, poor get poorer” pattern: need satisfaction in daily life positively correlated with need satisfaction in games. We derive methodological considerations and propose three potential explanations for this contradictory data pattern to pursue in future research.

Desktop notifications should be noticeable but are also subject to a number of design choices, e.g. concerning their size, placement, or opacity. It is currently unknown, however, how these choices interact with the desktop background and their influence on noticeability. To address this limitation, we introduce a software tool to automatically synthesize realistically looking desktop images for major operating systems and applications. Using these images, we present a user study (N=34) to investigate the noticeability of notifications during a primary task. We are first to show that visual importance of the background at the notification location significantly impacts whether users detect notifications. We analyse the utility of visual importance to compensate for suboptimal design choices with respect to noticeability, e.g. small notification size. Finally, we introduce noticeability maps - 2D maps encoding the predicted noticeability across the desktop and inform designers how to trade-off notification design and noticeability.

When people engage in urban exploration, the tool they are most likely to use today is a mobile phone. In this paper, we present observations of users’ “home” and “away” conducted to refine our understanding of situational Point-of-Interest (POI) needs. Our findings suggest three distinct categories of situations in which users seek POI information: On-the-spot, Refining plans, and Moments of boredom. Based on the similarities and differences of these three situations in five observed underlying constraints – distance of interest, engagement threshold, ambiguity of the search, profile matching, and other imperative constraints, we derive implications for designing and ranking POIs for a Situational Recommender. To further access our concept, we designed and prototyped Situational Recommender by providing an interactional representation of the situation, and ran a Wizard-of-Oz concept validation study. Our results suggest that participants understood the concept without much effort and appreciated its usefulness.

Large language models are increasingly mediating, modifying, and even generating messages for users, but the receivers of these messages may not be aware of the involvement of AI. To examine this emerging direction of AI-Mediated Communication (AI-MC), we investigate people’s perceptions of AI written messages. We analyze how such perceptions change in accordance with the interpersonal emphasis of a given message. We conducted both large-scale surveys and in-depth interviews to investigate how a diverse set of factors influence people’s perceived trust in AI-mediated writing of emails. We found that people’s trust in email writers decreased when they were told that AI was involved in the writing process. Surprisingly trust increased when AI was used for writing more interpersonal emails (as opposed to more transactional ones). Our study provides insights regarding how people perceive AI-MC and has practical design implications on building AI-based products to aid human interlocutors in communication1.

This paper revisits concepts of nudge in the context of helping consumers to make healthier food choices. We introduce a novel form of social influence nudge not yet investigated by HCI scholars, the out-group social comparison, and test whether this form of nudging works at the point of checkout rather than the more conventional point of product consideration. Across two online experiments, we measure the effectiveness of using nutritional information nudges with added in-group (people like you) and out-group (people not like you) social comparisons. Our preliminary findings suggest that out-group social comparison nudges can be effective in encouraging both normal weight and overweight adults to reduce calories, even when these adults indicate that they do not typically change their diet behaviors. This research has implications for digital information design, interactive marketing, and public health.

Online data visualizations play an important role in informing public opinion but are often inaccessible to screen reader users. To address the need for accessible data representations on the web that provide direct, multimodal, and up-to-date access to the data, we investigate audio data narratives –which combine textual descriptions and sonification (the mapping of data to non-speech sounds). We conduct two co-design workshops with screen reader users to define design principles that guide the structure, content, and duration of a data narrative. Based on these principles and relevant auditory processing characteristics, we propose a dynamic programming approach to automatically generate an audio data narrative from a given dataset. We evaluate our approach with 16 screen reader users. Findings show with audio narratives, users gain significantly more insights from the data. Users describe data narratives help them better extract and comprehend the information in both the sonification and description.

We increasingly rely on up-to-date, data-driven graphs to understand our environments and make informed decisions. However, many of the methods blind and visually impaired users (BVI) rely on to access data-driven information do not convey important shape-characteristics of graphs, are not refreshable, or are prohibitively expensive. To address these limitations, we introduce two refreshable, 1-DOF audio-haptic interfaces based on haptic cues fundamental to object shape perception. Slide-tone uses finger position with sonification, and Tilt-tone uses fingerpad contact inclination with sonification to provide shape feedback to users. Through formative design workshops (n = 3) and controlled evaluations (n = 8), we found that BVI participants appreciated the additional shape information, versatility, and reinforced understanding these interfaces provide; and that task accuracy was comparable to using interactive tactile graphics or sonification alone. Our research offers insight into the benefits, limitations, and considerations for adopting these haptic cues into a data visualization context.

JavaScript visualization libraries are widely used to create online data visualizations but provide limited access to their information for screen-reader users. Building on prior findings about the experiences of screen-reader users with online data visualizations, we present VoxLens, an open-source JavaScript plug-in that—with a single line of code—improves the accessibility of online data visualizations for screen-reader users using a multi-modal approach. Specifically, VoxLens enables screen-reader users to obtain a holistic summary of presented information, play sonified versions of the data, and interact with visualizations in a “drill-down” manner using voice-activated commands. Through task-based experiments with 21 screen-reader users, we show that VoxLens improves the accuracy of information extraction and interaction time by 122% and 36%, respectively, over existing conventional interaction with online data visualizations. Our interviews with screen-reader users suggest that VoxLens is a “game-changer” in making online data visualizations accessible to screen-reader users, saving them time and effort.

Many daily tasks rely on accurately identifying and distinguishing between different colours. However, these tasks can be frustrating and potentially dangerous for people with Colour Vision Deficiency (CVD). Despite prior work exploring how pattern overlays on top of colours can support people with CVD, the solutions were often unintuitive or required significant training to become proficient. We address this problem by creating two new colour patterns (ColourIconizer, ColourMix). We evaluated these patterns against a previously published colour pattern (ColourMeters) using an online evaluation with three new colour identification tasks (Selection Task, Transition Task, Sorting Task). ColourMeters helped with the Transition Task, but struggled with the Selection and Sorting Tasks. Conversely, ColourIconizer helped with the Selection and Sorting Tasks but struggled to help on the Transition Task. ColourMix provided general assistance on all tasks. Our combined results help inform and improve the design of future colour patterns.

Data visualisations are increasingly used online to engage readers and enable independent analysis of the data underlying news stories. However, access to such infographics is problematic for readers who are blind or have low vision (BLV). Equitable access to information is a basic human right and essential for independence and inclusion. We introduce infosonics, the audio equivalent of infographics, as a new style of interactive sonification that uses a spoken introduction and annotation, non-speech audio and sound design elements to present data in an understandable and engaging way. A controlled user evaluation with 18 BLV adults found a COVID-19 infosonic enabled a clearer mental image than a traditional sonification. Further, infosonics prove complementary to text descriptions and facilitate independent understanding of the data. Based on our findings, we provide preliminary suggestions for infosonics design, which we hope will enable BLV people to gain equitable access to online news and information.

In conventional software development, user experience (UX) designers and engineers collaborate through separation of concerns (SoC): designers create human interface specifications, and engineers build to those specifications. However, we argue that Human-AI systems thwart SoC because human needs must shape the design of the AI interface, the underlying AI sub-components, and training data. How do designers and engineers currently collaborate on AI and UX design? To find out, we interviewed 21 industry professionals (UX researchers, AI engineers, data scientists, and managers) across 14 organizations about their collaborative work practices and associated challenges. We find that hidden information encapsulated by SoC challenges collaboration across design and engineering concerns. Practitioners describe inventing ad-hoc representations exposing low-level design and implementation details (which we characterize as leaky abstractions) to “puncture” SoC and share information across expertise boundaries. We identify how leaky abstractions are employed to collaborate at the AI-UX boundary and formalize a process of creating and using leaky abstractions.

We present a performance-led inquiry that involved a live coder programming movement-based interactive sound and two dance improvisers. During two years of collaboration, we developed a joint improvisation practice where the interactions between the dancers’ movement and the sound feedback are programmed on the fly through live coding and movement sensing. To that end, we designed a new live coding environment called CO/DA that facilitates the real-time manipulation of continuous streams of the dancers’ motion data for interactive sound synthesis. Through an autoethnographic inquiry, we describe our practice of sound and movement improvisation where live coding dynamically changes how the dancers’ movements generate sound, which in turn influences the dancers’ improvisation. We then discuss the value, potential and challenges of our dance/code improvisation practice, along with its implications as a design method.

HCI research has explored AI as a design material, suggesting that designers can envision AI’s design opportunities to improve UX. Recent research claimed that enterprise applications offer an opportunity for AI innovation at the user experience level. We conducted design workshops to explore the practices of experienced designers who work on cross-functional AI teams in the enterprise. We discussed how designers successfully work with and struggle with AI. Our findings revealed that designers can innovate at the system and service levels. We also discovered that making a case for an AI feature’s return on investment is a barrier for designers when they propose AI concepts and ideas. Our discussions produced novel insights on designers’ role on AI teams, and the boundary objects they used for collaborating with data scientists. We discuss the implications of these findings as opportunities for future research aiming to empower designers in working with data and AI.

Participatory design facilitators have a significant impact on participatory activities, processes, and outcomes. However, the facilitator role has not yet been thoroughly debated in existing design discourse, and support for role-related reflections is limited. As the first steps towards an enriched collective understanding of this specific role and its realization, we interviewed 14 respondents with an academic background in participatory design and extensive facilitation experience. Based on a content analysis of the interviews, we identified six facets of the role: (1) trust builder, (2) enabler, (3) inquirer, (4) direction setter, (5) value provider, and (6) users’ advocate. Each facet is presented as consisting of the respondents’ perceived associated responsibilities and corresponding strategies. Our results paint a complex picture of participatory design facilitation. We propose the multi-faceted understanding of the facilitator role emerging from this work as a basis for problematized reflection on the role and its realization.

Smart home cameras present new challenges for understanding behaviors and relationships surrounding always-on, domestic recording systems. We designed a series of discursive activities involving 16 individuals from ten households for six weeks in their everyday settings. These activities functioned as speculative probes—prompting participants to reflect on themes of privacy and power through filming with cameras in their households. Our research design foregrounded critical-playful enactments that allowed participants to speculate potentials for relationships with cameras in the home beyond everyday use. We present four key dynamics with participants and home cameras by examining their relationships to: the camera's eye, filming, their data, and camera's societal contexts. We contribute discussions about the mundane, information privacy, and post-hoc reflection with one's camera footage. Overall, our findings reveal the camera as a strange, yet banal entity in the home—interrogating how participants compose and handle their own and others’ video data.

Three of the most common approaches used in recommender systems are content-based filtering (matching users’ preferences with products’ characteristics), collaborative filtering (matching users with similar preferences), and demographic filtering (catering to users based on demographic characteristics). Do users’ intuitions lead them to trust one of these approaches over others, independent of the actual operations of these different systems? Does their faith in one type or another depend on the quality of the recommendation, rather than how the recommendation appears to have been derived? We conducted an empirical study with a prototype of a movie recommender system to find out. A 3 (Ostensible Recommender Type: Content vs. Collaborative vs. Demographic Filtering) x 2 (Recommendation Quality: Good vs. Bad) experiment (N=226) investigated how users evaluate systems and attribute responsibility for the recommendations they receive. We found that users trust systems that use collaborative filtering more, regardless of the system's performance. They think that they themselves are responsible for good recommendations but that the system is responsible for bad recommendations (reflecting a self-serving bias). Theoretical insights, design implications and practical solutions for the cold start problem are discussed.

Model-agnostic explainable AI tools explain their predictions by means of ’local’ feature contributions. We empirically investigate two potential improvements over current approaches. The first one is to always present feature contributions in terms of the contribution to the outcome that is perceived as positive by the user (“positive framing”). The second one is to add “semantic labeling”, that explains the directionality of each feature contribution (“this feature leads to +5% eligibility”), reducing additional cognitive processing steps. In a user study, participants evaluated the understandability of explanations for different framing and labeling conditions for loan applications and music recommendations. We found that positive framing improves understandability even when the prediction is negative. Additionally, adding semantic labels eliminates any framing effects on understandability, with positive labels outperforming negative labels. We implemented our suggestions in a package ArgueView[11].

The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking. Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session, this paper compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience. It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience. On that basis, it presents guidance-design recommendations for future task-oriented chatbots.

Conversational recommender systems (CRSs) imitate human advisors to assist users in finding items through conversations and have recently gained increasing attention in domains such as media and e-commerce. Like in human communication, building trust in human-agent communication is essential given its significant influence on user behavior. However, inspiring user trust in CRSs with a “one-size-fits-all” design is difficult, as individual users may have their own expectations for conversational interactions (e.g., who, user or system, takes the initiative), which are potentially related to their personal characteristics. In this study, we investigated the impacts of three personal characteristics, namely personality traits, trust propensity, and domain knowledge, on user trust in two types of text-based CRSs, i.e., user-initiative and mixed-initiative. Our between-subjects user study (N=148) revealed that users’ trust propensity and domain knowledge positively influenced their trust in CRSs, and that users with high conscientiousness tended to trust the mixed-initiative system.

Proper statistical modeling incorporates domain theory about how concepts relate and details of how data were measured. However, data analysts currently lack tool support for recording and reasoning about domain assumptions, data collection, and modeling choices in an integrated manner, leading to mistakes that can compromise scientific validity. For instance, generalized linear mixed-effects models (GLMMs) help answer complex research questions, but omitting random effects impairs the generalizability of results. To address this need, we present Tisane, a mixed-initiative system for authoring generalized linear models with and without mixed-effects. Tisane introduces a study design specification language for expressing and asking questions about relationships between variables. Tisane contributes an interactive compilation process that represents relationships in a graph, infers candidate statistical models, and asks follow-up questions to disambiguate user queries to construct a valid model. In case studies with three researchers, we find that Tisane helps them focus on their goals and assumptions while avoiding past mistakes.

With the increasing growth and impact of machine learning and other math-intensive fields, it is more important than ever to broaden access to mathematical notation. Can new visual and interactive displays help a wider readership successfully engage with notation? This paper provides the first detailed qualitative analysis of math augmentation—the practice of embellishing notation with novel visual design patterns to improve its readability. We present two qualitative studies of the practice of math augmentation. First is an analysis of 1.1k augmentations to 281 formulas in 47 blogs, textbooks, and other documents containing mathematical expressions. Second is an interview study with 12 authors who had previously designed custom math augmentations (“maugs”). This paper contributes a comprehensive inventory of the kinds of maugs that appear in math documents, and a detailed account of how authors’ tools ought to be redesigned to support efficient creation of math augmentations. These studies open a critical new design space for HCI researchers and interface designers.

Most modern applications are immutable and turn-key despite the acknowledged benefits of empowering users to modify their software. Writing extensible software remains challenging, even for expert programmers. Reprogramming or extending existing software is often laborious or wholly blocked, requiring sophisticated knowledge of application architecture or setting up a development environment. We present Varv, a programming model representing reprogrammable interactive software as a declarative data structure. Varv defines interactive applications as a set of concepts that consist of a schema and actions. Applications in Varv support incremental modification, allowing users to reprogram through addition and selectively suppress, modify, or add behavior. Users can define high-level concepts, creating an abstraction layer and effectively a domain-specific language for their application domain, emphasizing reuse and modification. We demonstrate the reprogramming and collaboration capabilities of Varv in two case studies and illustrate how the event engine allows for extensive tooling support.

Emoticons are indispensable in online communications. With users’ growing needs for more customized and expressive emoticons, recent messaging applications begin to support (limited) multi-modal emoticons:, enhancing emoticons with animations or vibrotactile feedback. However, little empirical knowledge has been accumulated concerning how people create, share and experience multi-modal emoticons in everyday communication, and how to better support them through design. To tackle this, we developed VibEmoji, a user-authoring multi-modal emoticon interface for mobile messaging. Extending existing designs, VibEmoji grants users greater flexibility to combine various emoticons, vibrations, and animations on-the-fly, and offers non-aggressive recommendations based on these components’ emotional relevance. Using VibEmoji as a probe, we conducted a four-week field study with 20 participants, to gain new understandings from in-the-wild usage and experience, and extract implications for design. We thereby contribute to both a novel system and various insights for supporting users’ creation and communication of multi-modal emoticons.

Suggesting multiple target candidates based on touch input is a possible option for high-accuracy target selection on small touchscreen devices. But it can become overwhelming if suggestions are triggered too often. To address this, we propose SATS, a Suggestion-based Accurate Target Selection method, where target selection is formulated as a sequential decision problem. The objective is to maximize the utility: the negative time cost for the entire target selection procedure. The SATS decision process is dictated by a policy generated using reinforcement learning. It automatically decides when to provide suggestions and when to directly select the target. Our user studies show that SATS reduced error rate and selection time over Shift [51], a magnification-based method, and MUCS, a suggestion-based alternative that optimizes the utility for the current selection. SATS also significantly reduced error rate over BayesianCommand [58], which directly selects targets based on posteriors, with only a minor increase in selection time.

Some individuals with motor impairments communicate using a single switch — such as a button click, air puff, or blink. Row-column scanning provides a method for choosing items arranged in a grid using a single switch. An alternative, Nomon, allows potential selections to be arranged arbitrarily rather than requiring a grid (as desired for gaming, drawing, etc.) — and provides an alternative probabilistic selection method. While past results suggest that Nomon may be faster and easier to use than row-column scanning, no work has yet quantified performance of the two methods over longer time periods or in tasks beyond writing. In this paper, we also develop and validate a webcam-based switch that allows a user without a motor impairment to approximate the response times of a motor-impaired single switch user; although the approximation is not a replacement for testing with single-switch users, it allows us to better initialize, calibrate, and evaluate our method. Over 10 sessions with the webcam switch, we found users typed faster and more easily with Nomon than with row-column scanning. The benefits of Nomon were even more pronounced in a picture-selection task. Evaluation and feedback from a motor-impaired switch user further supports the promise of Nomon.

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the pre-existing gesture set. We further evaluate the usability of our real-time implementation with a user experience study (N=20). Our results highlight the effectiveness, learnability, and usability of our customization framework. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.

Despite the advent of touchscreens, typing on physical keyboards remains most efficient for entering text, because users can leverage all fingers across a full-size keyboard for convenient typing. As users increasingly type on the go, text input on mobile and wearable devices has had to compromise on full-size typing. In this paper, we present TapType, a mobile text entry system for full-size typing on passive surfaces—without an actual keyboard. From the inertial sensors inside a band on either wrist, TapType decodes and relates surface taps to a traditional QWERTY keyboard layout. The key novelty of our method is to predict the most likely character sequences by fusing the finger probabilities from our Bayesian neural network classifier with the characters’ prior probabilities from an n-gram language model. In our online evaluation, participants on average typed 19 words per minute with a character error rate of 0.6% after 30 minutes of training. Expert typists thereby consistently achieved more than 25 WPM at a similar error rate. We demonstrate applications of TapType in mobile use around smartphones and tablets, as a complement to interaction in situated Mixed Reality outside visual control, and as an eyes-free mobile text input method using an audio feedback-only interface.

Gesture elicitation studies are commonly used for designing novel gesture-based interfaces. There is a rich methodology literature on metrics and analysis methods that helps researchers understand and characterize data arising from such studies. However, deriving concrete gesture vocabularies from this data, which is often the ultimate goal, remains largely based on heuristics and ad hoc methods. In this paper, we treat the problem of deriving a gesture vocabulary from gesture elicitation data as a computational optimization problem. We show how to formalize it as an optimal assignment problem and discuss how to express objective functions and custom design constraints through integer programs. In addition, we introduce a set of tools for assessing the uncertainty of optimization outcomes due to random sampling, and for supporting researchers’ decisions on when to stop collecting data from a gesture elicitation study. We evaluate our methods on a large number of simulated studies.

Many videogame players livestream their gameplay so remote spectators can watch for enjoyment, fandom, and to learn strategies and techniques. Current approaches capture the player’s rendered RGB view of the game, and then encode and stream it as a 2D live video feed. We extend this basic concept by also capturing the depth buffer, camera pose, and projection matrix from the rendering pipeline of the videogame and package them all within a MPEG-4 media container. Combining these additional data streams with the RGB view, our system builds a real-time, cumulative 3D representation of the live game environment for spectators. This enables each spectator to individually control a personal game view in 3D. This means they can watch the game from multiple perspectives, enabling a new kind of videogame spectatorship experience.

Live streams usually last several hours with many viewers joining in the middle. Viewers who join in the middle often want to understand what has happened in the stream. However, catching up with the earlier parts is challenging because it is difficult to know which parts are important in the long, unedited stream while also keeping up with the ongoing stream. We present CatchLive, a system that provides a real-time summary of ongoing live streams by utilizing both the stream content and user interaction data. CatchLive provides viewers with an overview of the stream along with summaries of highlight moments with multiple levels of detail in a readable format. Results from deployments of three streams with 67 viewers show that CatchLive helps viewers grasp the overview of the stream, identify important moments, and stay engaged. Our findings provide insights into designing summarizations of live streams reflecting their characteristics.

Mobile video-based learning attracts many learners with its mobility and ease of access. However, most lectures are designed for desktops. Our formative study reveals mobile learners’ two major needs: more readable content and customizable video design. To support mobile-optimized learning, we present FitVid, a system that provides responsive and customizable video content. Our system consists of (1) an adaptation pipeline that reverse-engineers pixels to retrieve design elements (e.g., text, images) from videos, leveraging deep learning with a custom dataset, which powers (2) a UI that enables resizing, repositioning, and toggling in-video elements. The content adaptation improves the guideline compliance rate by 24% and 8% for word count and font size. The content evaluation study (n=198) shows that the adaptation significantly increases readability and user satisfaction. The user study (n=31) indicates that FitVid significantly improves learning experience, interactivity, and concentration. We discuss design implications for responsive and customizable video adaptation.

Explainer motion graphics videos that use a combination of graphical elements and movement to convey a visual message are becoming increasingly popular among amateur creators in different domains. But, to author motion graphics videos, amateurs either have to face a steep learning curve with professional design tools or struggle with re-purposing slide-sharing tools that are easier to access but have limited animation capabilities. To simplify the process of motion graphics authoring, we present the design and implementation of Katika, an end-to-end system for creating shots based on a script, adding artworks and animation from a crowdsourced library, and editing the video using semi-automated transitions. Our observational study illustrates that participants (N=11) enjoyed using Katika and, within a one-hour session, managed to create an explainer motion graphics video. We identify opportunities for future HCI research to lower the barriers to entry and democratize the authoring of motion graphics videos.

Historians use spatio-temporal navigation for their models and studies of historical evolutions and events. Their findings can then be exhibited in cultural mediation centers or museums. The latter, both to facilitate the transmission of knowledge and to make their exhibitions more attractive, are now exploiting new technologies. Indeed, digital systems allow, among other things, visitors to navigate spatially and temporally in virtual reconstructions of historical environments. We propose to combine these virtual representations with a tangible interface to provide visitors with an immersive experience and engaging interactions. To do so, we have set up a co-design process involving cultural mediation actors (museum directors, historians, etc.). The result is SABLIER, a tangible interactor to navigate through space and time based on the interaction metaphors and natural affordance of an hourglass. Finally, we have conducted an evaluation of the acceptability of our interactor, whose results are positive.

Consumer electronics are increasingly using everyday materials to blend into home environments, often using LEDs or symbol displays under textile meshes. Our surveys (n=1499 and n=1501) show interest in interactive graphical displays for hidden interfaces — however, covering such displays significantly limits brightness, material possibilities and legibility.

To overcome these limitations, we leverage parallel rendering to enable ultrabright graphics that can pass through everyday materials. We unlock expressive hidden interfaces using rectilinear graphics on low-cost, mass-produced passive-matrix OLED displays. A technical evaluation across materials, shapes and display techniques, suggests 3.6–40X brightness increase compared to more complex active-matrix OLEDs.

We present interactive prototypes that blend into wood, textile, plastic and mirrored surfaces. Survey feedback (n=1572) on our prototypes suggests that smart mirrors are particularly desirable. A lab evaluation (n=11) reinforced these findings and allowed us to also characterize performance from hands-on interaction with different content, materials and under varying lighting conditions.

Head-mounted augmented reality (AR) displays allow for the seamless integration of virtual visualisation with contextual tangible references, such as physical (tangible) globes. We explore the design of immersive geospatial data visualisation with AR and tangible globes. We investigate the “tangible-virtual interplay” of tangible globes with virtual data visualisation, and propose a conceptual approach for designing immersive geospatial globes. We demonstrate a set of use cases, such as augmenting a tangible globe with virtual overlays, using a physical globe as a tangible input device for interacting with virtual globes and maps, and linking an augmented globe to an abstract data visualisation. We gathered qualitative feedback from experts about our use case visualisations, and compiled a summary of key takeaways as well as ideas for envisioned future improvements. The proposed design space, example visualisations and lessons learned aim to guide the design of tangible globes for data visualisation in AR.

(Dis)Appearables is an approach for actuated Tangible User Interfaces (TUIs) to appear and disappear. This technique is supported by Stages: physical platforms inspired by theatrical stages. Self-propelled TUI’s autonomously move between front and back stage allowing them to dynamically appear and disappear from users’ attention. This platform opens up a novel interaction design space for expressive displays with dynamic physical affordances.

We demonstrate and explore this approach based on a proof-of-concept implementation using two-wheeled robots, and multiple stage design examples. We have implemented a stage design pipeline which allows users to plan and design stages that are composed with front and back stages, and transition portals such as trap doors or lifts. The pipeline includes control of the robots, which guides them on and off stage. With this proof-of-concept prototype, we demonstrated a range of applications including interactive mobility simulation, self re-configuring desktops, remote hockey, and storytelling/gaming. Inspired by theatrical stage designs, this is a new take on ‘controlling the existence of matter’ for user experience design.

Autonomous actuated-interfaces provide a unique research opportunity for shared-control interfaces, as the human and the interface collaborate using the physical interaction modality, manipulating the same physical elements at the same time. Prior studies show that sharing control with physical modality interfaces often results in frustration and low sense-of-control. We designed and implemented adaptive behavior for shared-control actuated-interfaces that extends prior work by providing humans the ability to anticipate the autonomous action, and then accept or override it. Results from a controlled study with 24 participants indicate better collaboration in the Adaptive condition compared with the Non-adaptive one, with improved sense-of-control, feelings of teamwork, and overall collaboration quality. Our work contributes to shared-control tangible, shape-change, and actuated interfaces. We show that leveraging minimal non-verbal social cues to physically communicate the actuated-interface’s intent, coupled with providing autonomy to the human to physically accept or override the shift-in-control, improves the shared-control collaboration.

Independent movie theaters (IMTs) are a part of the cultural infrastructure that offers shared spaces for patron communities to access, share, and engage with cultural artifacts. In light of the COVID-19 pandemic, IMTs were mandated to shut down, resulting in unanticipated infrastructural breakdown. Drawing insights from a preliminary survey and interviews with staff members from 18 IMTs in the U.S., this paper attends to how this breakdown disrupted art and community engagement within patron communities. We investigate the sociotechnical practices of maintaining cultural infrastructure through 1) collaborating with community partners and external stakeholders, 2) screening films through online virtual cinema platforms, and 3) retaining community members through online platforms. Our work highlights the tensions and invisible human labor in this maintenance work. Together, this work intends to foreground cultural infrastructure and discuss how HCI can support and contribute to the design and oft-invisible maintenance of cultural infrastructure.

The prevalence of social media blurs the boundaries between consumer and producer, work and play, and leads to new social roles, professions, and identities (e.g. blogger, YouTuber, micro-celebrity). However, we still lack a clear understanding of how people come to identify with these new roles and how individual professional development is digitally mediated. This paper presents a study based on Bilibili, a popular Chinese social media platform featuring user-generated videos, and highlights a professionalization process through which individuals consciously distinguish between the roles of uploaders and consumers, develop a shared work ethos around the role of the uploader, and, as uploaders, improve their technical-professional expertise. We conclude by discussing individualized professionalization as a concept that describes the bottom-up and community-based process of professional development for User Generated Content (UGC) taking place in contemporary digital media environments.

As more individuals consider permanently working from home, the online labor market continues to grow as an alternative working environment. While the flexibility and autonomy of these online gigs attracts many workers, success depends critically upon self-management and workers’ efficient allocation of scarce resources. To achieve this, freelancers may develop alternative work strategies, employing highly standardized schedules and communication patterns while taking on large work volumes, or engaging in smaller numbers of jobs whilst tailoring their activities to build relationships with individual employers. In this study, we consider this contrast in relation to worker communication patterns. We demonstrate the heterogeneous effects of standardization versus personalization across different stages of a project and examine the relative impact on job acquisition, project completion, and earnings. Our findings can inform the design of platforms and various worker support tools for the gig economy.

People often form small conversation groups during physical gatherings to have ad-hoc and informal conversations. As these groups are loosely defined, others can often overhear and join the conversation. However, current video-conferencing tools only allow for strict boundaries between small conversation groups, inhibiting fluid group formations and between-group conversations. This isolates small-group conversations from others and leads to inefficient transitions between conversations. We present FluidMeet, a virtual breakout meeting system that employs flexible conversation boundaries and cross-group conversation visualizations to enable fluid conversation group formations and ad-hoc, informal conversations. FluidMeet enables out-group members to overhear group conversations while allowing conversation groups to control their shared level of context. Users within conversation groups can also quickly switch between in-group and private conversations. A study of FluidMeet showed that it encouraged users to break group boundaries, made them feel less isolated in group conversations, and facilitated communication across different groups.

Any active entity that shares space with people is interpreted as a social actor. Based on this notion, we explore how robots that integrate functional utility with a social role and character can integrate meaningfully into daily practice. Informed by interviews and observations, we designed a zoomorphic floor cleaning robot which playfully interacts with care home residents affected by dementia. A field study shows that playful interaction can facilitate the introduction of utilitarian robots in care homes, being nonthreatening and easy to make sense of. Residents previously reacted with distress to a Roomba robot, but were now amused by and played with our cartoonish cat robot or simply tolerated its presence. They showed awareness of the machine-nature of the robot, even while engaging in pretend-play. A playful approach to the design of functional robots can thus explicitly conceptualize such robots as social actors in their context of use.

There is growing interest in HCI to study ways to support access to accurate, accessible, relevant online health information for different populations. Yet, there remains a need to understand the barriers that are posed by the way our platforms are designed as well as how we might overcome these barriers for people with dementia. To address this, we conducted sixteen interviews and observation sessions with people with mild to moderate dementia. Our analysis uncovered four barriers to online health information and corresponding mitigation strategies that participants employed. We discuss how HCI researchers may apply these findings towards new technical approaches and standards concerning information accessibility and credibility for neurodiverse populations. Finally, we broaden the scope of HCI research to include investigations of the accessibility and credibility of online information for people with age-related cognitive impairment independent of proxies.

People with dementia and their caregivers aging in place have expressed the need for social, emotional, and recreational interventions at home. Listening to everyday sounds evokes memories and provides conversational cues to support social relations and elicit emotional responses for people with dementia. However, research has yet to explore how these meaningful experiences can be transferred into home settings. This paper presents the insights from our co-design study that involved three people with dementia and their partners in developing an interactive sound player for listening to everyday sounds at home. We report on the motivations of people with dementia and their caregivers to engage in meaningful sound-based activities at home and present the Tumbler as a prototype to foster initiative and agency in exploring familiar everyday sounds. We present design implications of how sound can enrich the everyday experiences of dementia by facilitating social and pleasurable moments at home.

Dementia can hinder a person's ability to engage with their relatives. Existing communication technologies do not support people with dementia in maintaining social contact since they are not designed for their abilities and needs. This paper presents LivingMoments, a communication system that enables the engagement of people with dementia with their relatives. The system uses digital and physical interaction design considering people's different and changing abilities. Over a six-week field evaluation of LivingMoments, involving six participants living at home with different levels of dementia, we collected qualitative and quantitative data about the experiences of them and their relatives. Based on the data analysis, we found the need to adapt communication to individual abilities, lowering barriers through content calibration and establishing habits for continuous use. We evinced a set of design considerations for technologies to support a lasting engagement of people with dementia with different and changing abilities.

Teleportation has become the de facto standard of locomotion in Virtual Reality (VR) environments. However, teleportation with parabolic and linear target aiming methods is restricted to horizontal 2D planes and it is unknown how they transfer to the 3D space. In this paper, we propose six 3D teleportation methods in virtual environments based on the combination of two existing aiming methods (linear and parabolic) and three types of transitioning to a target (instant, interpolated and continuous). To investigate the performance of the proposed teleportation methods, we conducted a controlled lab experiment (N = 24) with a mid-air coin collection task to assess accuracy, efficiency and VR sickness. We discovered that the linear aiming method leads to faster and more accurate target selection. Moreover, a combination of linear aiming and instant transitioning leads to the highest efficiency and accuracy without increasing VR sickness.

Navigating large-scale virtual spaces is a major challenge in Virtual Reality (VR) applications due to real-world spatial limitations. Walking-in-place (WIP) locomotion solutions may provide a natural approach for VR use cases that require locomotion to share similar qualities with walking in real-life. However, there is limited knowledge on the range of experiences across common WIP methods to inform the design of usable WIP solutions using consumer-accessible components. This paper contributes to this knowledge via a user study with 40 participants that experienced several easy-to-setup WIP methods in a VR commuting simulation. A nuanced understanding of cybersickness and exertion relationships and walking affordances based on different tracker setups were among the findings derived from a corroborated analysis of think-aloud, interview, and observational data, supplemented with self-reports of VR sickness, presence and flow. Practical design insights were then constructed along the dimensions of cybersickness, affordances, space and user interfaces.

Teleportation, which instantly moves users from their current location to the target location, has become the most popular locomotion technique in VR games. It enables fast navigation with reduced VR sickness but results in significantly reduced immersion. We present HeadWind, a novel approach to improve the experience of teleportation by simulating the haptic sensation of air drag when rapidly moving through the air in real life. Specifically, HeadWind modulates bursts of compressed air to the face and uses multiple nozzles to provide directional cues. To design the wearable device and to model airflow speed and duration for teleportation, we conducted three formative studies and a design session. User experience evaluation with 24 participants showed that HeadWind significantly improved realism, immersion, and enjoyment of teleportation in VR (p<.01) with large effect sizes (r>0.5), and was preferred by 96% of participants.

Esports play can cultivate real world skills. However, the path to mastery is not easy, and difficulty progressing can result in discontinuation. In the absence of a human coach, computational tools may provide much-needed guidance. However, the specific improvement activities that players engage in and the exact challenges they face are not well defined in the context of computational support. As such, most tools can only support players based on a high level understanding of their practices. We present the results of an interview study (n=17) that identified four improvement activities: practicing, leveraging the knowledge of others, tracking performance, and reflecting on gameplay and setting goals for the future, and four challenges: coordinating and collaborating with teammates, knowing what to do next, tracking game state, and tracking skill and improvement. We discuss six implications for future design and development based on these results.

In this study, we analyzed how the performance of professional Go players has changed since the advent of AlphaGo, the first artificial intelligence (AI) application to defeat a human world Go champion. We interviewed and surveyed professional Go players and found that AI has been actively introduced into the Go training process since the advent of AlphaGo. The significant impact of AI-based training was confirmed in a subsequent analysis of 6,292 games in Korean Go tournaments and Elo rating data of 1,362 Go players worldwide. Overall, the tendency of players to make moves similar to those recommended by AI has sharply increased since 2017. The degree to which players’ expected win rates fluctuate during a game has also decreased significantly since 2017. We also found that AI-based training has provided more benefits to senior players and allowed them to achieve Elo ratings higher than those of junior players.

Millions of Americans forego medical care due to a lack of non-emergency transportation, particularly minorities, older adults, and those who have disabilities or chronic conditions. Our study investigates the potential for using timebanks—community-based voluntary services that encourage exchanges of services for “time dollars” rather than money—in interventions to address healthcare transportation barriers to seed design implications for a future affordable ridesharing platform. In partnership with a timebank and a federally qualified healthcare center (FQHC), 30 participants completed activity packets and 29 of them attended online workshop sessions. Our findings suggest that promoting trust between drivers and riders requires systems that prioritize safety and reliability; yet, there were discrepancies in the ability of the timebank and FQHC to moderate trust. We also found that timebank supports reciprocity, but healthcare transportation requires additional support to ensure balanced reciprocity. We explain these findings drawing from network closure and trust literature. Finally, we contribute design implications for systems that promote trust and facilitate relational over transactional interactions, which help to promote reciprocity and reflect participants’ values.

Although child participation is required for successful Type 1 Diabetes (T1D) management, it is challenging because the child’s young age and immaturity make it difficult to perform self-care. Thus, parental caregivers are expected to be heavily involved in their child’s everyday illness management. Our study aims to investigate how children and parents collaborate to manage T1D and examine how the children become more independent in their self-management through the support of their parents. Through semi-structured interviews with children with T1D and their parents (N=41), our study showed that children’s knowledge of illness management and motivation for self-care were crucial for their transition towards independence. Based on these two factors, we identified four types of children’s collaboration (i.e., dependent, resistant, eager, and independent) and parents’ strategies for supporting their children’s independence. We suggest design implications for technologies to support collaborative care by improving children’s transition to independent illness management.

Adolescent and young adult childhood cancer survivors experience health complications, late or long-term biomedical complications, as well as economic and psychosocial challenges that can have a lifelong impact on their quality-of-life. As childhood cancer survivors transition into adulthood, they must learn to balance their identity development with demands of everyday life and the near- and long-term consequences of their cancer experience, all of which have implications for the ways they use existing technologies and the design of novel technologies. In this study, we interviewed 24 childhood cancer survivors and six caregivers about their cancer survivorship experiences. The results of our analysis indicate that the challenges of transitioning to adulthood as a cancer survivor necessitate the development and management of multiple societal, relational, and personal boundaries, processes that social computing technologies can help or hinder. This paper contributes to the empirical understanding of adolescent and young adult cancer survivors’ social experiences. We further contribute sociotechnical design provocations for researchers, designers, and community members to support survivors.

We report on a Diary Study investigating daily practices of Self-care by seven UK adults living with Human Immunodeficiency Virus (HIV), to understand their routines, experiences, needs and concerns, informing Self-care technology design to support living well. We advance a developing HCI literature evidencing how digital tools for self-managing health do not meet the complex needs of those living with long-term conditions, especially those from marginalised communities. Our evaluation of using a Self-care Diary as Design Probe responds to calls to study Self-care practices so that future digital health tools are better grounded in lived experiences of managing multi-morbidity. We contribute to HCI discourses including Personal Health Informatics, Lived Informatics and Reflection by illuminating psychosocial challenges for practicing and self-reporting on Self-care. We offer design implications from a Critical Digital Health perspective, addressing barriers to technology use related to trust, privacy, and representation, gaining new significance during the COVID-19 pandemic.

Adolescents in sub-Saharan Africa are at the epicenter of the global HIV epidemic. Yet, technology to support HIV management overwhelmingly focuses on adult medication adherence, neglecting the complex lives of adolescents in low-income regions. We present findings from interviews and focus groups that included twelve HIV-positive adolescents in Ethiopia, and eleven adults from their care circles. We leverage the Integrated Behavioral Model to examine the lived experience of HIV and the space for technology. Additionally, we present an inductive thematic analysis, which highlights non-disclosure as a central theme, i.e., adolescents remaining unaware of their HIV status. Drawing from these findings, we discuss how to account for (the lack of) disclosure in the design of technology to support HIV management, and reflect on whether technology could (and should) support the process. We further highlight the risks that researchers and designers need to be aware of when designing HIV management technology for this audience.

We investigate instant-messaging (IM) users’ sense-making and practices around read-receipts: a feature of IM apps for supporting the awareness of turn-taking, i.e., whether a message recipient has read a message. Using a grounded-theory approach, we highlight the importance of five contextual factors – situational, relational, interactional, conversational, and personal – that shape the variety of IM users’ sense-making about read-receipts and strategies for utilizing them in different settings. This approach yields a 21-part typology comprising five types of senders’ speculation about why their messages with read-receipts have not been answered; eight types of recipients’ causes/reasons behind such non-response; and four types of senders’ and recipients’ subsequent strategies, respectively. Mismatches between senders’ speculations about un-responded-to read-receipted messages (URRMs) and recipients’ self-reported explanations are also discussed as sources of communicative friction. The findings reveal that, beyond indicating turn-taking, read-receipts have been leveraged as a strategic tool for various purposes in interpersonal relations.

Text chat applications are an integral part of daily social and professional communication. However, messages sent over text chat applications do not convey vocal or nonverbal information from the sender, and detecting the emotional tone in text-only messages is challenging. In this paper, we explore the effects of speech balloon shapes on the sender-receiver agreement regarding the emotionality of a text message. We first investigated the relationship between the shape of a speech balloon and the emotionality of speech text in Japanese manga. Based on these results, we created a system that automatically generates speech balloons matching linear emotional arousal intensity by Auxiliary Classifier Generative Adversarial Networks (ACGAN). Our evaluation results from a controlled experiment suggested that the use of emotional speech balloons outperforms the use of emoticons in decreasing the differences between message senders’ and receivers’ perceptions about the level of emotional arousal in text messages.

Virtual workspaces rapidly increased during the COVID-19 pandemic, and for many new collaborators, working remotely was their first introduction to their colleagues. Building rapport is essential for a healthy work environment, and while this can be achieved through non-textual responses within chat-based systems (e.g., emoji, GIF, stickers, memes), those non-textual responses are typically associated with personal relationships and informal settings. We studied the experiences of new collaborators (questionnaire N=49; interview N=14) in using non-textual responses to communicate with unacquainted teams and the effect of non-textual responses on new collaborators’ interpersonal bonds. We found new collaborators selectively and progressively use non-textual responses to establish interpersonal bonds. Moreover, the use of non-textual responses has exposed several limitations when used on various platforms. We conclude with design recommendations such as expanding the scope of interpretable non-textual responses and reducing selection time.

Attention-management tools can restrict online communication, but may cause collateral damage to their users’ fulfillment of communication expectations. This paper explores the idea of integrating attention management into instant messaging (IM), by 1) disclosing restriction status via an online status indicator (OSI) to manage contacts’ expectations, and 2) imposing communication limits to reduce communication distraction. We used a speed-dating design method to allow 43 participants to rapidly compare 48 types of OSI restriction in various conversational contexts. We identified two “tug-of-wars” that take place when attention management is integrated into IM apps: one between fulfilling one’s contacts’ expectations and protecting one’s own attention, and the other, between protecting one’s privacy and asserting the justifiability of using communication restrictions. We also highlighted the participants’ desire to be diplomatic for sustaining their positive images and maintaining relational connectedness. Finally, we provide design recommendations for integrating attention management into IM apps.

Service design has gained tractions in Human-Computer Interaction (HCI) as an approach to deal with changes of technology design scopes. In the meantime, there are confusions around definitions of service design and its relevance to HCI. Despite the co-existence of interests and confusions, little research has been done for a comprehensive overview of how HCI interprets and adopts service design. This research performed a systematic literature review on extant HCI publications that claim to use service design. The review findings from the 179 publications revealed varying dimensions of service design taken up in HCI, relations between service design scopes and emerging technologies, as well as unclarity to service design in HCI and HCI's current tendency to use service design for the interaction level rather than the system level. We discuss future design and research opportunities for HCI by integrating the system level dimensions of service design.

Opportunities for AI and machine learning (ML) are vast in current interactive systems development. However, comparatively little is known about how functionality for the system behind the interface is designed and how design methodologies such as user-centered design have influence. This research focuses on how interdisciplinary teams that include UX practitioners design real-world enterprise ML systems outside of big technology companies. We conducted a survey with product managers, and interviews with interdisciplinary teams and individual UX practitioners. The findings show that nontechnical UX practitioners are highly capable in designing AI/ML systems. In addition to applying UX and interaction design expertise to make decisions regarding functionality, they employ skills that aid collaboration across interdisciplinary teams. However, our findings suggest existing HCI design techniques such as prototyping and simulating complexity of enterprise ML systems are insufficient. We propose adaptations to design practices and conclude that some existing research should be reconsidered.

This paper offers a theoretical and methodological framework of ‘techno-aesthetic encounters’ that supports nonlinear (situated, materially-driven, and multi-sensory) and art-based modes of inquiry in HCI and the broader STEM fields. We first investigate recent literatures in HCI and science and technology studies (STS) that explore nonlinear modes of practice and creativity in processes of technology design, and argue that better recognition of these dynamics may open space for art-based and nonlinear leaners and makers to more actively engage in HCI research and design. To meet this need, we study three renowned art-and-engineering practitioners (Klüver, Paik, Moog) and our own experimental project titled ‘The Electronicists’ in which participants from different disciplines collaborated to produce three hybrid works. Based on this work, we propose a framework of ‘techno-aesthetic encounters’ that pursues event-based creativity through the mediation of engineering, art, and humanistic engagements. We suggest trust-based experiments, error-engaged studio, and art-based ethnography as promising methodological tenets of this approach.

Designing technology for emergency medical services (EMS) can be difficult, for example, due to limited access to domain experts. To support designers who aim to engage in a participatory design process in EMS environments, we created and evaluated the Contextual Secondary Video Toolkit (CSVT). This method uses secondary video material and design cards that allow domain experts to identify and prioritise challenges in their work environment and generate design ideas that address them. We illustrate the effects of the CSVT on design processes by analysing four workshops during which aeromedical EMS staff explored the potential of augmented reality to support their work. Our results indicate that the CSVT can support reflection about work practices, aid the generation of design ideas, and facilitate genuine participation. Furthermore, our data indicates that the use of secondary video in design projects is appropriate and even has certain advantages compared to primary field video.

Here we present a co-design exploration into the potential of technology to playfully re-signify urban spaces. We created a speculative catalog of urban tech and used it to facilitate multi-stakeholder discussions about the playful potential of smart cities. The learnings from our co-design engagements embody different people's ideas of how tech might and might not support rich forms of urban play, and contribute to ongoing efforts at exploring how to playfully reconfigure our cities. We present: (1) a list of inspirational play potentials of urban spaces—i.e. playful things already people do, and enjoy, in the public space; (2) a portfolio of speculative ideas that show how tech might help to realize that potential; and (3) a discussion of stakeholders’ responses to these ideas. Our work can provide designers with inspiration and actionable advice for cultivating forms of urban play that cater to people's socio-emotional needs.

Previous research shows that laypeople’s trust in a machine learning model can be affected by both performance measurements of the model on the aggregate level and performance estimates on individual predictions. However, it is unclear how people would trust the model when multiple performance indicators are presented at the same time. We conduct an exploratory human-subject experiment to answer this question. We find that while the level of model confidence significantly affects people’s belief in model accuracy, both the model’s stated and observed accuracy generally have a larger impact on people’s willingness to follow the model’s predictions as well as their self-reported levels of trust in the model, especially after observing the model’s performance in practice. We hope the empirical evidence reported in this work could open doors to further studies to advance understanding of how people perceive, process, and react to performance-related information of machine learning.

AI systems are becoming increasingly pervasive within children’s devices, apps, and services. However, it is not yet well-understood how risks and ethical considerations of AI relate to children. This paper makes three contributions to this area: first, it identifies ten areas of alignment between general AI frameworks and codes for age-appropriate design for children. Then, to understand how such principles relate to real application contexts, we conducted a landscape analysis of children’s AI systems, via a systematic literature review including 188 papers. This analysis revealed a wide assortment of applications, and that most systems’ designs addressed only a small subset of principles among those we identified. Finally, we synthesised our findings in a framework to inform a new “Code for Age-Appropriate AI”, which aims to provide timely input to emerging policies and standards, and inspire increased interactions between the AI and child-computer interaction communities.

Inspiration from design examples plays a crucial role in the creative process of user interface design. However, current tools and techniques that support inspiration usually only focus on example browsing with limited user control or similarity-based example retrieval, leading to undesirable design outcomes such as focus drift and design fixation. To address these issues, we propose the GANSpiration approach that suggests design examples for both targeted and serendipitous inspiration, leveraging a style-based Generative Adversarial Network. A quantitative evaluation revealed that the outputs of GANSpiration-based example suggestion approaches are relevant to the input design, and at the same time include diverse instances. A user study with professional UI/UX practitioners showed that the examples suggested by our approach serve as viable sources of inspiration for overall design concepts and specific design elements. Overall, our work paves the road of using advanced generative machine learning techniques in supporting the creative design practice.

Questionnaires are fundamental learning and research tools for gathering insights and information from individuals, and now can be created easily using online tools. However, existing resources for creating questionnaires are designed for written languages (e.g. English) and do not support sign languages (e.g. American Sign Language). Sign languages (SLs) have unique visual characteristics that do not fit into user interface paradigms designed for written, text-based languages. Through a series of formative studies with the ASL signing community, this paper takes steps towards understanding the viability, potential benefit, challenges, and user interest in SL-centric surveys, a novel approach for creating questionnaires that meet the needs of deaf individuals using sign languages, without obligatory reliance on a written language to complete a questionnaire.

Manual sign systems have been introduced to improve the communication of children with intellectual developmental disabilities (IDD). Due to the lack of learning support tools, teachers face many practical challenges in teaching manual sign to children, such as low attention span and the need for persistent intervention. To address these issues, we collaborated with teachers to develop the Sondam Rhythm Game, a gesture-based rhythm game that assists in teaching manual sign language, and ran a four-week empirical study with five teachers and eight children with IDD. Based on video annotation and post-hoc interviews, our game-based learning approach has the potential to be effective at teaching manual sign to children with IDD. Our approach improved children attention span and motivation while also increasing the number of voluntary gestures made without the need for prompting. Other practical issues and learning challenges were also uncovered to improve teaching paradigms for children with IDD.

Recent design research has shown an interest in diffraction and agential realism, which promise to offer generative alternatives when designing with data that resist treating data as objective or neutral. We explore engaging diffractively with ‘lived data’ to surface felt and prospective aspects of data as it is entangled in everyday lives of designers. This paper presents five biodata-based case studies demonstrating how design researchers can create knowledge about human bodies and behaviors via strategies that allow them to engage data diffractively. These studies suggest that designers can find insights for designing with data as it is lived by working with it in a slow, open-ended fashion that leaves room for messiness and time for discovering difference. Finally, we discuss the role of ambiguous, open-ended data interpretations to help surface different meanings and entanglements of data in everyday lives.

Design produces valuable knowledge by offering new perspectives that reframe problematic situations. Research through Design (RtD) contributes new frames along with design work demonstrating a frame's value. Interestingly, RtD papers rarely describe how reframing happens. This gap in documentation unintentionally implies a romantic account of design, it implies that the first step of an RtD project is to have a brilliant idea. This is especially problematic in cases where the reframing causes a pivot that leads to a new research program. To help address this gap, we describe a case where through a series of three design experiments we experienced a research pivot. We describe how our work to improve web-table navigation for screen-reader users broke our frame. The break led to a new research program focused on constructing a conversational internet. This paper offers our case along with reflection on reporting design work that drives reframing.

We explore the use of cinematic “pre-visualization” (previs) techniques as a rapid ideation and design futuring method for human computer interaction (HCI) research. Previs approaches, which are widely used in animation and film production, use digital design tools to create medium-fidelity videos that capture richer interaction, motion, and context than sketches or static illustrations. When used as a design futuring method, previs can facilitate rapid, iterative discussions that reveal tensions, challenges, and opportunities for new research. We performed eight one-week design futuring sprints, in which individual HCI researchers collaborated with a lead designer to produce concept sketches, storyboards, and videos that examined future applications of their research. From these experiences, we identify recurring themes and challenges and present a One Week Futuring Workbook that other researchers can use to guide their own futuring sprints. We also highlight how variations of our approach could support other speculative design practices.

Personas represent the needs of users in diverse populations and impact design by endearing empathy and improving communication. While personas have been lauded for their benefits, we could locate no prior review of persona use cases in design, prompting the question: how are personas actually used to achieve these benefits? To address this question, we review 95 articles containing persona application across multiple domains, and identify software development, healthcare, and higher education as the top domains that employ personas. We then present a three-stage design hierarchy of persona usage to describe how personas are used in design tasks. Finally, we assess the increasing trend of persona initiatives aimed towards social good rather than solely commercial interests. Our findings establish a roadmap of best practices for how practitioners can innovatively employ personas to increase the value of designs and highlight avenues of using personas for socially impactful purposes.

Trauma is the physical, emotional, or psychological harm caused by deeply distressing experiences. Research with communities that may experience high rates of trauma has shown that digital technologies can create or exacerbate traumatic experiences. Via three vignettes, we discuss how considering the possible effects of trauma and traumatic stress reactions provides an explanatory lens with new insights into people’s technology experiences. Then, we present a framework—trauma-informed computing—in which we adapt and show how to apply six key principles of trauma-informed approaches to computing: safety, trust, peer support, collaboration, enablement, and intersectionality. Through specific examples, we describe how to apply trauma-informed computing in four areas of computing research and practice: user experience research & design, security & privacy, artificial intelligence & machine learning, and organizational culture in tech companies. We conclude by discussing how adopting trauma-informed computing will lead to benefits for all users, not only those experiencing trauma.

Recent work in HCI has shed light on structural issues of inequality in computing. Building on this work, this study analyzes the relatively understudied phenomenon of caste in computing. Contrary to common rhetorics of ‘castelessness,’ we show how computing worlds in India and Indian diasporic communities continue to be shaped and inflected by caste relations. We study how, when and where Dalits (formerly ‘untouchables’) encounter caste in computing. We show how they artfully navigate these caste inscriptions by interpreting, interrupting and ambiguating caste and by finding caste communities. Drawing on the life stories of 16 Dalit engineers and anti-caste, queer-feminist and critical race theories, we argue that a dynamic and performative approach to caste, and other forms of inequality in HCI and computing, emphasizes the artfulness and agency of those at the margins as they challenge structural inequality in everyday life. Lastly, we suggest practical ways of addressing caste to build more open and inclusive cultures of global computing.

Maker culture encourages do-it-yourself practices to create, repair, and repurpose technology. In Human-Computer Interaction (HCI) research, it is seen as a means of empowering people, providing affordable and customisable technology with potential to enrich areas such as education or assistive technology. To investigate this alleged potential, we performed an anthropological inquiry at an elementary school for disabled children that lasted one year, participating in everyday activities with students, teachers, and therapists. We observed ‘heterogeneity in a fluid environment’ and ‘creativity in the moment’ in an ‘endemically underfunded’ setting. We saw how technology is ‘injecting dependencies’, ‘reinforcing disability’, and ‘occupying time and space’, changing our view on the role that making can have. Leveraging Empowerment Theory, we highlight how (making) technology risks ignoring the intertwined dynamics between the individual, the organisational, and the community, and articulate points for reflection for technology in schools for disabled children for the HCI research community.

Technology research for neurodivergent conditions is largely shaped by research aims which privilege neuro-normative outcomes. As such, there is an epistemic imbalance in meaning making about these technologies. We conducted a critical literature review of technologies designed for people with ADHD, focusing on how ADHD is framed, the research aims and approaches, the role of people with ADHD within the research process, and the types of systems being developed within Computing and HCI. Our analysis and review is conducted explicitly from an insider perspective, bringing our perspectives as neurodivergent researchers to the topic of technologies in the context of ADHD. We found that 1) technologies are largely used to ‘mitigate’ the experiences of ADHD which are perceived as disruptive to neurotypical standards of behaviour; 2) little HCI research in the area invites this population to co-construct the technologies or to leverage neurodivergent experiences in the construction of research aims; and 3) participant resistance to deficit frames can be read within the researchers’ own accounts of participant actions. We discuss the implications of this status quo for disabled people and technology researchers alike, and close with a set of recommendations for future work in this area.

The field of digital mental health is making strides in the application of technology to broaden access to care. We critically examine how these technology-mediated forms of care might amplify historical injustices, and erase minoritized experiences and expressions of mental distress and illness. We draw on decolonial thought and critiques of identity-based algorithmic bias to analyze the underlying power relations impacting digital mental health technologies today, and envision new pathways towards a decolonial digital mental health. We argue that a decolonial digital mental health is one that centers lived experience over rigid classification, is conscious of structural factors that influence mental wellbeing, and is fundamentally designed to deter the creation of power differentials that prevent people from having agency over their care. Stemming from this vision, we make recommendations for how researchers and designers can support more equitable futures for people experiencing mental distress and illness.

Diminished reality (DR) refers to the concept of removing content from a user’s visual environment. While its implementation is becoming feasible, it is still unclear how users perceive and interact in DR-enabled environments and what applications it benefits. To address this challenge, we first conduct a formative study to compare user perceptions of DR and mediated reality effects (e. g., changing the color or size of target elements) in four example scenarios. Participants preferred removing objects through opacity reduction (i. e., the standard DR implementation) and appreciated mechanisms for maintaining a contextual understanding of diminished items (e. g., outlining). In a second study, we explore the user experience of performing tasks within DR-enabled environments. Participants selected which objects to diminish and the magnitude of the effects when performing two separate tasks (video viewing, assembly). Participants were comfortable with decreased contextual understanding, particularly for less mobile tasks. Based on the results, we define guidelines for creating general DR-enabled environments.

Imagine in the future people comfortably wear augmented reality (AR) displays all day, how do we design interfaces that adapt to the contextual changes as people move around? In current operating systems, the majority of AR content defaults to staying at a fixed location until being manually moved by the users. However, this approach puts the burden of user interface (UI) transition solely on users. In this paper, we first ran a bodystorming design workshop to capture the limitations of existing manual UI transition approaches in spatially diverse tasks. Then we addressed these limitations by designing and evaluating three UI transition mechanisms with different levels of automation and controllability (low-effort manual, semi-automated, fully-automated). Furthermore, we simulated imperfect contextual awareness by introducing prediction errors with different costs to correct them. Our results provide valuable lessons about the trade-offs between UI automation levels, controllability, user agency, and the impact of prediction errors.

Optical see-through Head-Mounted Displays (OST HMDs, OHMDs) are known to facilitate situational awareness while accessing secondary information. However, information displayed on OHMDs can cause attention shifts, which distract users from natural social interactions. We hypothesize that information displayed in paracentral and near-peripheral vision can be better perceived while the user is maintaining eye contact during face-to-face conversations. Leveraging this idea, we designed a circular progress bar to provide progress updates in paracentral and near-peripheral vision. We compared it with textual and linear progress bars under two conversation settings: a simulated one with a digital conversation partner and a realistic one with a real partner. Results show that a circular progress bar can effectively reduce notification distractions without losing eye contact and is more preferred by users. Our findings highlight the potential of utilizing the paracentral and near-peripheral vision for secondary information presentation on OHMDs.

Sharing annotations encourages feedback, discussion, and knowledge passing among readers and can be beneficial for personal and public use. Prior augmented reality (AR) systems have expanded these benefits to both digital and printed documents. However, despite smartphone AR now being widely available, there is a lack of research about how to use AR effectively for interactive document annotation. We propose Dually Noted, a smartphone-based AR annotation system that recognizes the layout of structural elements in a printed document for real-time authoring and viewing of annotations. We conducted experience prototyping with eight users to elicit potential benefits and challenges within smartphone AR, and this informed the resulting Dually Noted system and annotation interactions with the document elements. AR annotation is often unwieldy, but during a 12-user empirical study our novel structural understanding component allows Dually Noted to improve precise highlighting and annotation interaction accuracy by 13%, increase interaction speed by 42%, and significantly lower cognitive load over a baseline method without document layout understanding. Qualitatively, participants commented that Dually Noted was a swift and portable annotation experience. Overall, our research provides new methods and insights for how to improve AR annotations for physical documents.

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics.

Document description languages such as LaTeX are used extensively to author scientific and technical documents, but editing them is cumbersome: code-based editors only provide generic features, while WYSIWYG interfaces only support a subset of the language. Our interviews with 11 LaTeX users highlighted their difficulties dealing with textually-encoded abstractions and with the mappings between source code and document output. To address some of these issues, we introduce Transitional Representations for document description languages, which enable the visualisation and manipulation of fragments of code in relation to their generated output. We present i-LaTeX, a LaTeX editor equipped with Transitional Representations of formulae, tables, images, and grid layouts. A 16-participant experiment shows that Transitional Representations let them complete common editing tasks significantly faster, with fewer compilations, and with a lower workload. We discuss how Transitional Representations affect editing strategies and conclude with directions for future work.

Recent improvements in hand-tracking technologies support novel applications and developments of gesture interactions in virtual reality (VR). Current implementations are mostly convention-based, originating in a Western technological context, thereby creating a legacy bias in gesture interaction implementations. With expanding application contexts and growing user groups and contexts, the design and selection of gestures need to be diversified. In this paper we present an exploration of natural gestures, followed by their implementation in a VR application and co-design of new gestures with a marginalized San community in Namibia. This study contributes to the still scarce empirical work in user-driven gesture design research, aiming to reduce legacy bias, on a methodological and technical level as well as through engaging non-WEIRD participants. Our findings confirm the applicability of our method, combined with Partner and Priming suggested by Morris et al., to the design of gestures inspired by natural interactions. We also consider the implementation of user-designed gestures to be necessary to asses usability, usefulness and technical issues in VR. Furthermore, the research directly advances the HCI agenda for diversity, through an ongoing research and design partnership with an indigenous community in Southern Africa, thereby challenging systemic bias and promoting design for the pluriverse.

Gesture recognition systems using nearest neighbor pattern matching are able to distinguish gesture from non-gesture actions by rejecting input whose recognition scores are poor. However, in the context of gesture customization, where training data is sparse, learning a tight rejection threshold that maximizes accuracy in the presence of continuous high activity (HA) data is a challenging problem. To this end, we present the Voight-Kampff Machine (VKM), a novel approach for rejection threshold selection. VKM uses new synthetic data techniques to select an initial threshold that the system thereafter adjusts based on the training set size and expected gesture production variability. We pair VKM with a state-of-the-art custom gesture segmenter and recognizer to evaluate our system across several HA datasets, where gestures are interleaved with non-gesture actions. Compared to alternative rejection threshold selection techniques, we show that our approach is the only one that consistently achieves high performance.

Mobile apps are indispensable for people’s daily life. Complementing with automated GUI testing, manual testing is the last line of defence for app quality. However, the repeated actions and easily missing of functionalities make manual testing time-consuming and inefficient. Inspired by the game candy crush with flashy candies as hint moves for players, we propose an approach named NaviDroid for navigating testers via highlighted next operations for more effective and efficient testing. Within NaviDroid, we construct an enriched state transition graph with the triggering actions as the edges for two involved states. Based on it, we utilize the dynamic programming algorithm to plan the exploration path, and augment the GUI with visualized hints for testers to quickly explore untested activities and avoid duplicate explorations. The automated experiments demonstrate the high coverage and efficient path planning of NaviDroid and a user study further confirms its usefulness. The NaviDroid can help us develop more robust software that works in more mission-critical settings, not only by performing more thorough testing with the same effort that has been put in before, but also by integrating these techniques into different parts of development pipeline.

Previous gesture elicitation studies have found that user proposals are influenced by legacy bias which may inhibit users from proposing gestures that are most appropriate for an interaction. Increasing production during elicitation studies has shown promise moving users beyond legacy gestures. However, variety decreases as more symbols are produced. While several studies have used increased production since its introduction, little research has focused on understanding the effect on the proposed gesture quality, on why variety decreases, and on whether increased production should be limited. In this paper, we present a gesture elicitation study aimed at understanding the impact of increased production. We show that users refine the most promising gestures and that how long it takes to find promising gestures varies by participant. We also show that gestural refinements provide insight into the gestural features that matter for users to assign semantic meaning and discuss implications for training gesture classifiers.

Expertise-centric citizen science games (ECCSGs) can be powerful tools for crowdsourcing scientific knowledge production. However, to be effective these games must train their players on how to become experts, which is difficult in practice. In this study, we investigated the path to expertise and the barriers involved by interviewing players of three ECCSGs: Foldit, Eterna, and Eyewire. We then applied reflexive thematic analysis to generate themes of their experiences and produce a model of expertise and its barriers. We found expertise is constructed through a cycle of exploratory and social learning but prevented by instructional design issues. Moreover, exploration is slowed by a lack of polish to the game artifact, and social learning is disrupted by a lack of clear communication. Based on our analysis we make several recommendations for CSG developers, including: collaborating with professionals of required skill sets; providing social features and feedback systems; and improving scientific communication.

Microtransactions have become a major monetisation model in digital games, shaping their design, impacting player experience, and raising ethical concerns. Research in this area has chiefly focused on loot boxes. This begs the question whether other microtransactions might actually be more relevant and problematic for players. We therefore conducted a content analysis of negative player reviews (n=801) of top-grossing mobile and desktop games to determine which problematic microtransactions are most prevalent and salient for players. We found that problematic microtransactions with mobile games featuring more frequent and different techniques compared to desktop games. Across both, players minded issues related to fairness, transparency, and degraded user experience, supporting prior theoretical work, and importantly take issue with monetisation-driven design as such. We identify future research needs on why microtransactions in particular spark this critique, and which player communities it may be more or less representative of.

Games are considered promising for engaging people with climate change. In virtual worlds, players can adopt empowering roles to mitigate greenhouse gas emissions and/or adapt to climate impacts. However, the lack of a comprehensive exploration of existing climate-related identities and actions prevents understanding their potential. Here, we analyze 80 video games and classify avatar identities, or expected player roles, into six types. Climate selves encourage direct life changes; climate citizens are easy to identify with and imitate; climate heroes are inspirational figures upholding environmental values; empowered individuals deliberate to avoid a tragedy of the commons; authorities should consider stakeholders and the environment; and faction leaders engage in bi- or multilateral relations. Adaptation is often for decision-making profiles, while empowered individuals, authorities, and faction leaders usually face conflicting objectives. We discuss our results in relation to avatar research and provide suggestions for researchers, designers, and educators.

Now more than ever, people are using online platforms to communicate. Twitch, the foremost platform for live game streaming, offers many communication modalities. However, the platform lacks representation of social cues and signals of the audience experience, which are innately present in live events. To address this, we present a technology probe that captures the audience energy and response in a game streaming context. We designed a game and integrated a custom-communication modality—Commons Sense—in which the audience members’ heart rates are sensed via webcam, averaged, and fed into a video game to affect sound, lighting, and difficulty. We conducted an ‘in-the-wild’ evaluation with four Twitch streamers and their audience members (N=55) to understand how these groups interacted through Commons Sense. Audience members and streamers indicated high levels of enjoyment and engagement with Commons Sense, suggesting the potential of physiological interaction as a beneficial communication tool in live streaming.

VR has received increased attention as an educational tool and many argue it is destined to influence educational practices, especially with the emergence of the Metaverse. Most prior research on educational VR reports on applications or systems designed for specified educational or training objectives. However, it is also crucial to understand current practices and attitudes across disciplines, having a holistic view to extend the body of knowledge in terms of VR adoption in an authentic setting. Taking a higher-level perception of people in different roles, we conducted a qualitative analysis based on 23 interviews with major stakeholders and a series of participatory design workshops with instructors and students. We identified the stakeholders who need to be considered for using VR in higher education, and highlighted the challenges and opportunities critical for VR current and potential practices in the university classroom. Finally, we discussed the design implications based on our findings. This study contributes a detailed description of current perceptions and considerations from a multi-stakeholder perspective, providing new empirical insights for designing novel VR and HCI technologies in higher education.

Despite the digital revolution, physical space remains the site for teaching and learning embodied knowledge and skills. Both teachers and students must develop spatial competencies to effectively use classroom spaces, enabling fluid verbal and non-verbal interaction. While video permits rich activity capture, it provides no support for quickly seeing activity patterns that can assist learning. In contrast, position tracking systems permit the automated modelling of spatial behaviour, opening new possibilities for feedback. This paper introduces the design rationale for ”Dandelion Diagrams” that integrate participant location, trajectory and body orientation over a variable period. Applied in two authentic teaching contexts (a science laboratory, and a nursing simulation) we show how heatmaps showing only teacher/student location led to misinterpretations that were resolved by overlaying Dandelion Diagrams. Teachers also identified a variety of ways they could aid professional development. We conclude Dandelion Diagrams assisted sensemaking, but discuss the ethical risks of over-interpretation.

Mannequin-based simulations are widely used to train novice nurses. However, current mannequins have no dynamic facial expressions, which decreases the mannequins’ fidelity and impacts students’ learning outcomes and experience. This study proposes a projection-based AR system for overlaying dynamic facial expressions on a mannequin and implements the system in a stroke simulation. Thirty-six undergraduate nursing students participated in the study and were equally divided into the control (without the system) and experimental group (with the system). The participants’ gaze behavior, simulation performance, and subjective evaluation were measured. Results illustrated that the participants focused more on the face-animated mannequin than the traditional mannequin during the simulation. Nursing experts believed that the face-animated mannequin increased the participants’ performance in recognizing deviations but decreased their performance in seeking additional information. Moreover, the participants reported that the face-animated mannequin was more interactive and helpful for performing appropriate assessments than the traditional mannequin.

The US manufacturing industry is currently facing a welding workforce shortage which is largely due to inadequacy of widespread welding training. To address this challenge, we present a Virtual Reality (VR)-based training system aimed at transforming state-of-the-art-welding simulations and in-person instruction into a widely accessible and engaging platform. We applied backward design principles to design a low-cost welding simulator in the form of modularized units through active consulting with welding training experts. Using a minimum viable prototype, we conducted a user study with 24 novices to test the system’s usability. Our findings show (1) greater effectiveness of the system in transferring skills to real-world environments as compared to accessible video-based alternatives and, (2) the visuo-haptic guidance during virtual welding enhances performance and provides a realistic learning experience to users. Using the solution, we expect inexperienced users to achieve competencies faster and be better prepared to enter actual work environments.

Meditation has become a popular option to manage stress. Though studies examine technologies to assist in meditation, few explore how technology supports development of such skills for independent practice. From a two-phase mixed-methods study, we contribute learner-centered insights from 36 participants in a virtual reality environment designed to teach meditation skills to novices. In Phase I, we gathered affective and behavioral learner needs from 21 meditation novices, experts, and instructors to synthesize insights for learning. We then designed ZenVR: an interactive system to deliver an eight-lesson meditation curriculum to support learners’ progress. In Phase II, we conducted a 6-week longitudinal lab-based evaluation with 15 novice meditation learners. We found statistically significant improvements in mindfulness and self-reported meditation ability. Their insights from a self-managed practice, two weeks after the study ended, offered opportunities to understand how technology can be designed to offer progressive support without creating dependence in technology-mediated meditation practice.

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.

We investigate how multiple sliders with and without feedforward visualizations influence users’ control of generative models. In an online study (N=138), we collected a dataset of people interacting with a generative adversarial network (StyleGAN2) in an image reconstruction task. We found that more control dimensions (sliders) significantly increase task difficulty and user actions. Visual feedforward partly mitigates this by enabling more goal-directed interaction. However, we found no evidence of faster or more accurate task performance. This indicates a tradeoff between feedforward detail and implied cognitive costs, such as attention. Moreover, we found that visualizations alone are not always sufficient for users to understand individual control dimensions. Our study quantifies fundamental UI design factors and resulting interaction behavior in this context, revealing opportunities for improvement in the UI design for interactive applications of generative models. We close by discussing design directions and further aspects.

Early conversational agents (CAs) focused on dyadic human-AI interaction between humans and the CAs, followed by the increasing popularity of polyadic human-AI interaction, in which CAs are designed to mediate human-human interactions. CAs for polyadic interactions are unique because they encompass hybrid social interactions, i.e., human-CA, human-to-human, and human-to-group behaviors. However, research on polyadic CAs is scattered across different fields, making it challenging to identify, compare, and accumulate existing knowledge. To promote the future design of CA systems, we conducted a literature review of ACM publications and identified a set of works that conducted UX (user experience) research. We qualitatively synthesized the effects of polyadic CAs into four aspects of human-human interactions, i.e., communication, engagement, connection, and relationship maintenance. Through a mixed-method analysis of the selected polyadic and dyadic CA studies, we developed a suite of evaluation measurements on the effects. Our findings show that designing with social boundaries, such as privacy, disclosure, and identification, is crucial for ethical polyadic CAs. Future research should also advance usability testing methods and trust-building guidelines for conversational AI.

Nowadays, social robots have become human's important companions. The anthropomorphic features of robots, which are important in building natural user experience and trustable human-robot partnership, have attracted increasing attention. Among these features, eyes attract most audience's attention and are particularly important. This study aims to investigate the influence of robot eye design on users’ trustworthiness perception. Specifically, a simulation robot model was developed. Three sets of experiments involving sixty-six participants were conducted to investigate the effects of (i) visual complexity of eye design, (ii) blink rate, and (iii) gaze aversion of social robots on users’ perceived trustworthiness. Results indicate that high visual complexity and gaze aversion lead to higher perceived trustworthiness and reveal a positive correlation between the perceived anthropomorphic effect of eye design and users’ perceived trust, while a non-significant effect of blink rate has been found. Preliminary suggestions are provided for the design of social robots in future works.

Data science and machine learning provide indispensable techniques for understanding phenomena at scale, but the discretionary choices made when doing this work are often not recognized. Drawing from qualitative research practices, we describe how the concepts of positionality and reflexivity can be adapted to provide a framework for understanding, discussing, and disclosing the discretionary choices and subjectivity inherent to data science work. We first introduce the concepts of model positionality and computational reflexivity that can help data scientists to reflect on and communicate the social and cultural context of a model’s development and use, the data annotators and their annotations, and the data scientists themselves. We then describe the unique challenges of adapting these concepts for data science work and offer annotator fingerprinting and position mining as promising solutions. Finally, we demonstrate these techniques in a case study of the development of classifiers for toxic commenting in online communities.

The psychological costs of the attention economy are often considered through the binary of harmful design and healthy use, with digital well-being chiefly characterised as a matter of personal responsibility. This article adopts an interdisciplinary approach to highlight the empirical, ideological, and political limits of embedding this individualised perspective in computational discourses and designs of digital well-being measurement. We will reveal well-being to be a culturally specific and environmentally conditioned concept and will problematize user engagement as a universal proxy for well-being. Instead, the contributing factors of user well-being will be located in environing social, cultural, and political conditions far beyond the control of individual users alone. In doing so, we hope to reinvigorate the issue of digital well-being measurement as a nexus point of political concern, through which multiple disciplines can study experiences of digital ill as symptomatic of wider social inequalities and (capitalist) relations of power.

Mentorship and other social and relational support have been vital to poverty alleviation and transformative change. It is crucial to understand the underlying factors in the success of mentoring models and subsequent programs to support them. Thus, we conducted a mixed-methods study consisting of longitudinal surveys of community participants followed by semi-structured interviews with 28 community members, eight mentors, and two coaches participating in a community-based mentorship program. Drawing from community-based participatory research in partnership with a non-profit located in a Midwestern United States (U.S.) city, we unpack how the program supported self-sufficiency and economic mobility among adults experiencing financial hardships. Through an infrastructural lens, we attend to individuals’ infrastructuring work in social support, flexibility, and trust to support a “village” model of community-based mentorship. Our results show how the village model differs from traditional mentorship models that assume dyadic, one-to-one, often didactic, and hierarchical relationships (e.g., expert and protégé, adult and child) and are used primarily in the workplace and educational settings. The village mentorship model advocates for less hierarchical and more balanced relationships in non-institutional settings and flexible communication and technological needs. We discuss new research opportunities and design strategies for rethinking technology-mediated mentorship to support poverty-stricken adults in the U.S.

With increased public scrutiny of policing and growing calls for community-based violence prevention, street outreach programs that hire residents to mediate conflicts in their neighborhoods are gaining support. To understand how street outreach workers (SOWs) use information and communication technologies (ICTs) and how they envision future ICTs that better support their work, we interviewed 25 SOWs across three organizations. Results suggest that SOWs leverage ICTs to: 1) identify and mediate conflict; 2) support collaboration and teamwork; and 3) invoke community connections and trust. SOWs posit that new ICTs could provide a seamless infrastructure for communication among SOWs and between community members, assist with training to sharpen conflict negotiation skills, and provide insight on effective conflict mediation strategies.

While the presence of religious content is rapidly increasing over digital media, the HCI literature on digital media production has remained mostly limited by its focus on secular contents and analyses. Hence, the production, politics, and impact of such religious videos from the Global South have remained understudied in HCI. In this paper, we shed light on this topic through our nine-month-long ethnographic study on the production, sharing, and consumption of Islamic sermon videos (locally known as Waz) in Bangladesh. We report how faith, informal learning, local collaboration, creativity, and care play crucial roles in creating Islamic sermon videos and their proliferation online. We discuss how the sermon videos create a religious counterpublic in Bangladesh. We further discuss how such faith-based media production makes important lessons pertinent to the national grassroots politics in the Global South, politics of social media platforms, and HCI4D scholarship.

The challenge of designing against child sexual abuse becomes more complicated in conservative societies where talking about sex is tabooed. Our mix-method study, comprised of an online survey, five FGDs, and 20 semi-structured interviews in Bangladesh, investigates the common nature, location, and time of the abuse, post-incident support, and possible combating strategies. Besides revealing important facts, our findings highlight the need of decentering the design from the victims (children and/or guardians) to the community. Hence, building on the theory of transformative justice, we prototyped and evaluated ‘ShishuShurokkha’ – an online tool that involves the whole community by allowing anonymous bystander reporting, visualizing case-maps, connecting with legal, medical, and social support, and raising awareness. The evaluation of ShishuShurokkha shows the promise for such a communal approach toward combating child sexual abuse, and highlights the needs for sincere involvement of the government, NGOs, the legal, educational, and religious services in this.

Interactive AI systems such as voice assistants are bound to make errors because of imperfect sensing and reasoning. Prior human-AI interaction research has illustrated the importance of various strategies for error mitigation in repairing the perception of an AI following a breakdown in service. These strategies include explanations, monetary rewards, and apologies. This paper extends prior work on error mitigation by exploring how different methods of apology conveyance may affect people’s perceptions of AI agents; we report an online study (N=37) that examines how varying the sincerity of an apology and the assignment of blame (on either the agent itself or others) affects participants’ perceptions and experience with erroneous AI agents. We found that agents that openly accepted the blame and apologized sincerely for mistakes were thought to be more intelligent, likeable, and effective in recovering from errors than agents that shifted the blame to others.

The Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced “functional fixedness.” This work contributes Technical Reasoning as a theoretical model for the design of digital tools.

If you were significantly impacted by an algorithmic decision, how would you want the decision to be reviewed? In this study, we explore perceptions of review processes for algorithmic decisions that differ across three dimensions: the reviewer, how the review is conducted, and how long the review takes. Using a choice-based conjoint analysis we find that people prefer review processes that provide for human review, the ability to participate in the review process, and a timely outcome. Using a survey, we find that people also see human review that provides for participation to be the fairest review process. Our qualitative analysis indicates that the fairest review process provides the greatest likelihood of a favourable outcome, an opportunity for the decision subject and their situation to be fully and accurately understood, human involvement, and dignity. These findings have implications for the design of contestation procedures and also the design of algorithmic decision-making processes.

In the media, in policy-making, but also in research articles, algorithmic decision-making (ADM) systems are referred to as algorithms, artificial intelligence, and computer programs, amongst other terms. We hypothesize that such terminological differences can affect people’s perceptions of properties of ADM systems, people’s evaluations of systems in application contexts, and the replicability of research as findings may be influenced by terminological differences. In two studies (N = 397, N = 622), we show that terminology does indeed affect laypeople’s perceptions of system properties (e.g., perceived complexity) and evaluations of systems (e.g., trust). Our findings highlight the need to be mindful when choosing terms to describe ADM systems, because terminology can have unintended consequences, and may impact the robustness and replicability of HCI research. Additionally, our findings indicate that terminology can be used strategically (e.g., in communication about ADM systems) to influence people’s perceptions and evaluations of these systems.

Data is fundamental to AI/ML models. This paper investigates the work practices concerning data annotation as performed in the industry, in India. Previous human-centred investigations have largely focused on annotators’ subjectivity, bias and efficiency. We present a wider perspective of the data annotation: following a grounded approach, we conducted three sets of interviews with 25 annotators, 10 industry experts and 12 ML/AI practitioners. Our results show that the work of annotators is dictated by the interests, priorities and values of others above their station. More than technical, we contend that data annotation is a systematic exercise of power through organizational structure and practice. We propose a set of implications for how we can cultivate and encourage better practice to balance the tension between the need for high quality data at low cost and the annotators’ aspiration for well-being, career perspective, and active participation in building the AI dream.

We explore the impact of Casual Affective Triggers (CAT) on response rates of online surveys. As CAT, we refer to objects that can be included in survey participation invitations and trigger participants’ affect. The hypothesis is that participants who receive CAT-enriched invitations are more likely to respond to a survey. We conducted a study where the control condition received invitations without affective triggers, and the experimental condition received CAT-enriched invitations. We differentiated the triggers within the experimental condition: one-third of the population received a personalized invitation, one-third received a picture of the surveyor’s cat, and one-third received both. We followed up with a survey to validate our findings. Our results suggest that CATs have a positive impact on response rates. We did not find CATs to induce response bias.

Academics and community organisations are increasingly adopting co-research practices where participants contribute to qualitative data collection, analysis, and dissemination. These qualitative practices can often lack transparency that can present a problem for stakeholders (such as funding agencies) who seek evidence of the rigour and accountability in these decision-making processes. When qualitative research is done digitally, paradata is available as interaction logs that reveal the underlying processes, such as the time spent engaging with different segments of an interview. In practice, paradata is seldom used to examine the decisions associated with undertaking qualitative research. This paper explores the role of paradata arising from a four-month engagement with a community-led charity that used a digital platform to support their qualitative co-research project. Through observations of platform use and reflective post-deployment interviews, our findings highlight examples of paradata generated through digital tools in qualitative research, e.g., listening coverage, engagement rate, thematic maps and data discards. From this, we contribute a conceptualisation of paradata and discuss its role in qualitative research to improve process transparency, enhance data sharing, and to create feedback loops with research participants.

Despite efforts to augment or replace the 2-dimensional spreadsheet grid with formal data structures such as arrays and tables to ease formula authoring and reduce errors, the flexible grid remains overwhelmingly successful. Why? We interviewed a diverse sample of 21 spreadsheet users about their use of structure in spreadsheets. It emerges that data structuring is subject to a complex network of incentives and constraints, including factors extrinsic to spreadsheets such as the user’s expertise, auxiliary tools, and collaborator needs. Moreover, we find that table columns are an important abstraction, and that operations such as conditional formatting, data validation, and formula authoring can be implemented on table columns, rather than cell ranges. To probe this, we designed 4 click-through prototypes for a follow-up study with 20 participants. We found that although column operations improved the value proposition of structured tables, they are unlikely to supplant the advantages of the flexible grid.

Integration of online and offline retail faces challenges in technology adoption, interaction style evolution and customer behavior shifts, while also being complicated by diverse perspectives from different stakeholders of consumers, retail staff, and retail business unit people. To explore how we can tackle the aforementioned challenges, this work applied the data-enabled design method and participatory data analysis to a case study, where 400 student consumers’ shopping behavior data was collected, cross-analyzed, and visualized in a campus chain store. We then invited 13 stakeholders to join a co-creation workshop for a further participatory data analysis. In the workshop, the different stakeholders came to a design consensuses which we summarized into a series of practical design recommendations for improving the current store. Finally, we generalized the case study process as a contextual-informed-aware model, which can contribute to professional design practice for the retail industry.

Field workers, like farmers and radiologists, play a crucial role in dataset collection for AI models in low-resource settings. However, we know little about how field workers’ expertise is leveraged in dataset and model development. Based on 68 interviews with AI developers building for low-resource contexts, we find that developers reduced field workers to data collectors. Attributing poor data quality to worker practices, developers conceived of workers as corrupt, lazy, non-compliant, and as datasets themselves, pursuing surveillance and gamification to discipline workers to collect better quality data. Even though models sought to emulate the expertise of field workers, AI developers treated workers as non-essential and deskilled their expertise in service of building machine intelligence. We make the case for why field workers should be recognised as domain experts and re-imagine domain expertise as an essential partnership for AI development.

Apple announced the introduction of app privacy details to their App Store in December 2020, marking the first ever real-world, large-scale deployment of the privacy nutrition label concept, which had been introduced by researchers over a decade earlier. The Apple labels are created by app developers, who self-report their app’s data practices. In this paper, we present the first study examining the usability and understandability of Apple’s privacy nutrition label creation process from the developer’s perspective. By observing and interviewing 12 iOS app developers about how they created the privacy label for a real-world app that they developed, we identified common challenges for correctly and efficiently creating privacy labels. We discuss design implications both for improving Apple’s privacy label design and for future deployment of other standardized privacy notices.

Incident response playbooks provide step-by-step guidelines to help security operations personnel quickly respond to specific threat scenarios. Although playbooks are common in the security industry, they have not been empirically evaluated for effectiveness. This paper takes a first step toward measuring playbooks and the frameworks used to design them, using two studies conducted in an enterprise environment. In the first study, twelve security professionals created two playbooks each, using two standard playbook design frameworks; the resulting playbooks were evaluated by experts for accuracy. In the second, we observed five personnel using the created playbooks in no-notice threat exercises within a live security-operations center. We find that playbooks can help simplify and support incident response efforts. However, playbooks designed using the frameworks we examined often lack sufficient detail for real-world use, particularly for more junior technicians. We provide recommendations for improving playbooks, playbook frameworks, and organizational processes surrounding playbook use.

Reliably recruiting participants with programming skills is an ongoing challenge for empirical studies involving software development technologies, often leading to the use of crowdsourcing platforms and computer science (CS) students. In this work, we use five existing survey instruments to explore the programming skills, privacy and security attitudes, and secure development self-efficacy of participants from a CS student mailing list and four crowdsourcing platforms (Appen, Clickworker, MTurk, and Prolific). We recruited 613 participants who claimed to have programming skills and assessed recruitment channels regarding costs, quality, programming skills, as well as privacy and security attitudes. We find that 27% of crowdsourcing participants, 40% of crowdsourcing participants who self-report to be developers, and 89% of CS students answered all programming skill questions correctly. CS students were the most cost-effective recruitment channel and rated themselves lower than crowdsourcing participants about secure development self-efficacy.

Decentralized finance (DeFi) enables crypto-asset holders to conduct complex financial transactions, while maintaining control over their assets in the blockchain ecosystem. However, the transparency of blockchain networks and the open mechanism of DeFi applications also cause new security issues. In this paper, we focus on sandwich attacks, where attackers take advantage of the transaction confirmation delay and cause financial losses for victims. We evaluate the impact and investigate users’ perceptions of sandwich attacks through a mix-method study. We find that due to users’ lack of technical background and insufficient notifications from the markets, many users were not aware of the existence and the impact of sandwich attacks. They also had a limited understanding of how to resolve the security issue. Interestingly, users showed high tolerance for the impact of sandwich attacks on individuals and the ecosystem, despite potential financial losses. We discuss general implications for users, DeFi applications, and the community.

Tactile maps can help people who are blind or have low-vision navigate and familiarize themselves with unfamiliar locations. Ideally, tactile maps can be customized to an individual’s unique needs and abilities because of their limited space for representation. We present Maptimizer, a tool that generates tactile maps based on users’ preferences and requirements. Maptimizer uses a two stage optimization process to pair representations with geographic information and tune those representations to present that information more clearly. In a small user study, Maptimizer helped participants more successfully and efficiently identify locations of interest in unknown areas. These results demonstrate the utility of optimization techniques and generative design in complex accessibility domains.

YouTube is a space where people with disabilities can reach a wider online audience to present what it is like to have disabilities. Thus, it is imperative to understand how content creators with disabilities strategically interact with algorithms to draw viewers around the world. However, considering that the algorithm carries the risk of making less inclusive decisions for users with disabilities, whether the current algorithmic experiences (AXs) on video platforms is inclusive for creators with disabilities is an open question. To address that, we conducted semi-structured interviews with eight YouTubers with disabilities. We found that they aimed to inform the public of diverse representations of disabilities, which led them to work with algorithms by strategically portraying disability identities. However, they were disappointed that the way the algorithms work did not sufficiently support their goals. Based on findings, we suggest implications for designing inclusive AXs that could embrace creators’ subtle needs.

Engagement with electronic toolkits enhances people’s creative abilities, self-esteem, problem-solving skills and enables the creation of personally meaningful artifacts. A variety of simplified electronics toolkits are increasingly available to help different user groups engage with technology. However, they are often inaccessible for people with intellectual disabilities (IDs), who experience a range of cognitive and physical impairments. We designed and developed TronicBoards, a curated set of accessible electronic modules, to address this gap. We evaluated it one-on-one with 10 participants using a guided exploration approach. Our analysis revealed that participants were able to create simple sensor-based interactive circuits with varying levels of assistance. We report the strengths and weaknesses of TronicBoards, considering participants’ successes and challenges in manipulating and comprehending toolkit components, circuit building activities, and troubleshooting processes. We discuss implications for designing inclusive electronics toolkits for people with IDs, particularly in considering design elements that improve functionality, comprehensibility and agency.

The data table is a basic but versatile representation to communicate data. From government reports to bank statements, tables effectively carry essential data-driven information by visually organizing data using rows, columns, and other arrangements (e.g., merged cells). However, many tables online neglect the accessibility requirements for people who rely on screen readers, such as people who are blind or have low vision (BLV). First, we consolidated guidelines to understand what makes a table inaccessible for BLV people. We conducted an interview study to understand the importance of tables and identify further design requirements for an accessible table. We built a tool that automatically detects HTML formatted tables online and transforms them into accessible tables. Our evaluative study demonstrates how our tool can help participants understand the table’s structure and layout and support smooth navigation when the table is large and complex.

Collaborative document editing tools are widely used in professional and academic workplaces. While these tools provide basic accessibility support, it is challenging for blind users to gain collaboration awareness that sighted people can easily obtain using visual cues (e.g., who is editing where and what). Through a series of co-design sessions with a blind coauthor, we identified the current practices and challenges in collaborative editing, and iteratively designed CollabAlly, a system that makes collaboration awareness in document editing accessible to blind users. CollabAlly extracts collaborator, comment, and text-change information and their context from a document and presents them in a dialog box to provide easy access and navigation. CollabAlly uses earcons to communicate background events unobtrusively, voice fonts to differentiate collaborators, and spatial audio to convey the location of document activity. In a study with 11 blind participants, we demonstrate that CollabAlly provides improved access to collaboration awareness by centralizing scattered information, sonifying visual information, and simplifying complex operations.

We explore the design and utility of situated manual self-tracking visualizations on dedicated displays that integrate data tracking into existing practices and physical environments. Situating self-tracking tools in relevant locations is a promising approach to enable reflection on and awareness of data without needing to rely on sensorized tracking or personal devices. In both a long-term autobiographical design process and a co-design study with six participants, we rapidly prototyped and deployed 30 situated self-tracking applications over a ten month period. Grounded in the experience of designing and living with these trackers, we contribute findings on logging and data entry, the use of situated displays, and the visual design and customization of trackers. Our results demonstrate the potential of customizable dedicated self-tracking visualizations that are situated in relevant physical spaces, and suggest future research opportunities and new potential applications for situated visualizations.

Basketball writers and journalists report on the sport that millions of fans follow and love. However, the recent emergence of pervasive data about the sport and the growth of new forms of sports analytics is changing writers’ jobs. While these writers seek to leverage the data and analytics to create engaging, data-driven stories, they typically lack the technical background to perform analytics or efficiently explore data. We investigated and analyzed the work and context of basketball writers, interviewed nine stakeholders to understand the challenges from a holistic view. Based on what we learned, we designed and constructed two interactive visualization systems that support rapid and in-depth sports data exploration and sense-making to enhance their articles and reporting. We deployed the systems during the recent NBA playoffs to gather initial feedback. This article describes the visualization design study we conducted, the resulting visualization systems, and what we learned to potentially help basketball writers in the future.

Data videos are an increasingly popular storytelling form. The opening of a data video critically influences its success as the opening either attracts the audience to continue watching or bores them to abandon watching. However, little is known about how to create an attractive opening. We draw inspiration from the openings of famous films to facilitate designing data video openings. First, by analyzing over 200 films from several sources, we derived six primary cinematic opening styles adaptable to data videos. Then, we consulted eight experts from the film industry to formulate 28 guidelines. To validate the usability and effectiveness of the guidelines, we asked participants to create data video openings with and without the guidelines, which were then evaluated by experts and the general public. Results showed that the openings designed with the guidelines were perceived to be more attractive, and the guidelines were praised for clarity and inspiration.

Designing responsive visualizations can be cast as applying transformations to a source view to render it suitable for a different screen size. However, designing responsive visualizations is often tedious as authors must manually apply and reason about candidate transformations. We present Cicero, a declarative grammar for concisely specifying responsive visualization transformations which paves the way for more intelligent responsive visualization authoring tools. Cicero’s flexible specifier syntax allows authors to select visualization elements to transform, independent of the source view’s structure. Cicero encodes a concise set of actions to encode a diverse set of transformations in both desktop-first and mobile-first design processes. Authors can ultimately reuse design-agnostic transformations across different visualizations. To demonstrate the utility of Cicero, we develop a compiler to an extended version of Vega-Lite, and provide principles for our compiler. We further discuss the incorporation of Cicero into responsive visualization authoring tools, such as a design recommender.

We present results of a survey (n=42) investigating how close others (family members, friends, or other community members) assist older adults with banking tasks in Canada. We asked what types of banking tasks they help with and what modalities they used. Our results show many close others help older adults by leveraging online banking, and this is especially true when the close other is an older adult themselves. We also found that close others who help older adults via online banking often know the online banking credentials for the older adults they assist, which presents privacy and security issues and could open the door to financial exploitation. Our results demonstrate the need for the design of online banking mechanisms that more explicitly acknowledge the nuanced and temporally changing role of close others in helping older adults with banking.

AAC research has traditionally focused on input speed, leaving higher-level communication goals such as relational maintenance under-explored. Through semi-structured interviews with AAC users with motor and speech impairments and their primary family caregivers, we offer a nuanced understanding of AAC’s roles in maintaining close relationships. Our inductive analysis reveals emerging themes including how AAC users and their partners share the physical and mental workload to overcome communication barriers in complex situations. Our deductive application of the Relational Maintenance Strategies framework exposes the efforts made and the challenges encountered in managing social engagements, providing mutual support, and decoding implicit expressions. From these insights, we propose novel research directions for better supporting maintenance strategies and social purposes of communication, including notably mediating relational tensions, leveraging empowerment and identity, and supporting interactions for social closeness and etiquette, which we hope will motivate discussion in HCI communities on expanding AAC research space.

Methods are fundamental to doing research and can directly impact who is included in scientific advances. Given accessibility research's increasing popularity and pervasive barriers to conducting and participating in research experienced by people with disabilities, it is critical to ask how methods are made accessible. Yet papers rarely describe their methods in detail. This paper reports on 17 interviews with accessibility experts about how they include both facilitators and participants with disabilities in popular user research methods. Our findings offer strategies for anticipating access needs while remaining flexible and responsive to unexpected access barriers. We emphasize the importance of considering accessibility at all stages of the research process, and contextualize access work in recent disability and accessibility literature. We explore how technology or processes could reflect a norm of accessibility. Finally, we discuss how various needs intersect and conflict and offer a practical structure for planning accessible research.

Black older adults from lower socioeconomic environments are often neglected in health technology interventions. Voice assistants have a potential to make healthcare more accessible to older adults, yet, little is known about their experiences with this type of health information seeking, especially Black older adults. Through a three-phase exploratory study, we explored health information seeking with 30 Black older adults in lower-income environments to understand how they ask health-related questions, and their perceptions of the Google Home being used for that purpose. Through our analysis, we identified the health information needs and common search topics, and discussed the communication breakdowns and types of repair performed. We contribute an understanding of cultural code-switching that has to be done by these older adults when interacting with voice assistants, and the importance of such phenomenon when designing for historically excluded groups.

Flying drones is an increasingly popular activity. However, it is challenging due to the required perceptual and motor skills for following and stabilizing the drone, especially for people with special needs. This paper describes CandyFly, an application supporting people with diverse sensory, cognitive and motor impairments to pilot drones. We observed an existing accessible piloting workshop and evaluated CandyFly during eight additional workshops over three and a half years using a research-through-design process and ability-based design methods. We identified users’ needs, formulated requirements and explored adaptive interactions such as using pressure-sensitive keys, adjusting controls to the pilots’ range of motion, or limiting the drone’s degrees of freedom to cope with a broad range of disabilities. Our results show that the pilots and their caregivers enjoyed flying and emphasized CandyFly’s ability to be tailored to specific needs. Our findings offer a framework for designing adaptable systems and can support the design of future assistive and recreational systems.

People with language impairments, such as aphasia, use a range of total communication strategies. These go beyond spoken language to include non-verbal utterances, props and gestures. The uptake of videoconferencing platforms necessitated by the Covid-19 pandemic means that people with aphasia now use these communication strategies online. However, no data exists on the impact of videoconferencing on communication for this population. Working with an aphasia charity that moved its conversation support sessions online, we investigated the experience of communication via a videoconferencing platform. We report a study which investigated this through: 1) observations of online conversation support sessions; 2) interviews with speech and language therapists and volunteers; and 3) interviews with people with aphasia. Our findings reveal the unique and creative ways that the charity and its members with aphasia adapted their communication to videoconferencing. We unpack specific, novel challenges relating to total communication via videoconferencing and the related impacts on social and privacy issues.

Webtoon is a type of digital comics read online where readers can leave comments to share their thoughts on the story. While it has experienced a surge in popularity internationally, people with visual impairments cannot enjoy webtoon with the lack of an accessible format. While traditional image description practices can be adopted, resulting descriptions cannot preserve webtoons’ unique values such as control over the reading pace and social engagement through comments. To improve the webtoon reading experience for BLV users, we propose Cocomix, an interactive webtoon reader that leverages comments into the design of novel webtoon interactions. Since comments can identify story highlights and provide additional context, we designed a system that provides 1) comments-based adaptive descriptions with selective access to details and 2) panel-anchored comments for easy access to relevant descriptive comments. Our evaluation (N=12) showed that Cocomix users could adapt the description for various needs and better utilize comments.

Most people who are blind interact with social media content with the assistance of a screen reader, a software that converts text to speech. However, the language used in social media is well-known to contain several informal out-of-vocabulary words (e.g., abbreviations, wordplays, slang), many of which do not have corresponding standard pronunciations. The narration behavior of screen readers for such out-of-vocabulary words and the corresponding impact on the social media experience of blind screen reader users are still uncharted research territories. Therefore we seek to plug this knowledge gap by examining how current popular screen readers narrate different types of out-of-vocabulary words found on Twitter, and also, how the presence of such words in tweets influences the interaction behavior and comprehension of blind screen reader users. Our investigation showed that screen readers rarely autocorrect out-of-vocabulary words, and moreover they do not always exhibit ideal behavior for certain prolific types of out-of-vocabulary words such as acronyms and initialisms. We also observed that blind users often rely on tedious and taxing workarounds to comprehend actual meanings of out-of-vocabulary words. Informed by the observations, we finally discuss methods that can potentially reduce this interaction burden for blind users on social media.

As video conferencing (VC) has become increasingly necessary for many aspects of daily life, many d/Deaf and hard of hearing people, particularly those who communicate via sign language (signers), face distinct accessibility barriers. To better understand the unique requirements for participating in VC using a visual-gestural language, such as ASL, and to identify practical design considerations for signer-inclusive videoconferencing, we conducted 12 interviews and four co-design sessions with a total of eight d/Deaf signers and eight ASL interpreters. We found that participants’ access needs regarding consuming information (e.g., visual clarity of signs), communicating (e.g., getting attention of others), and collaborating (e.g., working with interpreter teams) are not well-met on existing VC platforms. We share novel insights into attending and conducting signer-accessible video conferences, outline considerations for future VC design, and provide guidelines for conducting remote research with d/Deaf signers.

Pair-learning is beneficial for learning outcome, motivation, and social presence, and so is virtual reality (VR) by increasing immersion, engagement, motivation, and interest of students. Nevertheless, there is a research gap if the benefits of pair-learning and VR can be combined. Furthermore, it is not clear which influence it has if only one or both peers use VR. To investigate these aspects, we implemented two types of VR pair-learning systems, a symmetric system with both peers using VR and an asymmetric system with one using a tablet. In a user study (N=46), the symmetric system statistically significantly provided higher presence, immersion, player experience, and lower intrinsic cognitive load, which are all important for learning. Symmetric and asymmetric systems performed equally well regarding learning outcome, highlighting that both are valuable learning systems. We used these findings to define guidelines on how to design co-located VR pair-learning applications, including characteristics for symmetric and asymmetric systems.

Cross-reality systems empower users to transition along the reality-virtuality continuum or collaborate with others experiencing different manifestations of it. However, prototyping these systems is challenging, as it requires sophisticated technical skills, time, and often expensive hardware. We present VRception, a concept and toolkit for quick and easy prototyping of cross-reality systems. By simulating all levels of the reality-virtuality continuum entirely in Virtual Reality, our concept overcomes the asynchronicity of realities, eliminating technical obstacles. Our VRception Toolkit leverages this concept to allow rapid prototyping of cross-reality systems and easy remixing of elements from all continuum levels. We replicated six cross-reality papers using our toolkit and presented them to their authors. Interviews with them revealed that our toolkit sufficiently replicates their core functionalities and allows quick iterations. Additionally, remote participants used our toolkit in pairs to collaboratively implement prototypes in about eight minutes that they would have otherwise expected to take days.

“Virtual-Physical Perceptual Manipulations” (VPPMs) such as redirected walking and haptics expand the user’s capacity to interact with Virtual Reality (VR) beyond what would ordinarily physically be possible. VPPMs leverage knowledge of the limits of human perception to effect changes in the user’s physical movements, becoming able to (perceptibly and imperceptibly) nudge their physical actions to enhance interactivity in VR. We explore the risks posed by the malicious use of VPPMs. First, we define, conceptualize and demonstrate the existence of VPPMs. Next, using speculative design workshops, we explore and characterize the threats/risks posed, proposing mitigations and preventative recommendations against the malicious use of VPPMs. Finally, we implement two sample applications to demonstrate how existing VPPMs could be trivially subverted to create the potential for physical harm. This paper aims to raise awareness that the current way we apply and publish VPPMs can lead to malicious exploits of our perceptual vulnerabilities.

We performed a mixed-design study with 56 participants to compare the effect of horizontal FOV (hfov) and vertical FOV (vfov) on egocentric distance perception in four different realistic virtual environments (VEs). We also compared VE attributes of indoor/outdoor and cluttered/uncluttered. The participants blind-walked towards four different targets at 3m, 4m, 5m, and 6m distance while wearing a backpack computer and a wide FOV head-mounted display (HMD). The combinations of 165°, 110° and 45° hfovs, and 110° and 35° vfovs was simulated in the same HMD. The results indicated more accurate distance judgement with larger hfov with no significant effect of vfov. More accurate distance judgement in indoor VEs compared to outdoor VEs was observed. Also, participants judged distances more accurately in cluttered environments versus uncluttered environments. These results highlight that the environment is important in distance-critical VR applications and wider hfov should be considered for an improved distance judgment.

The computer as a creative medium has largely influenced how we make, use, and share images. This study investigates how similarly or distinctively human designers and the computers see (i.e., process visual information from the activity of seeing) in creative practice. Based on the review of the perspectives relevant to seeing (e.g., visual experience, visual creativity, visual computing), we practiced a visual inquiry by marking how we see photographic images in comparison to what computer vision processes from the same images. Taking a researcher-introspective approach, two authors reflectively analyzed the processes and outcomes of the inquiry. The findings revealed that during the creative practice, the activity of seeing is a meaning-making process that is guided, shifted, and diffracted through visual conversations between visible and invisible qualities suggested by compositional rules and computer vision. We discuss potentials of computational visual intelligence as creative agents and conclude with methodological and practical implications.

We present Patch-O, a novel deformable interface devised as a woven patch that enables diverse movement-based interactions adaptive to garments or on-skin wearing. Patch-O interfaces are uniquely detachable and relocatable soft actuation units that can be sewn or attached to clothing or skin at various locations. To optimize the morphing effect while preserving a slim form factor, we introduce a construction approach that integrates actuators at a structural level and varies the texture and stiffness of the woven substrate locally. We implement three basic actuation primitives, including bending, expanding, and shrinking, and experiment with aggregation parameters to exhaustively extend the design space. A final workshop study inviting textile practitioners to create personalized designs of Patch-O provides insights into the expressiveness of the approach for wearable interactions. We conclude with three applications inspired by users’ designs and showcase the aesthetic and functional usages enabled by the deformable woven patches.

DNA-based digital data storage technology is hailed as a potential solution for the issues around exponential global data production. However, while the technology continues to strive towards its full commercialization, there is a lack of discourse on how it could be applied to facilitate interactions that are meaningful, ethical, and socially sustainable. As an approach to address this gap, we hosted a series of online workshops, soliciting 15 participants to engage in grounded speculations on plausible futures of DNA data storage. Themes drawn from the resulting imaginaries and discussions were situated within a selection of existing HCI literature, to generate an initial set of design opportunities and challenges for DNA data storage. Early analysis suggests that the system could be designed to 1) facilitate meaningful interactions that are intangible and molecular, and 2) foster better human relationship with more-than-human entities. Furthermore, we highlight the imperative for cross-disciplinary collaborations and pedagogy, to ensure fair and high quality access to the technology.

The increased adoption of smart home cameras (SHCs) foregrounds issues of surveillance, power, and privacy in homes and neighborhoods. However, questions remain about how people are currently using these devices to monitor and surveil, what the benefits and limitations are for users, and what privacy and security tensions arise between primary users and other stakeholders. We present an empirical study with 14 SHC users to understand how these devices are used and integrated within everyday life. Based on semi-structured qualitative interviews, we investigate users’ motivations, practices, privacy concerns, and social negotiations. Our findings highlight the SHC as a perceptually powerful and spatially sensitive device that enables a variety of surveillant uses outside of basic home security—from formally surveilling domestic workers, to casually spying on neighbors, to capturing memories. We categorize surveillant SHC uses, clarify distinctions between primary and non-primary users, and highlight under-considered design directions for addressing power imbalances among primary and non-primary users.

Technological advancement has opened up opportunities for new sports and physical activities. We introduce a concept called machine-mediated teaming, in which a human and a surrogate machine form a team to participate in physical sports games. To understand the experience of machine-mediated teaming and the guidelines for designing the system to achieve the concept, we built a case study system based on tug-of-war. Our system is a sports game played by two against two. One team consists of a player who actually pulls the rope and another player who participates in the physical game by controlling the machine’s actuators. We conducted user studies using this system to investigate the sport experience in this form and to reveal insights to inform future research on machine-mediated teaming. Based on the data obtained from the user studies, we clarified three perspectives, machine stamina, action space, and explicit feedback, that should be considered when designing future machine-mediated teaming systems. The research presented in this paper offers a first step towards exploring how humans and machines can coexist in highly dynamic physical interactions.

This paper addresses calls for more research on privacy in the gig economy across a range of work platforms. To understand privacy risks, behaviors, and consequences from the perspective of workers, we analyzed workers’ posts about privacy and surveillance from 12 Reddit forums representing four main types of work (crowdwork, freelancing, ridesharing, and delivery). We found that workers perceive both platform companies and customers as sources of unnecessary and opaque data collection and surveillance that can threaten their privacy, safety, and economic outcomes. Workers also engage in many risk mitigation strategies, including self-protective surveillance behaviors such as video recording themselves and customers, as a costly but necessary response to power imbalances created by surveillance. Based on our multi-platform analysis, we present a guiding set of questions that workers, designers, and researchers can use to assess the privacy implications of current and future gig work platforms.

A person’s online security setup is tied to the security of their individual accounts. Some accounts are particularly critical as they provide access to other online services. For example, an email account can be used for external account recovery or to assist with single-sign-on. The connections between accounts are specific to each user’s setup and create unique security problems that are difficult to remedy by following generic security advice. In this paper, we develop a method to gather and analyze users’ online accounts systematically. We demonstrate this in a user study with 20 participants and obtain detailed insights on how users’ personal setup choices and behaviors affect their overall account security. We discuss concrete usability and privacy concerns that prevented our participants from improving their account security. Based on our findings, we provide recommendations for service providers and security experts to increase the adoption of security best practices.

Many websites have added cookie consent interfaces to meet regulatory consent requirements. While prior work has demonstrated that they often use dark patterns — design techniques that lead users to less privacy-protective options — other usability aspects of these interfaces have been less explored. This study contributes a comprehensive, two-stage usability assessment of cookie consent interfaces. We first inspected 191 consent interfaces against five dark pattern heuristics and identified design choices that may impact usability. We then conducted a 1,109-participant online between-subjects experiment exploring the usability impact of seven design parameters. Participants were exposed to one of 12 consent interface variants during a shopping task on a prototype e-commerce website and answered a survey about their experience. Our findings suggest that a fully-blocking consent interface with in-line cookie options accompanied by a persistent button enabling users to later change their consent decision best meets several design objectives.

COVID-19 exposure-notification apps have struggled to gain adoption. Existing literature posits as potential causes of this low adoption: privacy concerns, insufficient data transparency, and the type of appeal – collective- vs. individual-good – used to frame the app. As policy guidance suggests using tailored advertising to evaluate the effects of these factors, we present the first field study of COVID-19 contact tracing apps with a randomized, control trial of 14 different advertisements for CovidDefense, Louisiana’s COVID-19 exposure-notification app. We find that all three hypothesized factors – privacy, data transparency, and appeals framing – relate to app adoption, even when controlling for age, gender, and community density. Our results offer (1) the first field evidence supporting the use of collective-good appeals, (2) nuanced findings regarding the efficacy of data and privacy transparency, the effects of which are moderated by appeal framing and potential users’ demographics, and (3) field-evidence-based guidance for future efforts to encourage pro-social health technology adoption.

User adoption of security and privacy (S&P) best practices remains low, despite sustained efforts by researchers and practitioners. Social influence is a proven method for guiding user S&P behavior, though most work has focused on studying peer influence, which is only possible with a known social graph. In a study of 104 Facebook users, we instead demonstrate that crowdsourced S&P suggestions are significantly influential. We also tested how reflective writing affected participants’ S&P decisions, with and without suggestions. With reflective writing, participants were less likely to accept suggestions — both social and Facebook default suggestions. Of particular note, when reflective writing participants were shown the Facebook default suggestion, they not only rejected it but also (unknowingly) configured their settings in accordance with expert recommendations. Our work suggests that both non-personal social influence and reflective writing can positively influence users’ S&P decisions, but have negative interactions.

Too often, while interacting with online technologies, we blindly agree to services’ terms and conditions (T&Cs). We often disregard their content—believing it is not worth engaging with the long, hard-to-understand texts. The inconspicuous display of online T&Cs on the user interface (UI) adds to our lack of engagement. Nevertheless, certain information included in T&Cs could help us make optimal decisions. In this replication research, we investigate this issue in the purchasing context. We confirm and extend previous findings through an online experiment (N = 987), showing that differently presented T&Cs (icons, scroll, and cost-cue) compared to hyperlinked text affect whether people open them, becoming aware. We also show the effect of decision-making style on the relationship between awareness and satisfaction. We discuss the possible use of these findings to improve users’ informed decisions. We also highlight problems that different designs may pose, potentially increasing the information gap between users and service providers.

How do people form impressions of effect size when reading scientific results? We present a series of studies on how people perceive treatment effectiveness when scientific results are summarized in various ways. We first show that a prevalent form of summarizing results—presenting mean differences between conditions—can lead to significant overestimation of treatment effectiveness, and that including confidence intervals can exacerbate the problem. We attempt to remedy potential misperceptions by displaying information about variability in individual outcomes in different formats: statements about variance, a quantitative measure of standardized effect size, and analogies that compare the treatment with more familiar effects (e.g., height differences by age). We find that all of these formats substantially reduce potential misperceptions and that analogies can be as helpful as more precise quantitative statements of standardized effect size. These findings can be applied by scientists in HCI and beyond to improve the communication of results to laypeople.

Recent work in HCI suggests that users can be powerful in surfacing harmful algorithmic behaviors that formal auditing approaches fail to detect. However, it is not well understood how users are often able to be so effective, nor how we might support more effective user-driven auditing. To investigate, we conducted a series of think-aloud interviews, diary studies, and workshops, exploring how users find and make sense of harmful behaviors in algorithmic systems, both individually and collectively. Based on our findings, we present a process model capturing the dynamics of and influences on users’ search and sensemaking behaviors. We find that 1) users’ search strategies and interpretations are heavily guided by their personal experiences with and exposures to societal bias; and 2) collective sensemaking amongst multiple users is invaluable in user-driven algorithm audits. We offer directions for the design of future methods and tools that can better support user-driven auditing.

Future offices are likely reshaped by Augmented Reality (AR) extending the display space while maintaining awareness of surroundings, and thus promise to support collaborative tasks such as brainstorming or sensemaking. However, it is unclear how physical surroundings and co-located collaboration influence the spatial organization of virtual content for sensemaking. Therefore, we conducted a study (N=28) to investigate the effect of office environments and work styles during a document classification task using AR with regard to content placement, layout strategies, and sensemaking workflows. Results show that participants require furniture, especially tables and whiteboards, to assist sensemaking and collaboration regardless of room settings, while generous free spaces (e.g., walls) are likely used when available. Moreover, collaborating participants tend to use furniture despite personal layout preferences. We identified different placement and layout strategies, as well as the transitions in-between. Finally, we propose design implications for future immersive sensemaking applications and beyond.

Information extraction (IE) approaches often play a pivotal role in text analysis and require significant human intervention. Therefore, a deeper understanding of existing IE practices and related challenges from a human-in-the-loop perspective is warranted. In this work, we conducted semi-structured interviews in an industrial environment and analyzed the reported IE approaches and limitations. We observed that data science workers often follow an iterative task model consisting of information foraging and sensemaking loops across all the phases of an IE workflow. The task model is generalizable and captures diverse goals across these phases (e.g., data preparation, modeling, evaluation.) We found several limitations in both foraging (e.g., data exploration) and sensemaking (e.g., qualitative debugging) loops stemming from a lack of adherence to existing cognitive engineering principles. Moreover, we identified that due to the iterative nature of an IE workflow, the requirement of provenance is often implied but rarely supported by existing systems. Based on these findings, we discuss design implications for supporting IE workflows and future research directions.

Telephone-driven community forums have been a widely proposed solution to address the unreliable internet connectivity and large geographic scope that characterizes many international NGO contexts. Primarily, these applications support asynchronous activities, such as information portals or forums to access rural journalism, but opportunities for real-time experience sharing remain largely under-explored. In this paper, we explore the potential of such forums to support remote mentoring of NGO volunteers, a practice that requires synchronous, dialogical formats for experience sharing and peer discussion. We engaged 28 participants from a rural Indian NGO in the design of peer-mentoring sessions that leverage synchronous audio discussions, using the structure and format of traditional talk-show radio as a starting point. The participants favored an entertaining approach to mentoring and discussed the logistics required to achieve this within their resource constraints. We conclude with design implications for designing media-driven community engagement platforms and the ethical challenges around protecting marginalized community interests.

The rise of consumer augmented reality (AR) technology has opened up new possibilities for interventions intended to disrupt and subvert cultural conventions. From defacing corporate logos to erecting geofenced digital monuments, more and more people are creating AR experiences for social causes. We sought to understand this new form of activism, including why people use AR for these purposes, opportunities and challenges in using it, and how well it can support activist goals. We conducted semi-structured interviews with twenty people involved in projects that used AR for a social cause across six different countries. We found that AR can overcome physical world limitations of activism to convey immersive, multilayered narratives that aim to reveal invisible histories and perspectives. At the same time, people experienced challenges in creating, maintaining, and distributing their AR experiences to audiences. We discuss open questions and opportunities for creating AR tools and experiences for social change.

Research on social robots in care has often focused on either the care recipients or the technology itself, neglecting the care workers who, in and through their collaborative and coordinative practices, will need to work with the robots. To better understand these interactions with a social robot (Pepper), we undertook a 3 month long-term study within a care home to gain empirical insights into the way the robot was used. We observed how care workers learned to use the device, applied it to their daily work life, and encountered obstacles. Our findings show that the care workers used the robot regularly (1:07 hours/day) mostly in one-to-one interactions with residents. While the robot had a limited effect on reducing the workload of care workers, it had other positive effects, demonstrating the potential to enhance the quality of care.