CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Speech recognition is unreliable in noisy places, compromises privacy and security when around strangers, and inaccessible to people with speech disorders. Lip reading can mitigate many of these challenges but the existing silent speech recognizers for lip reading are error prone. Developing new recognizers and acquiring new datasets is impractical for many since it requires enormous amount of time, effort, and other resources. To address these, first, we develop LipType, an optimized version of LipNet for improved speed and accuracy. We then develop an independent repair model that processes video input for poor lighting conditions, when applicable, and corrects potential errors in output for increased accuracy. We then test this model with LipType and other speech and silent speech recognizers to demonstrate its effectiveness.

We propose Auth+Track, a novel authentication model that aims to reduce redundant authentication in everyday smartphone usage. By sparse authentication and continuous tracking of the user’s status, Auth+Track eliminates the “gap” authentication between fragmented sessions and enables “Authentication Free when User is Around”. To instantiate the Auth+Track model, we present PanoTrack, a prototype that integrates body and near field hand information for user tracking. We install a fisheye camera on the top of the phone to achieve a panoramic vision that can capture both user’s body and on-screen hands. Based on the captured video stream, we develop an algorithm to extract 1) features for user tracking, including body keypoints and their temporal and spatial association, near field hand status, and 2) features for user identity assignment. The results of our user studies validate the feasibility of PanoTrack and demonstrate that Auth+Track not only improves the authentication efficiency but also enhances user experiences with better usability.

We present ElectroRing, a wearable ring-based input device that reliably detects both onset and release of a subtle finger pinch, and more generally, contact of the fingertip with the user’s skin. ElectroRing addresses a common problem in ubiquitous touch interfaces, where subtle touch gestures with little movement or force are not detected by a wearable camera or IMU. ElectroRing’s active electrical sensing approach provides a step-function-like change in the raw signal, for both touch and release events, which can be easily detected using only basic signal processing techniques. Notably, ElectroRing requires no second point of instrumentation, but only the ring itself, which sets it apart from existing electrical touch detection methods. We built three demo applications to highlight the effectiveness of our approach when combined with a simple IMU-based 2D tracking system.

We present Project Tasca, a pocket-based textile sensor that detects user input and recognizes everyday objects that a user carries in the pockets of a pair of pants (e.g., keys, coins, electronic devices, or plastic items). By creating a new fabric-based sensor capable of detecting in-pocket touch and pressure, and recognizing metallic, non-metallic, and tagged objects inside the pocket, we enable a rich variety of subtle, eyes-free, and always-available input, as well as context-driven interactions in wearable scenarios. We developed our prototype by integrating four distinct types of sensing methods, namely: inductive sensing, capacitive sensing, resistive sensing, and NFC in a multi-layer fabric structure into the form factor of a jeans pocket. Through a ten-participant study, we evaluated the performance of our prototype across 11 common objects including hands, 8 force gestures, and 30 NFC tag placements. We yielded a 92.3% personal cross-validation accuracy for object recognition, 96.4% accuracy for gesture recognition, and a 100% accuracy for detecting NFC tags at close distance . We conclude by demonstrating the interactions enabled by our pocket-based sensor in several applications.

Gestures are a promising candidate as an input modality for ambient computing where conventional input modalities such as touchscreens are not available. Existing works have focused on gesture recognition using image sensors. However, their cost, high battery consumption, and privacy concerns made cameras challenging as an always-on solution. This paper introduces an efficient gesture recognition technique using a miniaturized 60 GHz radar sensor. The technique recognizes four directional swipes and an omni-swipe using a radar chip (6.5 × 5.0 mm) integrated into a mobile phone. We developed a convolutional neural network model efficient enough for battery powered and computationally constrained processors. Its model size and inference time is less than 1/5000 compared to an existing gesture recognition technique using radar. Our evaluations with large scale datasets consisting of 558,000 gesture samples and 3,920,000 negative samples demonstrated our algorithm’s efficiency, robustness, and readiness to be deployed outside of research laboratories.

We propose a novel modality for active biometric authentication: electrical muscle stimulation (EMS). To explore this, we engineered an interactive system, which we call ElectricAuth, that stimulates the user’s forearm muscles with a sequence of electrical impulses (i.e., EMS challenge) and measures the user’s involuntary finger movements (i.e., response to the challenge). ElectricAuth leverages EMS’s intersubject variability, where the same electrical stimulation results in different movements in different users because everybody’s physiology is unique (e.g., differences in bone and muscular structure, skin resistance and composition, etc.). As such, ElectricAuth allows users to login without memorizing passwords or PINs.

ElectricAuth’s challenge-response structure makes it secure against data breaches and replay attacks, a major vulnerability facing today’s biometrics such as facial recognition and fingerprints. Furthermore, ElectricAuth never reuses the same challenge twice in authentications – in just one second of stimulation it encodes one of 68M possible challenges. In our user studies, we found that ElectricAuth resists: (1) impersonation attacks (false acceptance rate: 0.17% at 5% false rejection rate); (2) replay attacks (false acceptance rate: 0.00% at 5% false rejection rate); and, (3) synthesis attacks (false acceptance rates: 0.2-2.5%). Our longitudinal study also shows that ElectricAuth produces consistent results over time and across different humidity and muscle conditions.

We present BackTrack, a trackpad placed on the back of a smartphone to track fine-grained finger motions. Our system has a small form factor, with all the circuits encapsulated in a thin layer attached to a phone case. It can be used with any off-the-shelf smartphone, requiring no power supply or modification of the operating systems. BackTrack simply extends the finger tracking area of the front screen, without interrupting the use of the front screen. It also provides a switch to prevent unintentional touch on the trackpad. All these features are enabled by a battery-free capacitive circuit, part of which is a transparent, thin-film conductor coated on a thin glass and attached to the front screen. To ensure accurate and robust tracking, the capacitive circuits are carefully designed. Our design is based on a circuit model of capacitive touchscreens, justified through both physics-based finite-element simulation and controlled laboratory experiments. We conduct user studies to evaluate the performance of using BackTrack. We also demonstrate its use in a number of smartphone applications.

Wake-up-free techniques (e.g., Raise-to-Speak) are important for improving the voice input experience. We present ProxiMic, a close-to-mic (within 5 cm) speech sensing technique using only one microphone. With ProxiMic, a user keeps a microphone-embedded device close to the mouth and speaks directly to the device without wake-up phrases or button presses. To detect close-to-mic speech, we use the feature from pop noise observed when a user speaks and blows air onto the microphone. Sound input is first passed through a low-pass adaptive threshold filter, then analyzed by a CNN which detects subtle close-to-mic features (mainly pop noise). Our two-stage algorithm can achieve 94.1% activation recall, 12.3 False Accepts per Week per User (FAWU) with 68 KB memory size, which can run at 352 fps on the smartphone. The user study shows that ProxiMic is efficient, user-friendly, and practical.

We present Pose-on-the-Go, a full-body pose estimation system that uses sensors already found in today’s smartphones. This stands in contrast to prior systems, which require worn or external sensors. We achieve this result via extensive sensor fusion, leveraging a phone’s front and rear cameras, the user-facing depth camera, touchscreen, and IMU. Even still, we are missing data about a user’s body (e.g., angle of the elbow joint), and so we use inverse kinematics to estimate and animate probable body poses. We provide a detailed evaluation of our system, benchmarking it against a professional-grade Vicon tracking system. We conclude with a series of demonstration applications that underscore the unique potential of our approach, which could be enabled on many modern smartphones with a simple software update.

We present FaceSight, a computer vision-based hand-to-face gesture sensing technique for AR glasses. FaceSight fixes an infrared camera onto the bridge of AR glasses to provide extra sensing capability of the lower face and hand behaviors. We obtained 21 hand-to-face gestures and demonstrated the potential interaction benefits through five AR applications. We designed and implemented an algorithm pipeline that segments facial regions, detects hand-face contact (f1 score: 98.36%), and trains convolutional neural network (CNN) models to classify the hand-to-face gestures. The input features include gesture recognition, nose deformation estimation, and continuous fingertip movement. Our algorithm achieves classification accuracy of all gestures at 83.06%, proved by the data of 10 users. Due to the compact form factor and rich gestures, we recognize FaceSight as a practical solution to augment input capability of AR glasses in the future.

Handheld controllers are an essential part of VR systems. Modern sensing techniques enable them to track users’ finger movements to support natural interaction using hands. The sensing techniques, however, often fail to precisely determine whether two fingertips touch each other, which is important for the robust detection of a pinch gesture. To address this problem, we propose AtaTouch, which is a novel, robust sensing technique for detecting the closure of a finger pinch. It utilizes a change in the coupled impedance of an antenna and human fingers when the thumb and finger form a loop. We implemented a prototype controller in which AtaTouch detects the finger pinch of the grabbing hand. A user test with the prototype showed a finger-touch detection accuracy of 96.4%. Another user test with the scenarios of moving virtual blocks demonstrated low object-drop rate (2.75%) and false-pinch rate (4.40%). The results and feedback from the participants support the robustness and sensitivity of AtaTouch.

Capacitive touchscreens are near-ubiquitous in today’s touch-driven devices, such as smartphones and tablets. By using rows and columns of electrodes, specialized touch controllers are able to capture a 2D image of capacitance at the surface of a screen. For over a decade, capacitive “pixels” have been around 4 millimeters in size – a surprisingly low resolution that precludes a wide range of interesting applications. In this paper, we show how super-resolution techniques, long used in fields such as biology and astronomy, can be applied to capacitive touchscreen data. By integrating data from many frames, our software-only process is able to resolve geometric details finer than the original sensor resolution. This opens the door to passive tangibles with higher-density fiducials and also recognition of every-day metal objects, such as keys and coins. We built several applications to illustrate the potential of our approach and report the findings of a multipart evaluation.

In this paper, we propose a reconfigurable electrode, RElectrode, using a microfluidic technique that can change the geometry and material properties of the electrode to satisfy the needs for sensing a variety of different types of user input through touch/touchless gestures, pressure, temperature, and distinguish between different types of objects or liquids. Unlike the existing approaches, which depend on the specific-shaped electrode for particular sensing (e.g., coil for inductive sensing), RElectrode enables capacity, inductance, resistance/pressure, temperature, pH sensings all in a single package. We demonstrate the design and fabrication of the microfluidic structure of our RElectrode, evaluate its sensing performance through several studies, and provide some unique applications. RElectrode demonstrates technical feasibility and application values of integrating physical and biochemical properties of microfluidics into novel sensing interfaces.

We contribute a method to automate parameter configurations for chart layouts by learning from human preferences. Existing charting tools usually determine the layout parameters using predefined heuristics, producing sub-optimal layouts. People can repeatedly adjust multiple parameters (e.g., chart size, gap) to achieve visually appealing layouts. However, this trial-and-error process is unsystematic and time-consuming, without a guarantee of improvement. To address this issue, we develop Layout Quality Quantifier (LQ2), a machine learning model that learns to score chart layouts from paired crowdsourcing data. Combined with optimization techniques, LQ2 recommends layout parameters that improve the charts’ layout quality. We apply LQ2 on bar charts and conduct user studies to evaluate its effectiveness by examining the quality of layouts it produces. Results show that LQ2 can generate more visually appealing layouts than both laypeople and baselines. This work demonstrates the feasibility and usages of quantifying human preferences and aesthetics for chart layouts.

Animation helps viewers follow transitions in data graphics. When authoring animations that incorporate data, designers must carefully coordinate the behaviors of visual objects such as entering, exiting, merging and splitting, and specify the temporal rhythms of transition through staging and staggering. We present Data Animator, a system for authoring animated data graphics without programming. Data Animator leverages the Data Illustrator framework to analyze and match objects between two static visualizations, and generates automated transitions by default. Designers have the flexibility to interpret and adjust the matching results through a visual interface. Data Animator also supports the division of a complex animation into stages through hierarchical keyframes, and uses data attributes to stagger the start time and vary the speed of animating objects through a novel timeline interface. We validate Data Animator’s expressiveness via a gallery of examples, and evaluate its usability in a re-creation study with designers.

Data-driven articles—i.e., articles featuring text and supporting charts—play a key role in communicating information to the public. New storytelling formats like scrollytelling apply compelling dynamics to these articles to help walk readers through complex insights, but are challenging to craft. In this work, we investigate ways to support authors of data-driven articles using such storytelling forms via a text-chart linking strategy. From formative interviews with 6 authors and an assessment of 43 scrollytelling stories, we built VizFlow, a prototype system that uses text-chart links to support a range of dynamic layouts. We validate our text-chart linking approach via an authoring study with 12 participants using VizFlow, and a reading study with 24 participants comparing versions of the same article with different VizFlow intervention levels. Assessments showed our approach enabled a rapid and expressive authoring experience, and informed key design recommendations for future efforts in the space.

Interfaces for creating visualizations typically embrace one of several common forms. Textual specification enables fine-grained control, shelf building facilitates rapid exploration, while chart choosing promotes immediacy and simplicity. Ideally these approaches could be unified to integrate the user- and usage-dependent benefits found in each modality, yet these forms remain distinct.

We propose parameterized declarative templates, a simple abstraction mechanism over JSON-based visualization grammars, as a foundation for multimodal visualization editors. We demonstrate how templates can facilitate organization and reuse by factoring the more than 160 charts that constitute Vega-Lite’s example gallery into approximately 40 templates. We exemplify the pliability of abstracting over charting grammars by implementing—as a template—the functionality of the shelf builder Polestar (a simulacra of Tableau) and a set of templates that emulate the Google Sheets chart chooser. We show how templates support multimodal visualization editing by implementing a prototype and evaluating it through an approachability study.

Interactive visualizations are widely used in exploratory data analysis, but existing systems provide limited support for confirmatory analysis. We introduce PredictMe, a tool for belief-driven visual analysis, enabling users to draw and test their beliefs against data, as an alternative to data-driven exploration. PredictMe combines belief elicitation with traditional visualization interactions to support mixed analysis styles. In a comparative study, we investigated how these affordances impact participants’ cognition. Results show that PredictMe prompts participants to incorporate their working knowledge more frequently in queries. Participants were more likely to attend to discrepancies between their mental models and the data. However, those same participants were also less likely to engage in interactions associated with exploration, and ultimately inspected fewer visualizations and made fewer discoveries. The results suggest that belief elicitation may moderate exploratory behaviors, instead nudging users to be more deliberate in their analysis. We discuss the implications for visualization design.

Current text visualization techniques typically provide overviews of document content and structure using intrinsic properties such as term frequencies, co-occurrences, and sentence structures. Such visualizations lack conceptual overviews incorporating domain-relevant knowledge, needed when examining documents such as research articles or technical reports. To address this shortcoming, we present ConceptScope, a technique that utilizes a domain ontology to represent the conceptual relationships in a document in the form of a Bubble Treemap visualization. Multiple coordinated views of document structure and concept hierarchy with text overviews further aid document analysis. ConceptScope facilitates exploration and comparison of single and multiple documents respectively. We demonstrate ConceptScope by visualizing research articles and transcripts of technical presentations in computer science. In a comparative study with DocuBurst, a popular document visualization tool, ConceptScope was found to be more informative in exploring and comparing domain-specific documents, but less so when it came to documents that spanned multiple disciplines.

Researchers have investigated a number of strategies for capturing and analyzing data analyst event logs in order to design better tools, identify failure points, and guide users. However, this remains challenging because individual- and session-level behavioral differences lead to an explosion of complexity and there are few guarantees that log observations map to user cognition. In this paper we introduce a technique for segmenting sequential analyst event logs which combines data, interaction, and user features in order to create discrete blocks of goal-directed activity. Using measures of inter-dependency and comparisons between analysis states, these blocks identify patterns in interaction logs coupled with the current view that users are examining. Through an analysis of publicly available data and data from a lab study across a variety of analysis tasks, we validate that our segmentation approach aligns with users’ changing goals and tasks. Finally, we identify several downstream applications for our approach.

Visual analytics systems enable highly interactive exploratory data analysis. Across a range of fields, these technologies have been successfully employed to help users learn from complex data. However, these same exploratory visualization techniques make it easy for users to discover spurious findings. This paper proposes new methods to monitor a user’s analytic focus during visual analysis of structured datasets and use it to surface relevant articles that contextualize the visualized findings. Motivated by interactive analyses of electronic health data, this paper introduces a formal model of analytic focus, a computational approach to dynamically update the focus model at the time of user interaction, and a prototype application that leverages this model to surface relevant medical publications to users during visual analysis of a large corpus of medical records. Evaluation results with 24 users show that the modeling approach has high levels of accuracy and is able to surface highly relevant medical abstracts.

Course selection is a crucial activity for students as it directly impacts their workload and performance. It is also time-consuming, prone to subjectivity, and often carried out based on incomplete information. This task can, nevertheless, be assisted with computational tools, for instance, by predicting performance based on historical data. We investigate the effects of showing grade predictions to students through an interactive visualization tool. A qualitative study suggests that in the presence of predictions, students may focus too much on maximizing their performance, to the detriment of other factors such as the workload. A follow-up quantitative study explored whether these effects are mitigated by changing how predictions are conveyed. Our observations suggest the presence of a framing effect that induces students to put more effort into course selection when faced with more specific predictions. We discuss these and other findings and outline considerations for designing better data-driven course selection tools.

Time-series forecasting contributes crucial information to industrial and institutional decision-making with multivariate time-series input. Although various models have been developed to facilitate the forecasting process, they make inconsistent forecasts. Thus, it is critical to select the model appropriately. The existing selection methods based on the error measures fail to reveal deep insights into the model’s performance, such as the identification of salient features and the impact of temporal factors (e.g., periods). This paper introduces mTSeer, an interactive system for the exploration, explanation, and evaluation of multivariate time-series forecasting models. Our system integrates a set of algorithms to steer the process, and rich interactions and visualization designs to help interpret the differences between models in both model and instance level. We demonstrate the effectiveness of mTSeer through three case studies with two domain experts on real-world data, qualitative interviews with the two experts, and quantitative evaluation of the three case studies.

We present CAST, an authoring tool that enables the interactive creation of chart animations. It introduces the visual specification of chart animations consisting of keyframes that can be played sequentially or simultaneously, and animation parameters (e.g., duration, delay). Building on Canis [19], a declarative chart animation grammar that leverages data-enriched SVG charts, CAST supports auto-completion for constructing both keyframes and keyframe sequences. It also enables users to refine the animation specification (e.g., aligning keyframes across tracks to play them together, adjusting delay) with direct manipulation and other parameters for animation effects (e.g., animation type, easing function) using a control panel. In addition to describing how CAST infers recommendations for auto-completion, we present a gallery of examples to demonstrate the expressiveness of CAST and a user study to verify its learnability and usability. Finally, we discuss the limitations and potentials of CAST as well as directions for future research.

Quantifying user performance with metrics such as time and accuracy does not show the whole picture when researchers evaluate complex, interactive visualization tools. In such systems, performance is often influenced by different analysis strategies that statistical analysis methods cannot account for. To remedy this lack of nuance, we propose a novel analysis methodology for evaluating complex interactive visualizations at scale. We implement our analysis methods in reVISit, which enables analysts to explore participant interaction performance metrics and responses in the context of users’ analysis strategies. Replays of participant sessions can aid in identifying usability problems during pilot studies and make individual analysis processes salient. To demonstrate the applicability of reVISit to visualization studies, we analyze participant data from two published crowdsourced studies. Our findings show that reVISit can be used to reveal and describe novel interaction patterns, to analyze performance differences between different analysis strategies, and to validate or challenge design decisions.

Most of the existing scientific visualizations toward interpretive grammar aim to enhance customizability in either the computation stage or the rendering stage or both, while few approaches focus on the data presentation stage. Besides, most of these approaches leverage the existing components from the general-purpose programming languages (GPLs) instead of developing a standalone compiler, which pose a great challenge about learning curves for the domain experts who have limited knowledge about programming. In this paper, we propose IGScript, a novel script-based interaction grammar tool, to help build scientific data presentation animations for communication. We design a dual-space interface and a compiler which converts natural language-like grammar statements or scripts into a data story animation to make an interactive customization on script-driven data presentations, and then develop a code generator (decompiler) to translate the interactive data exploration animations back into script codes to achieve statement parameters. IGScript makes the presentation animations editable, e.g., it allows to cut, copy, paste, append, or even delete some animation clips. We demonstrate the usability, customizability, and flexibility of IGScript by a user study, four case studies conducted by using four types of commonly-used scientific data, and performance evaluations.

Raised-line graphics are tactile documents made for people with visual impairments (VI). Their exploration relies on a complex two-handed behavior. To better understand the cognitive processes underlying this exploration, we proposed a new method based on “tactile fixations”. A tactile fixation occurs when a finger is stationary within a specific spatial and temporal window. It is known that stationary fingers play an active role when exploring tactile graphics, but they have never been defined or studied before. In this study, we first defined the concept of tactile fixation, then we conducted a behavioral study with ten participants with VI in order to assess the role of tactile fixations under different conditions. The results show that tactile fixations vary according to different factors such as the graphic type as well as the involved hand and the aim of the exploration.

Accurate and effective directional feedback is crucial for an electronic traveling aid device that guides visually impaired people in walking through paths. This paper presents Tactile Compass, a hand-held device that provides continuous directional feedback with a rotatable needle pointing toward the planned direction. We conducted two lab studies to evaluate the effectiveness of the feedback solution. Results showed that, using Tactile Compass, participants could reach the target direction in place with a mean deviation of 3.03° and could smoothly navigate along paths of 60cm width, with a mean deviation from the centerline of 12.1cm. Subjective feedback showed that Tactile Compass was easy to learn and use.

Users with visual impairments find it difficult to enjoy real-time 2D interactive applications on the touchscreen. Touchscreen applications such as sports games often require simultaneous recognition of and interaction with multiple moving targets through vision. To mitigate this issue, we propose ThroughHand, a novel tactile interaction that enables users with visual impairments to interact with multiple dynamic objects in real time. We designed the ThroughHand interaction to utilize the potential of the human tactile sense that spatially registers both sides of the hand with respect to each other. ThroughHand allows interaction with multiple objects by enabling users to perceive the objects using the palm while providing a touch input space on the back of the same hand. A user study verified that ThroughHand enables users to locate stimuli on the palm with a margin of error of approximately 13 mm and effectively provides a real-time 2D interaction experience for users with visual impairments.

Teachers of the Visually Impaired (TVIs) teach academic and functional living skills simultaneously to prepare students with vision impairment to be successful and independent. Current educational tools primarily focus on academic instruction rather than this multifaceted approach needed for students. Our work aims to understand how technology can integrate behavioral skills, like independence, and support TVIs in their preferred teaching strategy. We observed elementary classrooms at a school for the blind for six weeks to study how educators design lessons and use technology to supplement their instruction in different subjects. After the observational study, we conducted remote interviews with educators to understand how technology can support students in building academic and behavioral skills in-person and remotely. Educators suggested incorporating audio feedback that motivates students to play and learn consistently, student progress tracking for parents and educators, and designing features that help students build independence and develop collaborative skills.

Visually impaired children (VI) face challenges in collaborative learning in classrooms. Robots have the potential to support inclusive classroom experiences by leveraging their physicality, bespoke social behaviors, sensors, and multimodal feedback. However, the design of social robots for mixed-visual abilities classrooms remains mostly unexplored. This paper presents a four-month-long community-based design process where we engaged with a school community. We provide insights into the barriers experienced by children and how social robots can address them. We also report on a participatory design activity with mixed-visual abilities children, highlighting the expected roles, attitudes, and physical characteristics of robots. Findings contextualize social robots within inclusive classroom settings as a holistic solution that can interact anywhere when needed and suggest a broader view of inclusion beyond disability. These include children’s personality traits, technology access, and mastery of school subjects. We finish by providing reflections on the community-based design process.

Learning to write is challenging for blind and low vision (BLV) people because of the lack of visual feedback. Regardless of the drastic advancement of digital technology, handwriting is still an essential part of daily life. Although tools designed for teaching BLV to write exist, many are expensive and require the help of sighted teachers. We propose LightWrite, a low-cost, easy-to-access smartphone application that uses voice-based descriptive instruction and feedback to teach BLV users to write English lowercase letters and Arabian digits in a specifically designed font. A two-stage study with 15 BLV users with little prior writing knowledge shows that LightWrite can successfully teach users to learn handwriting characters in an average of 1.09 minutes for each letter. After initial training and 20-minute daily practice for 5 days, participants were able to write an average of 19.9 out of 26 letters that are recognizable by sighted raters.

Standing in line is one of the most common social behaviors in public spaces but can be challenging for blind people. We propose an assistive system named LineChaser, which navigates a blind user to the end of a line and continuously reports the distance and direction to the last person in the line so that they can be followed. LineChaser uses the RGB camera in a smartphone to detect nearby pedestrians, and the built-in infrared depth sensor to estimate their position. Via pedestrian position estimations, LineChaser determines whether nearby pedestrians are standing in line, and uses audio and vibration signals to notify the user when they should start/stop moving forward. In this way, users can stay correctly positioned while maintaining social distance. We have conducted a usability study with 12 blind participants. LineChaser allowed blind participants to successfully navigate lines, significantly increasing their confidence in standing in lines.

People with vision impairments access smartphones with the help of screen reader apps such as TalkBack for Android and VoiceOver for iPhone. Prior research has mostly focused on understanding touchscreen phone adoption and typing performance of novice blind users by logging their real-world smartphone usage. Understanding smartphone usage pattern and practices of expert users can help in developing tools and tutorials for transitioning novice and intermediate users to expert users. In this work, we logged smartphone usage data of eight expert Android smartphone users with visual impairments for four weeks, and then interviewed them. This paper presents a detailed analysis that uncovered novel usage patterns, such as extensive usage of directional gestures, reliance on voice and external keyboard for text input, and repurposed explore by touch for single-tap. We conclude with design recommendations to inform the future of mobile accessibility, including hardware guidelines and rethinking accessible software design.

Visual semantics provide spatial information like size, shape, and position, which are necessary to understand and efficiently use interfaces and documents. Yet little is known about whether blind and low-vision (BLV) technology users want to interact with visual affordances, and, if so, for which task scenarios. In this work, through semi-structured and task-based interviews, we explore preferences, interest levels, and use of visual semantics among BLV technology users across two device platforms (smartphones and laptops), and information seeking and interactions common in apps and web browsing. Findings show that participants could benefit from access to visual semantics for collaboration, navigation, and design. To learn this information, our participants used trial and error, sighted assistance, and features in existing screen reading technology like touch exploration. Finally, we found that missing information and inconsistent screen reader representations of user interfaces hinder learning. We discuss potential applications and future work to equip BLV users with necessary information to engage with visual semantics.

Tactile guiding surfaces in the built environment have held a contentious place in the process of navigation by people who are blind or visually impaired. Despite standards for tactile guiding surfaces, problems persist with inconsistent implementation, perception, and geographic orientation. We investigate the role of tactile cues in non-visual navigation and attitudes surrounding guiding surfaces through a survey of 67 people with vision impairments and ten interviews with navigation and public accessibility experts. Our participants revealed several opportunities to augment existing tactile surfaces while envisioning novel multimodal feedback solutions in immediately relevant contexts. We also propose an approach for designing and exploring low cost, multimodal tactile surfaces, which we call navtiles. Finally, we discuss practical aspects of implementation for new design alternatives such as standardization, installation, movability, discoverability, and a need for transparency. Collectively, these insights contribute to the production and implementation of novel multimodal navigation aids.

Keyboard-based emoji entry can be challenging for people with visual impairments: users have to sequentially navigate emoji lists using screen readers to find their desired emojis, which is a slow and tedious process. In this work, we explore the design and benefits of emoji entry with speech input, a popular text entry method among people with visual impairments. After conducting interviews to understand blind or low vision (BLV) users’ current emoji input experiences, we developed Voicemoji, which (1) outputs relevant emojis in response to voice commands, and (2) provides context-sensitive emoji suggestions through speech output. We also conducted a multi-stage evaluation study with six BLV participants from the United States and six BLV participants from China, finding that Voicemoji significantly reduced entry time by 91.2% and was preferred by all participants over the Apple iOS keyboard. Based on our findings, we present Voicemoji as a feasible solution for voice-based emoji entry.

While the growth of financial technologies (FinTech) is making the flow of money faster, easier, and more secure, such technologies are often unable to serve many countries due to the global political environment. Despite its severe impact, this issue has remained understudied in the HCI literature. We address this gap by presenting our findings from a three-month-long ethnography with the Iranian community in Toronto, Canada. We present their struggles in transferring money to and from their home country - a process that entails financial loss, fear, uncertainty, and privacy breaches. We also outline the informal workarounds that allow this community to circumvent these challenges, along with the associated hassles. This paper contributes to broadening the scope of FinTech in the HCI literature by connecting it with the politics surrounding transnational transactions. We discuss the design implications of our findings and their contribution to the broader interests of HCI in mobilities and social justice.

AI models are increasingly applied in high-stakes domains like health and conservation. Data quality carries an elevated significance in high-stakes AI due to its heightened downstream impact, impacting predictions like cancer detection, wildlife poaching, and loan allocations. Paradoxically, data is the most under-valued and de-glamorised aspect of AI. In this paper, we report on data practices in high-stakes AI, from interviews with 53 AI practitioners in India, East and West African countries, and USA. We define, identify, and present empirical evidence on Data Cascades—compounding events causing negative, downstream effects from data issues—triggered by conventional AI/ML practices that undervalue data quality. Data cascades are pervasive (92% prevalence), invisible, delayed, but often avoidable. We discuss HCI opportunities in designing and incentivizing data excellence as a first-class citizen of AI, resulting in safer and more robust systems for all.

Head-worn displays (HWDs) offer their users high mobility, hands-free operation, and “see-what-I-see” features. In the prehospital environment, emergency medical services (EMS) staff could benefit from the unique characteristics of HWDs. We conducted a field study to analyze work practices of EMS staff and the potential of HWDs to support their activities. Based on our observations and the comments of EMS staff, we propose three use cases for HWDs in the prehospital environment. They are (1) enhanced communication between different care providers, (2) hands-free access to clinical monitoring and imaging, (3) and improved realism of training scenarios. We conclude with a set of design considerations and suggest that for the successful implementation of HWDs in EMS environments, researchers, designers, and clinical stakeholders should consider the harsh outdoor environment in which HWDs will be used, the extensive workload of staff, the complex collaboration performed, privacy requirements, and the high variability of work.

With a plethora of off-the-shelf smart home devices available commercially, people are increasingly taking a do-it-yourself approach to configuring their smart homes. While this allows for customization, users responsible for smart home configuration often end up with more control over the devices than other household members. This separates those who introduce new functionality to the smart home (pilot users) from those who do not (passenger users). To investigate the prevalence and impact of pilot-passenger user relationships, we conducted a Mechanical Turk survey and a series of one-hour interviews. Our results suggest that pilot-passenger relationships are common in multi-user households and shape how people form habits around devices. We find from interview data that smart homes reflect the values of their pilot users, making it harder for passenger users to incorporate their devices into daily life. We conclude the paper with design recommendations to improve passenger and pilot user experience.

Voice-based Conversational Agents (VCA) have served as personal assistants that support individuals with special needs. Adolescents with Autism Spectrum Disorder (ASD) may also benefit from VCAs to deal with their everyday needs and challenges, ranging from self-care to social communications. In this study, we explored how VCAs could encourage adolescents with ASD in navigating various aspects of their daily lives through the two-week use of VCAs and a series of participatory design workshops. Our findings demonstrated that VCAs could be an engaging, empowering, emancipating tool that supports adolescents with ASD to address their needs, personalities, and expectations, such as promoting self-care skills, regulating negative emotions, and practicing conversational skills. We propose implications of using off-the-shelf technologies as a personal assistant to ASD users in Assistive Technology design. We suggest design implications for promoting positive opportunities while mitigating the remaining challenges of VCAs for adolescents with ASD.

Most children who are blind live in low-resource settings and attend schools that have poor technical infrastructure, overburdened teachers, and outdated curriculum. Our work explores the role digital games can play to develop digital skills of such children in the early grades. Recognizing the critical role of teachers in introducing children to technology, we conducted a mixed-methods study to examine which attributes of digital games teachers find useful for children and what challenges they perceive in integrating digital games in schools for the blind. Our findings indicate that teachers prefer games that align well with curriculum objectives, promote learning, improve soft skills, and increase engagement with computers. Despite being overburdened and lacking technological support, teachers expressed strong enthusiasm to integrate these games in school curriculum and schedule. We conclude by discussing design implications for designers of accessible games in low-resource settings.

Disability professionals could use digital fabrication tools to provide customised assistive technologies or accessible media beneficial to the education of Blind or visually impaired youth. However, there is little documentation of long-term practices with these tools by professionals in this field, limiting our ability to support their work. We report on such practices in a French organisation, providing disability educational services and using digital fabrication since 2013, for six years. We trace how professionals defined how digital fabrication could and should be used through a range of projects, based on pedagogical uses and the constraints in creation, production and maintenance. We outline new research perspectives going beyond 3D printers and its promises of automation to embrace hybrid approaches currently supported by laser cutters, the learning and documentation process, and the production of accessible tactile media at a regional or national scale.

Technology has become increasingly pervasive in the creative and experimental environment of the avant-garde fashion runway, particularly in relation to its garments. However, several disciplines are often necessary when exploring technologies for the construction of expressive garments (e.g. garments that respond to their environment), creating a barrier for fashion designers that has limited their ability to leverage new technologies. To help overcome this barrier, we designed and deployed Brookdale, a prototyping system for wearable technology consisting of new plug-and-play hardware that can be programmed using drag-and-drop software. Brookdale was created using a 24-week participatory design process with 17 novice fashion-tech designers. At the end of the 24 week process, designers showcased their Brookdale-enhanced garment collections at an avant-garde fashion-tech runway show in New York City. We report on the experiences, outcomes, and lessons learned throughout this process, and describe results from interviews with the fashion-tech designers 16 weeks after the fashion show, demonstrating the lasting positive impact of Brookdale.

The industry for children’s apps is thriving at the cost of children’s privacy: these apps routinely disclose children’s data to multiple data trackers and ad networks. As children spend increasing time online, such exposure accumulates to long-term privacy risks. In this paper, we used a mixed-methods approach to investigate why this is happening and how developers might change their practices. We base our analysis against 5 leading data protection frameworks that set out requirements and recommendations for data collection in children’s apps. To understand developers’ perspectives and constraints, we conducted 134 surveys and 20 semi-structured interviews with popular Android children’s app developers. Our analysis revealed that developers largely respect children’s best interests; however, they have to make compromises due to limited monetisation options, perceived harmlessness of certain third-party libraries, and lack of availability of design guidelines. We identified concrete approaches and directions for future research to help overcome these barriers.

Repeated exposure to poor air quality in indoor environments such as office, home, and classroom can have substantial adverse effects on our health and productivity. The problem is especially recognized in closed indoor spaces shared by several people. We have studied the evolution of carbon dioxide level in office-meeting spaces, during 1007 meeting sessions. The collected data is employed to examine machine learning models aimed to indicate the CO2 evolution pattern and to forecast when fresh air should be supplied. In addition, to gain insight into the relations and interdependencies of social factors in meetings that may influence the users’ perception of an interactive solution, we have conducted a series of online surveys. Building on the results of the two studies, a solution is proposed that predicts the evolution of air quality in naturally-ventilated meeting rooms and engages the users in preventive actions when risk is forecast.

It is well documented that people living with mental health conditions are more likely to experience financial difficulties. When explaining this association, emphasis has often been placed on financial capability, i.e. the capacity of those living with poor mental health to manage their money. This paper challenges such capability-based explanations by reporting on a diary study and interviews with 14 people who self-identify as living with a mental health condition. We focused on their experiences of financial technology use, and explored the role technology played in their strategies to minimise the impact of mental health on their economic circumstances. Rather than lacking capability, participants’ practices revealed shortcomings of existing financial technologies and how they sought to work around these. We conclude by providing a set of design directions for technologies that engage those living with poor mental health not as vulnerable targets for financial inclusion, but as full financial citizens.

Smart Donations is a blockchain-based platform that offers users ‘contracts’ that donate funds to certain causes in response to real-world events e.g., whenever an earthquake is detected or an activist tweets about refugees. We designed Smart donations with Oxfam Australia, trialled it for 8-weeks with 86 people, recorded platform analytics and qualitatively analysed questionnaires and interviews about user experiences. Temporal qualities emerge when automation enforces conditions that contributed to participants’ awareness of events that are usually unconscious, and senses of immediacy in contributing to crisis response and ongoing involvement in situations far-away while awaiting conditions to be met. We suggest data driven automation can reveal diverse temporal registers, in real-world phenomena, sociality, morality and everyday life, which contributes to experiencing a ‘right time’ to donate that is not limited to productivity or efficiency. Thus, we recommend a sensitivity to right time in designing for multiple temporalities in FinTech more generally.

Running is a widely popular physical activity that offers many health benefits. As runners progress with their training, understanding one’s own body becomes a key concern in achieving wellbeing through running. While extensive bodily sensing opportunities exist for runners, understanding complex sensor data is a challenge. In this paper, we investigate how data from shoe-worn sensors can be visualised to empower runners to improve their technique. We designed GraFeet—an augmented running shoe that visualises kinesiological data about the runner’s feet and gait. We compared our prototype with a standard sensor dashboard in a user study where users ran with the sensor and analysed the generated data after the run. GraFeet was perceived as more usable; producing more insights and less confusion in the users. Based on our inquiry, we contribute findings about using data from body-worn sensors to support physically active individuals.

Maximizing system scalability and quality are sometimes at odds. This work provides an example showing scalability and quality can be achieved at the same time in instructional design, contrary to what instructors may believe or expect. We situate our study in the education of HCI methods, and provide suggestions to improve active learning within the HCI education community. While designing learning and assessment activities, many instructors face the choice of using open-ended or close-ended activities. Close-ended activities such as multiple-choice questions (MCQs) enable automated feedback to students. However, a survey with 22 HCI professors revealed a belief that MCQs are less valuable than open-ended questions, and thus, using them entails making a quality sacrifice in order to achieve scalability. A study with 178 students produced no evidence to support the teacher belief. This paper indicates more promise than concern in using MCQs for scalable instruction and assessment in at least some HCI domains.

We designed and tested an attention-aware learning technology (AALT) that detects and responds to mind wandering (MW), a shift in attention from task-related to task-unrelated thoughts, that is negatively associated with learning. We leveraged an existing gaze-based mind wandering detector that uses commercial off the shelf eye tracking to inform real-time interventions during learning with an Intelligent Tutoring System in real-world classrooms. The intervention strategies, co-designed with students and teachers, consisted of using student names, reiterating content, and asking questions, with the aim to reengage wandering minds and improve learning. After several rounds of iterative refinement, we tested our AALT in two classroom studies with 287 high-school students. We found that interventions successfully reoriented attention, and compared to two control conditions, reduced mind wandering, and improved retention (measured via a delayed assessment) for students with low prior-knowledge who occasionally (but not excessively) mind wandered. We discuss implications for developing gaze-based AALTs for real-world contexts.

Learning to recognize and apply programming patterns — reusable abstractions of code — is critical to becoming a proficient computer scientist. However, many introductory Computer Science courses do not teach patterns, in part because teaching these concepts requires significant curriculum changes. As an alternative, we explore how a novel user interface for practicing coding — Faded Parsons Problems — can support introductory Computer Science students in learning to apply programming patterns. We ran a classroom-based study with 237 students which found that Faded Parsons Problems, or rearranging and completing partially blank lines of code into a valid program, are an effective exercise interface for teaching programming patterns, significantly surpassing the performance of the more standard approaches of code writing and code tracing exercises. Faded Parsons Problems also improve overall code writing ability at a comparable level to code writing exercises, but are preferred by students.

Computational thinking (CT) education reaches only a fraction of young children, in part because CT learning tools often require expensive hardware or fluent literacy. Informed by needfinding interviews, we developed a voice-guided smartphone application leveraging storytelling as a creative activity by which to teach CT concepts to 5- to 8-year-old children. The app includes two storytelling games where users create and listen to stories as well as four CT games where users then modify those stories to learn about sequences, loops, events, and variables. We improved upon the app design through wizard-of-oz testing (N = 28) and iterative design testing (N = 22) before conducting an evaluation study (N = 22). Children were successfully able to navigate the app, effectively learn about the target computing concepts, and, after using the app, children demonstrated above-chance performance on a near transfer CT concept recognition task.

Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that influence programming error message readability. In the first two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We find evidence that several key factors significantly affect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested long-standing guidelines on message design, and informs future efforts to create readability metrics for programming error messages.

Program tracing, or mentally simulating a program on concrete inputs, is an important part of general program comprehension. Programs involve many kinds of virtual state that must be held in memory, such as variable/value pairs and a call stack. In this work, we examine the influence of short-term working memory (WM) on a person’s ability to remember program state during tracing. We first confirm that previous findings in cognitive psychology transfer to the programming domain: people can keep about 7 variable/value pairs in WM, and people will accidentally swap associations between variables due to WM load. We use a restricted focus viewing interface to further analyze the strategies people use to trace through programs, and the relationship of tracing strategy to WM. Given a straight-line program, we find half of our participants traced a program from the top-down line-by-line (linearly), and the other half start at the bottom and trace upward based on data dependencies (on-demand). Participants with an on-demand strategy made more WM errors while tracing straight-line code than with a linear strategy, but the two strategies contained an equal number of WM errors when tracing code with functions. We conclude with the implications of these findings for the design of programming tools: first, programs should be analyzed to identify and refactor human-memory-intensive sections of code. Second, programming environments should interactively visualize variable metadata to reduce WM load in accordance with a person’s tracing strategy. Third, tools for program comprehension should enable externalizing program state while tracing.

HCI has historically provided little support for moving from fieldwork insights or theories to design outcomes. Having witnessed many students struggle and then justify their designs with a form of marketing hype, we developed a supporting approach of “field theories”. A field theory is a working theory about salient interactions in a particular domain and sensitizing concepts in order to frame design investigations. It is presented visually in a field theory diagram to support succinct communication and critique. Studying use of design prototypes that have been informed by a field theory helps to reflect upon and refine the theory. In this paper we present examples from our HCI classes and reflections based on interviews with students. We discuss how field theories offer an orientation in the spirit of a ‘bricoleur’ who harnesses elements of theory and practice to produce deeper understandings and more fitting outcomes for the task at hand.

We present the design and evaluation of a web-based intelligent writing assistant that helps students recognize their revisions of argumentative essays. To understand how our revision assistant can best support students, we have implemented four versions of our system with differences in the unit span (sentence versus sub-sentence) of revision analysis and the level of feedback provided (none, binary, or detailed revision purpose categorization). We first discuss the design decisions behind relevant components of the system, then analyze the efficacy of the different versions through a Wizard of Oz study with university students. Our results show that while a simple interface with no revision feedback is easier to use, an interface that provides a detailed categorization of sentence-level revisions is the most helpful based on user survey data, as well as the most effective based on improvement in writing outcomes.

Tangible interfaces have much potential for engendering shared interaction and reflection, as well as for promoting playful experiences. How can their properties be capitalised on to enable students to reflect on their learning, both individually and together, throughout learning sessions? This Research through Design paper describes our development of EvalMe, a flexible, tangible tool aimed at being playful, enjoyable to use and enabling children to reflect on their learning, both in the moment and after a learning session has ended. We discuss the insights gained through the process of designing EvalMe, co-defining its functionality with two groups of collaborators and deploying it in two workshop settings. Through this process, we map key contextual considerations for the design of technologies for in situ evaluation of learning experiences. Finally, we discuss how tangible evaluation technologies deployed throughout a learning session, can positively contribute to students’ reflection about their learning.

Novice programmers need differentiated assessments (such as adaptive Parsons problems) to maximize their ability to learn how to program. Parsons problems require learners to place mixed-up code blocks in the correct order to solve a problem. We conducted a within-subjects experiment to compare the efficiency and cognitive load of solving adaptive Parsons problems versus writing the equivalent (isomorphic) code. Undergraduates were usually more significantly efficient at solving a Parsons problem than writing the equivalent code, but not when the solution to the Parsons problem was unusual. This has implications for problem creators. This paper also reports on the mean cognitive load ratings of the two problem types and the relationship between efficiency and cognitive load ratings. Lastly, it reports on think-aloud observations of 11 students solving both adaptive Parsons problems and write-code problems and the results from an end-of-course student survey.

Conversational programmers want to learn about code primarily to communicate with technical co-workers, not to develop software. However, existing instructional materials don’t meet the needs of conversational programmers because they prioritize syntax and semantics over concepts and applications. This mismatch results in feelings of failure and low self-efficacy. To motivate conversational programmers, we propose purpose-first programming, a new approach that focuses on learning a handful of domain-specific code patterns and assembling them to create authentic and useful programs. We report on the development of a purpose-first programming prototype that teaches five patterns in the domain of web scraping. We show that learning with purpose-first programming is motivating for conversational programmers because it engenders a feeling of success and aligns with these learners’ goals. Purpose-first programming learning enabled novice conversational programmers to complete scaffolded code writing, debugging, and explaining activities after only 30 minutes of instruction.

Surveillance of communication between incarcerated and non-incarcerated people has steadily increased, enabled partly by technological advancements. Third-party vendors control communication tools for most U.S. prisons and jails and offer surveillance capabilities beyond what individual facilities could realistically implement. Frequent communication with family improves mental health and post-carceral outcomes for incarcerated people, but does discomfort about surveillance affect how their relatives communicate with them? To explore this and the understanding, attitudes, and reactions to surveillance, we conducted 16 semi-structured interviews with participants who have incarcerated relatives. Among other findings, we learn that participants communicate despite privacy concerns that they felt helpless to address. We also observe inaccuracies in participants’ beliefs about surveillance practices. We discuss implications of inaccurate understandings of surveillance, misaligned incentives between end-users and vendors, how our findings enhance ongoing conversations about carceral justice, and recommendations for more privacy-sensitive communication tools.

Increasingly, icons are being proposed to concisely convey privacy-related information and choices to users. However, complex privacy concepts can be difficult to communicate. We investigate which icons effectively signal the presence of privacy choices. In a series of user studies, we designed and evaluated icons and accompanying textual descriptions (link texts) conveying choice, opting-out, and sale of personal information — the latter an opt-out mandated by the California Consumer Privacy Act (CCPA). We identified icon-link text pairings that conveyed the presence of privacy choices without creating misconceptions, with a blue stylized toggle icon paired with “Privacy Options” performing best. The two CCPA-mandated link texts (“Do Not Sell My Personal Information” and “Do Not Sell My Info”) accurately communicated the presence of do-not-sell opt-outs with most icons. Our results provide insights for the design of privacy choice indicators and highlight the necessity of incorporating user testing into policy making.

“Notice and choice” is the predominant approach for data privacy protection today. There is considerable user-centered research on providing effective privacy notices but not enough guidance on designing privacy choices. Recent data privacy regulations worldwide established new requirements for privacy choices, but system practitioners struggle to implement legally compliant privacy choices that also provide users meaningful privacy control. We construct a design space for privacy choices based on a user-centered analysis of how people exercise privacy choices in real-world systems. This work contributes a conceptual framework that considers privacy choice as a user-centered process as well as a taxonomy for practitioners to design meaningful privacy choices in their systems. We also present a use case of how we leverage the design space to finalize the design decisions for a real-world privacy choice platform, the Internet of Things (IoT) Assistant, to provide meaningful privacy control in the IoT.

The deployment of technologies to track and mitigate the spread COVID-19 has surfaced tensions between individual autonomy and the collective good. These tensions reflect a conflict between two central concerns: (i) effectively controlling the spread of the pandemic and (ii) respecting individual rights, values, and freedoms. We explored these tensions in an online experiment (n = 389) designed to identify the influence of social orientation and communicative framing on perceptions and expected use of pandemic-tracking apps. We found that social orientation is a statistically significant predictor of app perception and expected use, with collectivist social orientation associated with higher levels and individualist social orientation with lower levels for both aspects. We found interactions between social orientation and communicative framing, as well as a connection between privacy concerns and expected duration of app use. Our findings hold important implications for the design, deployment, and adoption of technology for the public good. Shaping the post-pandemic social contract requires considering the long-term sociocultural impact of these technological solutions.

Online privacy policies should enable users to make informed decisions. Current text policies, however, lack usability: users often miss crucial information and consent to them without reading. Visual representation formats may increase comprehension, but are rarely used in practice. In an iterative design process we gathered qualitative feedback on typical policy contents and on existing and newly designed representation formats. We developed design guidelines and a Visual Interactive Privacy Policy based on the Privacy Policy Nutrition Label enriched with control options and further interactive elements. In an empirical evaluation, both visual representations received higher ratings of attractiveness, stimulation, novelty and transparency compared to a standard policy long text. Interactivity improved time spent with the policy. There were no effects on conversion rate, perceived control or perceived trust, efficiency and perspicuity. More research is needed, especially with regard to the cost-benefit ratio of visual privacy policies.

This paper addresses the question whether the recently proposed approach of concise privacy notices in apps and on websites is effective in raising user awareness. To assess the effectiveness in a realistic setting, we included concise notices in a fictitious but realistic fitness tracking app and asked participants recruited from an online panel to provide their feedback on the usability of the app as a cover story. Importantly, after giving feedback, users were also asked to recall the data practices described in the notices. The experimental setup included the variation of different levels of saliency and riskiness of the privacy notices. Based on a total sample of 2,274 participants, our findings indicate that concise privacy notices are indeed a promising approach to raise user awareness for privacy information when displayed in a salient way, especially in case the notices describe risky data practices. Our results may be helpful for regulators, user advocates and transparency-oriented companies in creating or enforcing better privacy transparency towards average users that do not read traditional privacy policies.

Homomorphic encryption, secure multi-party computation, and differential privacy are part of an emerging class of Privacy Enhancing Technologies which share a common promise: to preserve privacy whilst also obtaining the benefits of computational analysis. Due to their relative novelty, complexity, and opacity, these technologies provoke a variety of novel questions for design and governance. We interviewed researchers, developers, industry leaders, policymakers, and designers involved in their deployment to explore motivations, expectations, perceived opportunities and barriers to adoption. This provided insight into several pertinent challenges facing the adoption of these technologies, including: how they might make a nebulous concept like privacy computationally tractable; how to make them more usable by developers; and how they could be explained and made accountable to stakeholders and wider society. We conclude with implications for the development, deployment, and responsible governance of these privacy-preserving computation techniques.

We present PriView, a concept that allows privacy-invasive devices in the users’ vicinity to be visualised. PriView is motivated by an ever-increasing number of sensors in our environments tracking potentially sensitive data (e.g., audio and video). At the same time, users are oftentimes unaware of this, which violates their privacy. Knowledge about potential recording would enable users to avoid accessing such areas or not to disclose certain information. We built two prototypes: a) a mobile application capable of detecting smart devices in the environment using a thermal camera, and b) VR mockups of six scenarios where PriView might be useful (e.g., a rental apartment). In both, we included several types of visualisation. Results of our lab study (N=24) indicate that users prefer simple, permanent indicators while wishing for detailed visualisations on demand. Our exploration is meant to support future designs of privacy visualisations for varying smart environments.

The COVID-19 pandemic has fueled the development of smartphone applications to assist disease management. Many “corona apps” require widespread adoption to be effective, which has sparked public debates about the privacy, security, and societal implications of government-backed health applications. We conducted a representative online study in Germany (n = 1003), the US (n = 1003), and China (n = 1019) to investigate user acceptance of corona apps, using a vignette design based on the contextual integrity framework. We explored apps for contact tracing, symptom checks, quarantine enforcement, health certificates, and mere information. Our results provide insights into data processing practices that foster adoption and reveal significant differences between countries, with user acceptance being highest in China and lowest in the US. Chinese participants prefer the collection of personalized data, while German and US participants favor anonymity. Across countries, contact tracing is viewed more positively than quarantine enforcement, and technical malfunctions negatively impact user acceptance.

The shutdown measures necessary to stop the spread of COVID-19 have amplified the role of technology in intimate partner violence (IPV). Survivors may be forced to endure lockdowns with their abusers, intensifying the dangers of technology-enabled abuse (e.g. stalking, harassment, monitoring, surveillance). They may also be forced to rely on potentially compromised devices to reach support networks: a dangerous dilemma for digital safety. This qualitative study examines how technologists with computer security expertise provided remote assistance to IPV survivors during the pandemic. Findings from 24 consults with survivors and five focus groups with technologist consultants show how remote delivery of technology support services raised three fundamental challenges: (1) ensuring safety for survivors and consultants; (2) assessing device security over a remote connection; and (3) navigating new burdens for consultants, including emotional labor. We highlight implications for HCI researchers creating systems that enable access to remote expert services for vulnerable people.

Automated decision support can accelerate tedious tasks as users can focus their attention where it is needed most. However, a key concern is whether users overly trust or cede agency to automation. In this paper, we investigate the effects of introducing automation to annotating clinical texts — a multi-step, error-prone task of identifying clinical concepts (e.g., procedures) in medical notes, and mapping them to labels in a large ontology. We consider two forms of decision aid: recommending which labels to map concepts to, and pre-populating annotation suggestions. Through laboratory studies, we find that 18 clinicians generally build intuition of when to rely on automation and when to exercise their own judgement. However, when presented with fully pre-populated suggestions, these expert users exhibit less agency: accepting improper mentions, and taking less initiative in creating additional annotations. Our findings inform how systems and algorithms should be designed to mitigate the observed issues.

The popularity of racket sports (e.g., tennis and table tennis) leads to high demands for data analysis, such as notational analysis, on player performance. While sports videos offer many benefits for such analysis, retrieving accurate information from sports videos could be challenging. In this paper, we propose EventAnchor, a data analysis framework to facilitate interactive annotation of racket sports video with the support of computer vision algorithms. Our approach uses machine learning models in computer vision to help users acquire essential events from videos (e.g., serve, the ball bouncing on the court) and offers users a set of interactive tools for data annotation. An evaluation study on a table tennis annotation system built on this framework shows significant improvement of user performances in simple annotation tasks on objects of interest and complex annotation tasks requiring domain knowledge.

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders’ knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

Training datasets fundamentally impact the performance of machine learning (ML) systems. Any biases introduced during training (implicit or explicit) are often reflected in the system's behaviors leading to questions about fairness and loss of trust in the system. Yet, information on training data is rarely communicated to stakeholders. In this work, we explore the concept of data-centric explanations for ML systems that describe the training data to end-users. Through a formative study, we investigate the potential utility of such an approach, including the information about training data that participants find most compelling. In a second study, we investigate reactions to our explanations across four different system scenarios. Our results suggest that data-centric explanations have the potential to impact how users judge the trustworthiness of a system and to assist users in assessing fairness. We discuss the implications of our findings for designing explanations to support users’ perceptions of ML systems.

Generative Adversarial Networks (GANs) can automatically generate quality images from learned model parameters. However, it remains challenging to explore and objectively assess the quality of all possible images generated using a GAN. Currently, model creators evaluate their GANs via tedious visual examination of generated images sampled from narrow prior probability distributions on model parameters. Here, we introduce an interactive method to explore and sample quality images from GANs. Our first two user studies showed that participants can use the tool to explore a GAN and select quality images. Our third user study showed that images sampled from a posterior probability distribution using a Markov Chain Monte Carlo (MCMC) method on parameters of images collected in our first study resulted in on average higher quality and more diverse images than existing baselines. Our work enables principled qualitative GAN exploration and evaluation.

The advent of artificial intelligence (AI) and machine learning (ML) bring human-AI interaction to the forefront of HCI research. This paper argues that games are an ideal domain for studying and experimenting with how humans interact with AI. Through a systematic survey of neural network games (n = 38), we identified the dominant interaction metaphors and AI interaction patterns in these games. In addition, we applied existing human-AI interaction guidelines to further shed light on player-AI interaction in the context of AI-infused systems. Our core finding is that AI as play can expand current notions of human-AI interaction, which are predominantly productivity-based. In particular, our work suggests that game and UX designers should consider flow to structure the learning curve of human-AI interaction, incorporate discovery-based learning to play around with the AI and observe the consequences, and offer users an invitation to play to explore new forms of human-AI interaction.

This paper addresses an under-explored problem of AI-assisted decision-making: when objective performance information of the machine learning model underlying a decision aid is absent or scarce, how do people decide their reliance on the model? Through three randomized experiments, we explore the heuristics people may use to adjust their reliance on machine learning models when performance feedback is limited. We find that the level of agreement between people and a model on decision-making tasks that people have high confidence in significantly affects reliance on the model if people receive no information about the model’s performance, but this impact will change after aggregate-level model performance information becomes available. Furthermore, the influence of high confidence human-model agreement on people’s reliance on a model is moderated by people’s confidence in cases where they disagree with the model. We discuss potential risks of these heuristics, and provide design implications on promoting appropriate reliance on AI.

Data science (DS) projects often follow a lifecycle that consists of laborious tasks for data scientists and domain experts (e.g., data exploration, model training, etc.). Only till recently, machine learning(ML) researchers have developed promising automation techniques to aid data workers in these tasks. This paper introduces AutoDS, an automated machine learning (AutoML) system that aims to leverage the latest ML automation techniques to support data science projects. Data workers only need to upload their dataset, then the system can automatically suggest ML configurations, preprocess data, select algorithm, and train the model. These suggestions are presented to the user via a web-based graphical user interface and a notebook-based programming user interface. Our goal is to offer a systematic investigation of user interaction and perceptions of using an AutoDS system in solving a data science task. We studied AutoDS with 30 professional data scientists, where one group used AutoDS, and the other did not, to complete a data science project. As expected, AutoDS improves productivity; Yet surprisingly, we find that the models produced by the AutoDS group have higher quality and less errors, but lower human confidence scores. We reflect on the findings by presenting design implications for incorporating automation techniques into human work in the data science lifecycle.

For machine learning models to be most useful in numerous sociotechnical systems, many have argued that they must be human-interpretable. However, despite increasing interest in interpretability, there remains no firm consensus on how to measure it. This is especially true in representation learning, where interpretability research has focused on “disentanglement” measures only applicable to synthetic datasets and not grounded in human factors. We introduce a task to quantify the human-interpretability of generative model representations, where users interactively modify representations to reconstruct target instances. On synthetic datasets, we find performance on this task much more reliably differentiates entangled and disentangled models than baseline approaches. On a real dataset, we find it differentiates between representation learning methods widely believed but never shown to produce more or less interpretable models. In both cases, we ran small-scale think-aloud studies and large-scale experiments on Amazon Mechanical Turk to confirm that our qualitative and quantitative results agreed.

Many researchers motivate explainable AI with studies showing that human-AI team performance on decision-making tasks improves when the AI explains its recommendations. However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team. Can explanations help lead to complementary performance, where team accuracy is higher than either the human or the AI working solo? We conduct mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions). While we observed complementary improvements from AI augmentation, they were not increased by explanations. Rather, explanations increased the chance that humans will accept the AI’s recommendation, regardless of its correctness. Our result poses new challenges for human-centered AI: Can we develop explanatory approaches that encourage appropriate trust in AI, and therefore help generate (or improve) complementary performance?

As AI-powered systems increasingly mediate consequential decision-making, their explainability is critical for end-users to take informed and accountable actions. Explanations in human-human interactions are socially-situated. AI systems are often socio-organizationally embedded. However, Explainable AI (XAI) approaches have been predominantly algorithm-centered. We take a developmental step towards socially-situated XAI by introducing and exploring Social Transparency (ST), a sociotechnically informed perspective that incorporates the socio-organizational context into explaining AI-mediated decision-making. To explore ST conceptually, we conducted interviews with 29 AI users and practitioners grounded in a speculative design scenario. We suggested constitutive design elements of ST and developed a conceptual framework to unpack ST’s effect and implications at the technical, decision-making, and organizational level. The framework showcases how ST can potentially calibrate trust in AI, improve decision-making, facilitate organizational collective actions, and cultivate holistic explainability. Our work contributes to the discourse of Human-Centered XAI by expanding the design space of XAI.

Efforts to make machine learning more widely accessible have led to a rapid increase in Auto-ML tools that aim to automate the process of training and deploying machine learning. To understand how Auto-ML tools are used in practice today, we performed a qualitative study with participants ranging from novice hobbyists to industry researchers who use Auto-ML tools. We present insights into the benefits and deficiencies of existing tools, as well as the respective roles of the human and automation in ML workflows. Finally, we discuss design implications for the future of Auto-ML tool development. We argue that instead of full automation being the ultimate goal of Auto-ML, designers of these tools should focus on supporting a partnership between the user and the Auto-ML tool. This means that a range of Auto-ML tools will need to be developed to support varying user goals such as simplicity, reproducibility, and reliability.

While HCI research has often addressed the needs of older adults, they are often framed as being sceptical of digital technologies. We argue that while many older adults are circumspect users of digital technology, they bring rich and critical perspectives on the role of technology in society that are grounded in lived experiences across their life courses. We report on 20 technology life story interviews conducted with retirees over the age of 60. Our analysis shows how experiences of technology across their life courses significantly undermined participants’ sense of competency, independence, resilience, agency and control. Dissonances between what our participants valued and the perceived values of technology have led them to become critical adopters of technology, and resist its intrusion into certain aspects of their lives. We discuss how the critical perspectives of older adults and the value dissonances they experience are valuable for designing future digital technologies.

Critical consumerism is complex as ethical values are difficult to negotiate, appropriate products are hard to find, and product information is overwhelming. Although recommender systems offer solutions to reduce such complexity, current designs are not appropriate for niche practices and use non-personalized intransparent ethics. To support critical consumption, we conducted a design case study on a personalized food recommender system. Therefore, we first conducted an empirical pre-study with 24 consumers to understand value negotiations and current practices, co-designed the recommender system, and finally evaluated it in a real-world trial with ten consumers. Our findings show how recommender systems can support the negotiation of ethical values within the context of consumption practices, reduce the complexity of finding products and stores, and strengthen consumers. In addition to providing implications for the design to support critical consumption practices, we critically reflect on the scope of such recommender systems and its appropriation.

Older adults can struggle to access relevant community expertise when faced with new situations. One such situation is the number of cyberattacks they may face when interacting online. This paper reports on an initiative which recruited, trained, and supported older adults to become community cybersecurity educators (CyberGuardians), tasked with promoting cybersecurity best practice within their communities to prevent older adults falling victim to opportunistic cyberattacks. This initiative utilised an embedded peer-to-peer information dissemination strategy, rather than expert-to-citizen, facilitating the inclusion of individuals who would ordinarily be unlikely to seek cybersecurity information and thus may be vulnerable to cyberattacks. We report on ways the CyberGuardians used informal methods to create more aware communities, served as role models for behaviour change and indirectly improved their personal wellbeing. We discuss considerations for supporting CyberGuardians, including implications for sustainability and for replicating this model in other digital contexts, e.g., recognising misinformation or improving mental health.

Civic tech initiatives dedicated to environmental issues have become a worldwide phenomenon and made invaluable contributions to data, community building, and publics. However, many of them stop after a relatively short time. Therefore, we studied two long-lasting civic tech initiatives of global scale, to understand what makes them sustain over time. To this end, we conducted two mixed-method case studies, combining social network analysis and qualitative content analysis of Twitter data with insights from expert interviews. Drawing on our findings, we identified a set of key factors that help the studied civic tech initiatives to grow and last. Contributing to Digital Civics in HCI, we argue that the civic tech initiatives’ scaling and sustaining are configured through the entanglement of (1) civic data both captured and owned by the citizens for the citizens, (2) the use of open and accessible technology, and (3) the initiatives’ public narrative, giving them a voice on the environmental issue.

On digital media services, uncivil commenting is a persistent issue causing negative emotional reactions. One enabler for such problematic behavior is the user interface, conditioning, and structuring text-based communication online. However, the specific roles and influences of UIs are little understood, which calls for critical analysis of the current UI solutions as well as speculative exploration of alternative designs. This paper reports a research-through-design study on the problematic phenomenon regarding uncivil and inconsiderate commenting on online news, envisioning unconventional solutions with a critical voice. We unpack this problem area and outline critical perspectives to possible solutions by describing and analyzing four designs that propose to support emotion regulation by facilitating self-reflection. The design choices are further discussed in respect to interviews of ten news media experts. The findings are reflected against the question of how can critique meaningfully manifest in this challenging problem area.

HCI research predominantly uses scientific rationality to explain users’ behaviors, decisions, and interactions with statistical models based and data-driven systems. However, such interactions are often more diverse in real life, and may straddle beyond scientific and economic rationality. Building on a ten-month ethnography at seven Bangladeshi villages, we explore the social and cultural factors that influence the online betting practices among the villagers. We describe how bets harmonize with users’ faith, hunch, and cultural practices, along with statistical recommendations. Drawing on a rich body of social science work on gambling, we contribute to the HCI scholarship in rationality, justification, and postcolonial computing. Finally, we present such betting as an under-appreciated site for HCI that contradicts with the ideological hegemony of statistical rationality, and recommend a smooth integration of AI system with the “other” rationalities of the Global South.

From Fanger's seminal work on thermal comfort in the 1970s, standards governing temperatures in the workplace enshrine clothing level calculations based on full business suits, and building regulations developed using only male metabolic data, locking in a default male perspective. Even later work that highlights gender biases with regard to metabolism calculation, inclusive of both genders has focused on younger women, and the voices of older working women are missing from this discourse. We invited women over 45 to explore what they find important in workplace thermal comfort, and how devices and interfaces might meet their needs and also encourage thermal adaptivity. Our study highlights factors such as 'fresh air', and the importance of empathy to fellow inhabitants. We bring new voices to the thermal comfort discourse which supports reducing energy use in the workplace, improving thermal environments and ensuring the needs of a diverse, aging workforce are considered.

This paper joins the growing body of critical HCI work that studies the digitization of the Global South and reports the elements of ‘secularization’ in it. Based on a year-long ethnography on the contemporary transformations in religious practices in Dhaka, Bangladesh, this paper presents how the emerging “digital” cattle marketplaces subdue various forms of traditional manifestations of urban religiosity during Eid-ul-Adha, the second-largest Islamic festival in the city. This paper further depicts how such secularization contributes to diminishing rural-urban linkages, affecting electoral politics, and reducing the tolerance to religious celebrations in a city. Drawing from a rich body of work in critical urban studies, postcolonial computing, and sociology of religions, we explain how such oft-overlooked embedding of secularization in computing affects the religious fabrics in the urban regions of the Global South, and discuss its implication for HCI scholarship in diversity, inclusion, and development.

Research on how lived experiences with technology intersect with home and work are core themes within HCI. Prior work has primarily focused on conventional life and work in Western countries. However, the unconventional is becoming conventional—several rising subcultures are coming into prominence due to socio-economic pressures, aided by social media. One example—#vanlife—is now practised by an estimated three million people in North America. #vanlife combines travel, home, and work by their occupants (vanlifers (vanlifers)) living full-time in cargo vans that they usually convert themselves into living spaces. We present a portrait of vanlifers’ current technology practices gleaned through ~200 hours of fieldwork and interviews. Following a thematic analysis of our data, we identified unique opportunities for integrating technology across culture, design, homesteading, offline organization, and gaming. We have distilled these opportunities into eleven provocations to inspire critical design and informed inquiry for technological interventions for #vanlife.

Multiple methods have been used to study how social values and ethics are implicated in technology design and use, including empirical qualitative studies of technologists’ work. Recently, more experimental approaches such as design fiction explore these themes through fictional worldbuilding. This paper combines these approaches by adapting design fictions as a form of memoing, a qualitative analysis technique. The paper uses design fiction memos to analyze and reflect on ethnographic interviews and observational data about how user experience (UX) professionals at large technology companies engage with values and ethical issues in their work. The design fictions help explore and articulate themes about the values work practices and relationships of power that UX professionals grapple with. Through these fictions, the paper contributes a case study showing how design fiction can be used for qualitative analysis, and provides insights into the role of organizational and power dynamics in UX professionals’ values work.

Ground-truth labeling is an important activity in machine learning. Many studies have examined how crowdworkers apply labels to records in machine learning datasets. However, there have been few studies that have examined the work of domain experts when their knowledge and expertise are needed to apply labels.

We provide a grounded account of the work of labeling teams with domain experts, including the experiences of labeling, collaborative configurations and work-practices, and quality issues. We show three major patterns in the social design of ground truth data: Principled design, Iterative design, and Improvisational design. We interpret our results through theories of from Human Centered Data Science, and particularly work on human interventions in data science work through the design and creation of data.

The promise AI’s proponents have made for decades is one in which our needs are predicted, anticipated, and met - often before we even realize it. Instead, algorithmic systems, particularly AIs trained on large datasets and deployed to massive scales, seem to keep making the wrong decisions, causing harm and rewarding absurd outcomes. Attempts to make sense of why AIs make wrong calls in the moment explain the instances of errors, but how the environment surrounding these systems precipitate those instances remains murky. This paper draws from anthropological work on bureaucracies, states, and power, translating these ideas into a theory describing the structural tendency for powerful algorithmic systems to cause tremendous harm. I show how administrative models and projections of the world create marginalization, just as algorithmic models cause representational and allocative harm. This paper concludes with a recommendation to avoid the absurdity algorithmic systems produce by denying them power.

Dance provides unique opportunities for embodied interdisciplinary learning experiences that can be personally and culturally relevant. danceON is a system that supports learners to leverage their body movement as they engage in artistic practices across data science, computing, and dance. The technology includes a Domain Specific Language (DSL) with declarative syntax and reactive behavior, a media player with pose detection and classification, and a web-based IDE. danceON provides a low-floor allowing users to bind virtual shapes to body positions in under three lines of code, while also enabling complex, dynamic animations that users can design working with conditionals and past position data. We developed danceON to support distance learning and deployed it in two consecutive cohorts of a remote, two-week summer camp for young women of color. We present our findings from an analysis of the experience and the resulting computational performances. The work identifies implications for how design can support learners’ expression across culturally relevant themes and examines challenges from the lens of usability of the computing language and technology.

Directly manipulating the timeline, such as scrubbing for thumbnails, is the standard way of controlling how-to videos. However, when how-to videos involve physical activities, people inconveniently alternate between controlling the video and performing the tasks. Adopting a voice user interface allows people to control the video with voice while performing the tasks with hands. However, naively translating timeline manipulation into voice user interfaces (VUI) results in temporal referencing (e.g. “rewind 20 seconds”), which requires a different mental model for navigation and thereby limiting users’ ability to peek into the content. We present RubySlippers, a system that supports efficient content-based voice navigation through keyword-based queries. Our computational pipeline automatically detects referenceable elements in the video, and finds the video segmentation that minimizes the number of needed navigational commands. Our evaluation (N=12) shows that participants could perform three representative navigation tasks with fewer commands and less frustration using RubySlippers than the conventional voice-enabled video interface.

Learning musical instruments using online instructional videos has become increasingly prevalent. However, pre-recorded videos lack the instantaneous feedback and personal tailoring that human tutors provide. In addition, existing video navigations are not optimized for instrument learning, making the learning experience encumbered. Guided by our formative interviews with guitar players and prior literature, we designed Soloist, a mixed-initiative learning framework that automatically generates customizable curriculums from off-the-shelf guitar video lessons. Soloist takes raw videos as input and leverages deep-learning based audio processing to extract musical information. This back-end processing is used to provide an interactive visualization to support effective video navigation and real-time feedback on the user's performance, creating a guided learning experience. We demonstrate the capabilities and specific use-cases of Soloist within the domain of learning electric guitar solos using instructional YouTube videos. A remote user study, conducted to gather feedback from guitar players, shows encouraging results as the users unanimously preferred learning with Soloist over unconverted instructional videos.

Explicitly alerting users is not always an optimal intervention, especially when they are not motivated to obey. For example, in video-based learning, learners who are distracted from the video would not follow an alert asking them to pay attention. Inspired by the concept of Mindless Computing, we propose a novel intervention approach, Mindless Attractor, that leverages the nature of human speech communication to help learners refocus their attention without relying on their motivation. Specifically, it perturbs the voice in the video to direct their attention without consuming their conscious awareness. Our experiments not only confirmed the validity of the proposed approach but also emphasized its advantages in combination with a machine learning-based sensing module. Namely, it would not frustrate users even though the intervention is activated by false-positive detection of their attentive state. Our intervention approach can be a reliable way to induce behavioral change in human–AI symbiosis.

The need to find or construct tables arises routinely to accomplish many tasks in everyday life, as a table is a common format for organizing data. However, when relevant data is found on the web, it is often scattered across multiple tables on different web pages, requiring tedious manual searching and copy-pasting to collect data. We propose KTabulator, an interactive system to effectively extract, build, or extend ad hoc tables from large corpora, by leveraging their computerized structures in the form of knowledge graphs. We developed and evaluated KTabulator using Wikipedia and its knowledge graph DBpedia as our testbed. Starting from an entity or an existing table, KTabulator allows users to extend their tables by finding relevant entities, their properties, and other relevant tables, while providing meaningful suggestions and guidance. The results of a user study indicate the usefulness and efficiency of KTabulator in ad hoc table creation.

Toolkits for shape-changing interfaces (SCIs) enable designers and researchers to easily explore the broad design space of SCIs. However, despite their utility, existing approaches are often limited in the number of shape-change features they can express. This paper introduces MorpheesPlug , a toolkit for creating SCIs that covers seven of the eleven shape-change features identified in the literature. MorpheesPlug is comprised of (1) a set of six standardized widgets that express the shape-change features with user-definable parameters; (2) software for 3D-modeling the widgets to create 3D-printable pneumatic SCIs; and (3) a hardware platform to control the widgets. To evaluate MorpheesPlug we carried out ten open-ended interviews with novice and expert designers who were asked to design a SCI using our software. Participants highlighted the ease of use and expressivity of the MorpheesPlug.

Embodied conversational agents have changed the ways we can interact with machines. However, these systems often do not meet users’ expectations. A limitation is that the agents are monotonic in behavior and do not adapt to an interlocutor. We present SIVA (a Socially Intelligent Virtual Agent), an expressive, embodied conversational agent that can recognize human behavior during open-ended conversations and automatically align its responses to the conversational and expressive style of the other party. SIVA leverages multimodal inputs to produce rich and perceptually valid responses (lip syncing and facial expressions) during the conversation. We conducted a user study (N=30) in which participants rated SIVA as being more empathetic and believable than the control (agent without style matching). Based on almost 10 hours of interaction, participants who preferred interpersonal involvement evaluated SIVA as significantly more animate than the participants who valued consideration and independence.

Automated vehicles promise a future where drivers can engage in non-driving tasks without hands on the steering wheels for a prolonged period. Nevertheless, automated vehicles may still need to occasionally hand the control back to drivers due to technology limitations and legal requirements. While some systems determine the need for driver takeover using driver context and road condition to initiate a takeover request, studies show that the driver may not react to it. We present DeepTake, a novel deep neural network-based framework that predicts multiple aspects of takeover behavior to ensure that the driver is able to safely take over the control when engaged in non-driving tasks. Using features from vehicle data, driver biometrics, and subjective measurements, DeepTake predicts the driver’s intention, time, and quality of takeover. We evaluate DeepTake performance using multiple evaluation metrics. Results show that DeepTake reliably predicts the takeover intention, time, and quality, with an accuracy of 96%, 93%, and 83%, respectively. Results also indicate that DeepTake outperforms previous state-of-the-art methods on predicting driver takeover time and quality. Our findings have implications for the algorithm development of driver monitoring and state detection.

This paper describes how machine learning training data and symbolic knowledge from curators of conversational systems can be used together to improve the accuracy of those systems and to enable better curatorial tools. This is done in the context of a real-world practice of curators of conversational systems who often embed taxonomically-structured meta-knowledge into their documentation. The paper provides evidence that the practice is quite common among curators, that is used as part of their collaborative practices, and that the embedded knowledge can be mined by algorithms. Further, this meta-knowledge can be integrated, using neuro-symbolic algorithms, to the machine learning-based conversational system, to improve its run-time accuracy and to enable tools to support curatorial tasks. Those results point towards new ways of designing development tools which explore an integrated use of code and documentation by machines.

Program synthesis, which generates programs based on user-provided specifications, can be obscure and brittle: users have few ways to understand and recover from synthesis failures. We propose interpretable program synthesis, a novel approach that unveils the synthesis process and enables users to monitor and guide a synthesizer. We designed three representations that explain the underlying synthesis process with different levels of fidelity. We implemented an interpretable synthesizer for regular expressions and conducted a within-subjects study with eighteen participants on three challenging regex tasks. With interpretable synthesis, participants were able to reason about synthesis failures and provide strategic feedback, achieving a significantly higher success rate compared with a state-of-the-art synthesizer. In particular, participants with a high engagement tendency (as measured by NCS-6) preferred a deductive representation that shows the synthesis process in a search tree, while participants with a relatively low engagement tendency preferred an inductive representation that renders representative samples of programs enumerated during synthesis.

Modern visualization tools aim to allow data analysts to easily create exploratory visualizations. When the input data layout conforms to the visualization design, users can easily specify visualizations by mapping data columns to visual channels of the design. However, when there is a mismatch between data layout and the design, users need to spend significant effort on data transformation.

We propose Falx, a synthesis-powered visualization tool that allows users to specify visualizations in a similarly simple way but without needing to worry about data layout. In Falx, users specify visualizations using examples of how concrete values in the input are mapped to visual channels, and Falx automatically infers the visualization specification and transforms the data to match the design. In a study with 33 data analysts on four visualization tasks involving data transformation, we found that users can effectively adopt Falx to create visualizations they otherwise cannot implement.

There is increased interest in using virtual reality in education, but it often remains an isolated experience that is difficult to integrate into current instructional experiences. In this work, we adapt virtual production techniques from filmmaking to enable mixed reality capture of instructors so that they appear to be standing directly in the virtual scene. We also capitalize on the growing popularity of live streaming software for video conferencing and live production. With XRStudio, we develop a pipeline for giving lectures in VR, enabling live compositing using a variety of presets and real-time output to traditional video and more immersive formats. We present interviews with media designers experienced in film and MOOC production that informed our design. Through walkthrough demonstrations of XRStudio with instructors experienced with VR, we learn how it could be used in a variety of domains. In end-to-end evaluations with students, we analyze and compare differences of traditional video vs. more immersive lectures with XRStudio.

We present a multi-modal approach for automatically generating hierarchical tutorials from instructional makeup videos. Our approach is inspired by prior research in cognitive psychology, which suggests that people mentally segment procedural tasks into event hierarchies, where coarse-grained events focus on objects while fine-grained events focus on actions. In the instructional makeup domain, we find that objects correspond to facial parts while fine-grained steps correspond to actions on those facial parts. Given an input instructional makeup video, we apply a set of heuristics that combine computer vision techniques with transcript text analysis to automatically identify the fine-level action steps and group these steps by facial part to form the coarse-level events. We provide a voice-enabled, mixed-media UI to visualize the resulting hierarchy and allow users to efficiently navigate the tutorial (e.g., skip ahead, return to previous steps) at their own pace. Users can navigate the hierarchy at both the facial-part and action-step levels using click-based interactions and voice commands. We demonstrate the effectiveness of segmentation algorithms and the resulting mixed-media UI on a variety of input makeup videos. A user study shows that users prefer following instructional makeup videos in our mixed-media format to the standard video UI and that they find our format much easier to navigate.

Live streaming is gaining popularity across diverse application domains in recent years. A core part of the experience is streamer-viewer interaction, which has been mainly text-based. Recent systems explored extending viewer interaction to include visual elements with richer expression and increased engagement. However, understanding expressive visual inputs becomes challenging with many viewers, primarily due to the relative lack of structure in visual input. On the other hand, adding rigid structures can limit viewer interactions to narrow use cases or decrease the expressiveness of viewer inputs. To facilitate the sensemaking of many visual inputs while retaining the expressiveness or versatility of viewer interactions, we introduce a visual input management framework (VIMF) and a system, VisPoll, that help streamers specify, aggregate, and visualize many visual inputs. A pilot evaluation indicated that VisPoll can expand the types of viewer interactions. Our framework provides insights for designing scalable and expressive visual communication for live streaming.

Two social motives are distinguished by Motive Disposition Theory: affiliation and power. Motives orient, select and energize our behaviour, suggesting that the choices of power-motivated individuals should be guided by power cues, such as the appearance of strength in a game character or avatar. In study 1 we demonstrate that participants were more likely to pick strong-looking Pokémon for a fight and cute Pokémon as a companion. In addition, we show that even when considering these contexts, the power motive predicts preferences for a powerful appearance, whereas affiliation does not. In study 2 we replicate the study 1 findings and distinguish between two ways to enact the power motive (prosocial and dominant power). We demonstrate that the dominance, but not the prosociality, facet drives the preference for strong-looking Pokémon. Our findings suggest that the need to influence others—the power motive—drives the choice for battle companions who symbolize strength.

Increasingly, modern boardgames incorporate digital apps and tools to deliver content in novel ways. Despite disparate approaches to incorporating digital tools in otherwise non-digital boardgames, there has to date been no detailed classification of the different roles that these tools play in supporting gameplay. In this paper, we present a model for understanding hybrid boardgame play as performing a set of discrete functions. Through a mixed-methods approach incorporating critical play sessions, a survey of 237 boardgame players, and interviews with 18 boardgame designers and publishers, we identified the key functions performed by the digital tools in these games. Using affinity mapping, we grouped these functions into eight core categories, which were tested and refined using a survey of 44 boardgame players and designers. This model defines and classifies the diverse functions that digital tools perform in hybrid digital boardgames, demonstrating their range and potential application for researchers and game designers.

Graphical user authentication (GUA) is a common alternative to text-based user authentication, where people are required to draw graphical passwords on background images. Such schemes are theoretically considered remarkably secure because they offer a large password space. However, people tend to create their passwords on salient image areas introducing high password predictability. Aiming to help people use the password space more effectively, we propose a gameful password creation process. In this paper, we present GamePass, a gamified mechanism that integrates the GUA password creation process. We provide the first evidence that it is possible to nudge people towards better password choices by gamifying the process. GamePass randomly guides participants’ attention to areas other than the salient areas of authentication images, makes the password creation process more fun, and people are more engaged. Gamifying the password creation process enables users to interact better and make less predictable graphical password choices instead of being forced to use a strict password policy.

Many serious games are most effective when played regularly; however, little is known about how individual game elements support player adherence over time. This work draws on evidence from existing frameworks and game design theories as well as from the design of casual games to investigate how individual game mechanics affect player attrition in a serious game. We implemented a math-learning game in which we could individually layer various game mechanics, and over the course of 3 weeks, 99 participants played one of six versions: Baseline, Rewards, Novelty, Completion, Waiting, or Blocking. We compared the game versions by analyzing the players’ performance as well as behaviour. Using survival analysis, we identified that the addition of Completion and Blocking mechanics facilitated the strongest sustained engagement. These findings are congruent with existing theories of player experience and promote the development of guidelines on designing for sustained engagement in serious games.

The landscape of digital games is segregated by player ability. For example, sighted players have a multitude of highly visual games at their disposal, while blind players may choose from a variety of audio games. Attempts at improving cross-ability access to any of those are often limited in the experience they provide, or disregard multiplayer experiences. We explore ability-based asymmetric roles as a design approach to create engaging and challenging mixed-ability play. Our team designed and developed two collaborative testbed games exploring asymmetric interdependent roles. In a remote study with 13 mixed-visual-ability pairs we assessed how roles affected perceptions of engagement, competence, and autonomy, using a mixed-methods approach. The games provided an engaging and challenging experience, in which differences in visual ability were not limiting. Our results underline how experiences unequal by design can give rise to an equitable joint experience.

Videogames’ increasing cultural relevance and suffusion into everyday use contexts suggests they can no longer be considered novelties. Broadly speaking, games research at CHI has concerned two forms of peak experience—historically, research aimed to support flow, or maximise enjoyment and positive emotions; more recently, scholarship engages with more varied experiences of intense emotion, such as emotional challenge. In different ways, both approaches emphasise extra-ordinary player experience (PX). Conversely, videogame play and PX have become more routine—indeed, more ordinary—as the medium’s cultural presence grows. In this paper, we argue that HCI games research is conceptually ill-equipped to investigate these increasingly common and often desirable experiences. We conceptualise “ordinary player experience” – as familiar, emotionally moderate, co-attentive, and abstractly memorable – articulating a phenomenon whose apparent mundanity has seen it elude description to date. We discuss opportunities to productively employ ordinary PX in HCI games research, alongside conceptual implications for PX and player wellbeing.

We present an autobiographical design journey exploring the experience of returning to long-term single player games. Continuing progress from a previously saved game, particularly when substantial time has passed, is an understudied area in games research. To begin our exploration in this domain, we investigated what the return experience is like first-hand. By returning to four long-term single player games played extensively in the past, we revealed a phenomenon we call The Pivot Point, a ‘eureka’ moment in return gameplay. The pivot point anchors our design explorations, where we created prototypes to leverage the pivot point in reconnecting with the experience. These return experiences and subsequent prototyping iterations inform our understanding of how to design better returns to gameplay, which can benefit both producers and consumers of long-term single player games.

Video games are engaging multimedia experiences that require players’ cognitive, emotional, physical (in terms of controllers and exertion), and social faculties. Recent theorizing has suggested that these dimensions of demand can explain processes by which players engage with and respond to gameplay. A relatively new measure—the five-factor, 26-item Video Game Demand Scale (VGDS)—has been tested for dimensionality and measurement validity with English- and German-speaking players, but not for other play populations. Given the popularity of video games among Chinese-speaking players, this brief report demonstrates a successful translation of VGDS into Traditional Chinese (VGDS-C). A sample of N = 863 Chinese speakers in Taiwan were asked to recall and describe a recent gaming experience before completing the VGDS-C along with other gaming-related measures (tests of construct validity). VGDS-C was shown to be a reliable and valid way of assessing players’ perceptions of the myriad demands of video gaming.

Self-report assessment is important for research and game development, e.g., to gather data during play. Games can use dialogues with non-player characters (NPCs) to gather self-report data; however, players might respond differently to dialogues than questionnaires. Without guidance on how in-game assessment affects player perceptions and experiences, designers and researchers are in danger of making decisions that harm data quantity and quality, and perceptions of privacy. We conducted a user study to understand self-report collection from NPC dialogues and traditional in-game overlay questionnaires. Data quality and player experience measures autonomy, curiosity, immersion, and mastery did not differ significantly, although NPC dialogues enhanced meaning. NPC dialogues supported an increase in data quantity through voluntary 5-point scales but not via open responses; however, they also increased the perceived intimacy of shared information despite comparable objective intimacy. NPC dialogues are useful to gather quantitative self-report data. They enable a meaningful play experience but could facilitate negative effects related to privacy.

The current study explores the relationship between perceived cognitive and physical demands of a simple video game, and the balance of reward and effort that results in flow states during gameplay. Cognitive demands and both exertion-based and controller-based physical demands were perceived as lowest in situations where reward was high and effort was low (boredom), moderate when reward and effort were balanced (flow), and highest when the reward was low and effort was high (frustration). Surprisingly, player response times to a secondary task showed the greatest improvement when playing the frustrating video game condition. We interpret this latter finding as evidence of an observed task-switching effect: players initially tried to master the game's over-challenging primary task before giving up and, instead, diverted attention toward a secondary in-game task that required less effort and thus, gave greater attentional rewards to the player. The implications of this cognitive offloading are discussed.

Emotionally impactful game experiences have garnered increasing interest within HCI games research. Yet the perspectives of designers have, to date, remained largely overlooked. We interviewed 14 indie game designers regarding their values and practices in designing emotionally impactful games. Counter to the focus of recent player experience (PX) studies, we find that while designers typically have a clear vision for the intended emotional impact, they aim for their games to provide a space for players to have their own personal experiences and interpretations. Despite this player-centric orientation, players were rarely involved before and during the production to evaluate the emotional experience. Based on these findings, we identify gaps between design practice and PX research, raise open questions around the design and evaluation of emotionally impactful game experiences, and outline opportunities for HCI games research to more productively support game designers.

Player agency is central to interactive narrative and games. While previous work focuses on analyzing player perception of agency through various lenses and phenomena, like meaningful choice and expectations, it is largely theoretical. Few user studies within games explore how players reason about and judge their own agency within interactive narratives. We present an interview study where participants rated their agency experiences within narrative-focused games and described their reasoning. The analysis suggests that agency perception depends on multiple factors beyond meaningful choice, such as social investment and genre-conventions. Participants described varying preferences and value judgements for different factors, indicating that individual differences have a deep impact on agency perception in narrative-focused gameplay. We discuss the implications of these cognitive variables on design, how they can be leveraged with other factors, and how our findings can help future work enhance and measure player agency, within interactive narrative and beyond.

Worlds-in-Miniature (WiMs) are interactive worlds within a world and combine the advantages of an input space, a cartographic map, and an overview+detail interface. They have been used across the extended virtuality spectrum for a variety of applications. Building on an analysis of examples of WiMs from the research literature we contribute a design space for WiMs based on seven design dimensions. Further, we expand upon existing definitions of WiMs to provide a definition that applies across the extended reality spectrum. We identify the design dimensions of size-scope-scale, abstraction, geometry, reference frame, links, multiples, and virtuality. Using our framework we describe existing Worlds-in-Miniature from the research literature and reveal unexplored research areas. Finally, we generate new examples of WiMs using our framework to fill some of these gaps. With our findings, we identify opportunities that can guide future research into WiMs.

In virtual reality (VR) environments, asymmetric bimanual interaction techniques can increase users’ input bandwidth by complementing their perceptual and motor systems (e.g., using the dominant hand to select 3D UI controls anchored around the non-dominant arm). However, it is unclear how to optimize the layout of such 3D UI controls for near-body and mid-air interactions. We evaluate the performance and limitations of non-dominant arm-anchored 3D UIs in VR environments through a bimanual pointing study. Results demonstrated that targets appearing closer to the skin, located around the wrist, or placed on the medial side of the forearm could be selected more quickly than targets farther away from the skin, located around the elbow, or on the lateral side of the forearm. Based on these results, we developed Armstrong guidelines, demonstrated through a Unity plugin to enable designers to create performance-optimized arm-anchored 3D UI layouts.

JetController is a novel haptic technology capable of supporting high-speed and persistent 3-DoF ungrounded force feedback. It uses high-speed pneumatic solenoid valves to modulate compressed air to achieve 20-50Hz of full impulses at 4.0-1.0N, and combines multiple air propulsion jets to generate 3-DoF force feedback. Compared to propeller-based approaches, JetController supports 10-30 times faster impulse frequency, and its handheld device is significantly lighter and more compact. JetController supports a wide range of haptic events in games and VR experiences, from firing automatic weapons in games like Halo (15Hz) to slicing fruits in Fruit Ninja (up to 45Hz). To evaluate JetController, we integrated our prototype with two popular VR games, Half-life: Alyx and Beat Saber, to support a variety of 3D interactions. Study results showed that JetController significantly improved realism, enjoyment, and overall experience compared to commercial vibrating controllers, and was preferred by most participants.

Virtual Reality experiences, such as games and simulations, typically support the usage of bimanual controllers to interact with virtual objects. To recreate the haptic sensation of holding objects of various shapes and behaviors with both hands, previous researchers have used mechanical linkages between the controllers that render adjustable stiffness. However, the linkage cannot quickly adapt to simulate dynamic objects, nor it can be removed to support free movements. This paper introduces GamesBond, a pair of 4-DoF controllers without physical linkage but capable to create the illusion of being connected as a single device, forming a virtual bond. The two controllers work together by dynamically displaying and physically rendering deformations of hand grips, and so allowing users to perceive a single connected object between the hands, such as a jumping rope. With a user study and various applications we show that GamesBond increases the realism, immersion, and enjoyment of bimanual interaction.

Game designers and researchers employ a sophisticated language for producing great player experiences with concepts such as juiciness, which refers to excessive positive feedback. However, much of their discourse excludes the role and value of haptic feedback. In this paper, we adapt terminology from game design to study haptic feedback. Specifically, we define haptic embellishments (HEs) as haptic feedback that reinforce information already provided through other means (e.g., via visual feedback) and juicy haptics as excessive positive haptic feedback with the intention of improving user experience in games and other interactive media. We report two empirical studies of users’ experiences interacting with visuo-haptic content on their phones to 1) study participants’ preferences for ten design principles for HEs and 2) measure the added value of juicy haptics, implemented as HEs, on player experience in a game. Results indicate that juicy haptics can enhance enjoyability, aesthetic appeal, immersion, and meaning.

Current head-mounted displays enable users to explore virtual worlds by simply walking through them (i.e., real-walking VR). This led researchers to create haptic displays that can also simulate different types of elevation shapes. However, existing shape-changing floors are limited by their tabletop scale or the coarse resolution of the terrains they can display due to the limited number of actuators and low vertical resolution. To tackle this challenge, we introduce Elevate, a dynamic and walkable pin-array floor on which users can experience not only large variations in shapes but also the details of the underlying terrain. Our system achieves this by packing 1200 pins arranged on a 1.80 × 0.60m platform, in which each pin can be actuated to one of ten height levels (resolution: 15mm/level). To demonstrate its applicability, we present our haptic floor combined with four walkable applications and a user study that reported increased realism and enjoyment.

Numerous techniques have been proposed for locomotion in virtual reality (VR). Several taxonomies consider a large number of attributes (e.g., hardware, accessibility) to characterize these techniques. However, finding the appropriate locomotion technique (LT) and identifying gaps for future designs in the high-dimensional space of attributes can be quite challenging. To aid analysis and innovation, we devised Locomotion Vault (https://locomotionvault.github.io/), a database and visualization of over 100 LTs from academia and industry. We propose similarity between LTs as a metric to aid navigation and visualization. We show that similarity based on attribute values correlates with expert similarity assessments (a method that does not scale). Our analysis also highlights an inherent trade-off between simulation sickness and accessibility across LTs. As such, Locomotion Vault shows to be a tool that unifies information on LTs and enables their standardization and large-scale comparison to help understand the space of possibilities in VR locomotion.

Smartphone touch screens are potentially attractive for interaction in virtual reality (VR). However, the user cannot see the phone or their hands in a fully immersive VR setting, impeding their ability for precise touch input. We propose mounting a mirror above the phone screen such that the front-facing camera captures the thumbs on or near the screen. This enables the creation of semi-transparent overlays of thumb shadows and inference of fingertip hover points with deep learning, which help the user aim for targets on the phone. A study compares the effect of visual feedback on touch precision in a controlled task and qualitatively evaluates three example applications demonstrating the potential of the technique. The results show that the enabled style of feedback is effective for thumb-size targets, and that the VR experience can be enriched by using smartphones as VR controllers supporting precise touch input.

Selection and manipulation in virtual reality often happen using an avatar’s hands. However, objects outside the immediate reach require effort to select. We develop a target selection technique called Ninja Hands. It maps the movement of a single real hand to many virtual hands, decreasing the distance to targets. We evaluate Ninja Hands in two studies. The first study shows that compared to a single hand, 4 and 8 hands are significantly faster for selecting targets. The second study complements this finding by using a larger target layout with many distractors. We find no decrease in selection time across 8, 27, and 64 hands, but an increase in the time spent deciding which hand to use. Thereby, net movement time still decreases significantly. In both studies, the physical motion exerted also decreases significantly with more hands. We discuss how these findings can inform future implementations of the Ninja Hands technique.

Augmented Reality (AR) is increasingly being used for providing guidance and supporting troubleshooting in industrial settings. While the general application of AR has been shown to provide clear benefits regarding physical tasks, it is important to understand how different visualization types influence user’s performance during the execution of the tasks. Previous studies evaluating AR and user’s performance compared different media types or types of AR hardware as opposed to different types of visualization for the same hardware type. This paper provides details of our comparative study in which we identified the influence of visualization types on the performance of complex machine set-up processes. Although our results show clear advantages to using concrete rather than abstract visualizations, we also find abstract visualizations coupled with videos leads to similar user performance as with concrete visualizations.

Interacting with out of reach or occluded VR objects can be cumbersome. Although users can change their position and orientation, such as via teleporting, to help observe and select, doing so frequently may cause loss of spatial orientation or motion sickness. We present vMirror, an interactive widget leveraging reflection of mirrors to observe and select distant or occluded objects. We first designed interaction techniques for placing mirrors and interacting with objects through mirrors. We then conducted a formative study to explore a semi-automated mirror placement method with manual adjustments. Next, we conducted a target-selection experiment to measure the effect of the mirror’s orientation on users’ performance. Results showed that vMirror can be as efficient as direct target selection for most mirror orientations. We further compared vMirror with teleport technique in a virtual treasure hunt game and measured participants’ task performance and subjective experiences. Finally, we discuss vMirorr user experience and present future directions.

Tactile feedback is widely used to enhance realism in virtual reality (VR). When touching virtual objects, stiffness and roughness are common and obvious factors perceived by the users. Furthermore, when touching a surface with complicated surface structure, differences from not only stiffness and roughness but also surface height are crucial. To integrate these factors, we propose a pin-based handheld device, HairTouch, to provide stiffness differences, roughness differences, surface height differences and their combinations. HairTouch consists of two pins for the two finger segments close to the index fingertip, respectively. By controlling brush hairs’ length and bending direction to change the hairs’ elasticity and hair tip direction, each pin renders various stiffness and roughness, respectively. By further independently controlling the hairs’ configuration and pins’ height, versatile stiffness, roughness and surface height differences are achieved. We conducted a perception study to realize users’ distinguishability of stiffness and roughness on each of the segments. Based on the results, we performed a VR experience study to verify that the tactile feedback from HairTouch enhances VR realism.

For haptic guidance, vibrotactile feedback is a commonly-used mechanism, but requires users to interpret its complicated patterns especially in 3D guidance, which is not intuitive and increases their mental effort. Furthermore, for haptic guidance in virtual reality (VR), not only guidance performance but also realism should be considered. Since vibrotactile feedback interferes with and reduces VR realism, it may not be proper for VR haptic guidance. Therefore, we propose a wearable device, GuideBand, to provide intuitive 3D multilevel force guidance upon the forearm, which reproduces an effect that the forearm is pulled and guided by a virtual guider or telepresent person in VR. GuideBand uses three motors to pull a wristband at different force levels in 3D space. Such feedback usually requires much larger and heavier robotic arms or exoskeletons. We conducted a just-noticeable difference study to understand users’ force level distinguishability. Based on the results, we performed a study to verify that compared with state-of-the-art vibrotactile guidance, GuideBand is more intuitive, needs a lower level of mental effort, and achieves similar guidance performance. We further conducted a VR experience study to observe how users combine and complement visual and force guidance, and prove that GuideBand enhances realism in VR guidance.

The growing online gig economy provides ways for women to participate in a flexible, remote workforce and close the offline gender pay and participation gap. While women in online labor marketplaces earn about as much overall as men, women set lower bill rates suggesting gender differences in pricing strategies. In this study, we surveyed 392 freelancers in the United States (US) on the popular marketplace platform, Upwork, to understand strategies used to set hourly bill rates. We did not find gender differences in pricing strategies that were significantly related to bill rate. Instead, we found that other factors, such as full-time freelancer status and level of self-esteem, may help explain gender differences in bill rates. To better support equity and fairness in the growing gig economy, CHI researchers must identify, assess, and address the complex interaction between societal conditions in online labor markets.

Maintaining an awareness of one’s well-being and making work-related decisions to achieve work-life balance is critical for flexible long-hour workers. In this study, we propose that social sensing could address bottlenecks in worker’s awareness, interpretation of the informatics, and subsequent behavioral change. We conducted a four-week technology probe study by recruiting flexible long-hour professional drivers (Taxi and Uber drivers) and their significant others to use a social sensing prototype which collects data from the drivers and shares it with their partners as well as incorporates partners’ observations. We interviewed them before and after the probe study and found that while technological sensing was able to increase drivers’ awareness of their well-being status and intention to modify behaviors. The “social sensing” design was able to further shape such awareness or intention into action, highlighting the potential of using the sociotechnical approach in promoting work-life balance among long-hour workers.

Live streaming has become increasingly popular, with most streamers presenting their real-life appearance. However, Virtual YouTubers (VTubers), virtual 2D or 3D avatars that are voiced by humans, are emerging as live streamers and attracting a growing viewership in East Asia. Although prior research has found that many viewers seek real-life interpersonal interactions with real-person streamers, it is currently unknown what makes VTuber live streams engaging or how they are perceived differently than real-person streamers. We conducted an interview study to understand how viewers engage with VTubers and perceive the identities of the voice actors behind the avatars (i.e., Nakanohito). The data revealed that Virtual avatars bring unique performative opportunities which result in different viewer expectations and interpretations of VTuber behavior. Viewers intentionally upheld the disembodiment of VTuber avatars from their voice actors. We uncover the nuances in viewer perceptions and attitudes and further discuss the implications of VTuber practices to the understanding of live streaming in general.

Emerging research suggests that people trust algorithmic decisions less than human decisions. However, different populations, particularly in marginalized communities, may have different levels of trust in human decision-makers. Do people who mistrust human decision-makers perceive human decisions to be more trustworthy and fairer than algorithmic decisions? Or do they trust algorithmic decisions as much as or more than human decisions? We examine the role of mistrust in human systems in people’s perceptions of algorithmic decisions. We focus on healthcare Artificial Intelligence (AI), group-based medical mistrust, and Black people in the United States. We conducted a between-subjects online experiment to examine people’s perceptions of skin cancer screening decisions made by an AI versus a human physician depending on their medical mistrust, and we conducted interviews to understand how to cultivate trust in healthcare AI. Our findings highlight that research around human experiences of AI should consider critical differences in social groups.

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers.

Online video games support the development of social relationships through gameplay, however, gamers often cannot cultivate and maintain relationships based on social factors such as personality when using in-game matchmaking services. To address this, teammate matching sites external to games have emerged and enable gamers to offer to play games with others in exchange for payment. The affordances of these services are different from other existing gamer social sites, e.g., live streaming. Interviews were conducted with 16 dedicated users on Bixin, one of China’s largest paid teammate matching sites, to examine user motivations, practices, and perceptions. The interviews found that gamers selected paid teammates on Bixin using different criteria compared to in-game matchmaking services and emphasized the importance of real-life characteristics such as voice. To maintain connections, paid teammates often also extended communication to external communication services such as WeChat. Although most gamers expected to communicate with paid teammates as if they were friends, very few reported building real friendships with their matched counterparts.

The user study is a fundamental method used in HCI. In designing user studies, we often use compensation strategies to incentivize recruitment. However, compensation can also lead to ethical issues, such as coercion. The CHI community has yet to establish best practices for participant compensation. Through a systematic review of manuscripts at CHI and other associated publication venues, we found high levels of variation in the compensation strategies used within the community and how we report on this aspect of the study methods. A qualitative analysis of justifications offered for compensation sheds light into how some researchers are currently contextualizing this practice. This paper provides a description of current compensation strategies and information that can inform the design of compensation strategies in future studies. The findings may be helpful to generate productive discourse in the HCI community towards the development of best practices for participant compensation in user studies.

A jury of one’s peers is a prominent way to adjudicate disputes and is increasingly used in participatory governance online. The fairness of this approach rests on the assumption that juries are consistent: that the same jury would hand down similar judgments to similar cases. However, prior literature suggests that social influence would instead cause early interactions to cascade into different judgments for similar cases. In this paper, we report an online experiment that changes participants’ pseudonyms as they appear to collaborators, temporarily masking a jury’s awareness that they have deliberated together before. This technique allows us to measure consistency by reconvening the same jury on similar cases. Counter to expectation, juries are equally consistent as individuals, a result that is “good for democracy.” But this consistency arises in part due to group polarization, as consensus develops by hardening initial majority opinions. Furthermore, we find that aggregating groups’ perspectives without deliberation erodes consistency.

Computer technology is often designed in technology hubs in Western countries, invariably making it “WEIRD”, because it is based on the intuition, knowledge, and values of people who are Western, Educated, Industrialized, Rich, and Democratic. Developing technology that is universally useful and engaging requires knowledge about members of WEIRD and non-WEIRD societies alike. In other words, it requires us, the CHI community, to generate this knowledge by studying representative participant samples. To find out to what extent CHI participant samples are from Western societies, we analyzed papers published in the CHI proceedings between 2016-2020. Our findings show that 73% of CHI study findings are based on Western participant samples, representing less than 12% of the world’s population. Furthermore, we show that most participant samples at CHI tend to come from industrialized, rich, and democratic countries with generally highly educated populations. Encouragingly, recent years have seen a slight increase in non-Western samples and those that include several countries. We discuss suggestions for further broadening the international representation of CHI participant samples.

It is widely acknowledged in HCI that culture is embodied in many aspects of an individual's identity and interaction with technology. Whilst existing cultural models have been criticized for providing a deterministic view of culture, alternative methods for incorporating culture in design remain scarce. We introduce the use of Qualitative Secondary Analysis (QSA) as a bottom-up approach to construct a richer and more dynamic understanding of culture to inform our best practices in Cross-Cultural Design. We demonstrate the use of QSA within a culturally specific context, namely Saudi transnationals. We draw upon two case studies (with 55 participants) to investigate the cultural factors underpinning our participants views. We conclude with a reflection on key affordances and challenges of QSA, illustrating how QSA can be leveraged to unravel otherwise overlooked knowledge present in many qualitative HCI studies.

Leaving the field when conducting research in situated contexts, and finding ways to sustain project outcomes beyond academia, is an ongoing struggle in HCI. In our research project, we co-designed technologies with children in classroom contexts for three years. Nearing the projects’ end, we focused on creating resources that enable teachers to continue our work with their pupils. In collaboration with teachers, we developed socio-material tools that support them in empowering neurodiverse children to engage with technology in creative ways and create their own technologies. While the majority of technology design toolkits are stand-alone artefacts, part of our toolkit is an infrastructure to keep guiding and supporting teachers beyond the project’s end. In this paper, we discuss the teachers expectations and requirements for a toolkit and argue that an infrastructure must be part of a toolkit. We present a set of guidelines for researchers planning for a project’s end.

Search and rescue (SAR), a disaster response activity performed to locate and save victims, primarily involves collective sensemaking and planning. SAR responders learn to search and navigate the environment, process information about buildings, and collaboratively plan with maps. We synthesize data from five sources, including field observations and interviews, to understand the informational components of SAR and how information is recorded and communicated. We apply activity theory, uncovering unforeseen factors that are relevant to the design of collaboration systems and training solutions. Through our analysis, we derive design implications to support collaborative information technology and training systems: mixing physical and digital mapping; mixing individual and collective mapping; building for different levels and sources of information; and building for different rules, roles, and activities.

Hackathons are increasingly embraced across diverse sectors as a way of democratizing the design of technology. Several attempts have been made to redefine the format and desired end goal of hackathons in recent years thereby warranting closer methodological scrutiny. In this paper, we apply program theory to analyze the processes and effects of 16 hackathon case studies through published research literature. Building upon existing research on hackathons, our work offers a critical perspective examining the methodological validity of hackathons and exemplifies how specific processes for organizing hackathons are modified for different purposes. Our main contribution is a program theory analysis of hackathon formats that provides an exploration and juxtaposition of 16 case studies in terms of causal relations between the input, process and the effects of hackathons. Our cataloguing of examples can serve as an inspirational planning resource for future organizers of hackathons.

Tabs have become integral to browsing the Web yet have changed little since their introduction nearly 20 years ago. In contrast, the internet has gone through dramatic changes, with users increasingly moving from navigating to websites to exploring information across many sources to support online sensemaking. This paper investigates how tabs today are overloaded with a diverse set of functionalities and issues users face when managing them. We interviewed ten information workers asking about their tab management strategies and walk through each open tab on their work computers four times over two weeks. We uncovered competing pressures pushing for keeping tabs open (ranging from interaction to emotional costs) versus pushing for closing them (such as limited attention and resources). We then surveyed 103 participants to estimate the frequencies of these pressures at scale. Finally, we developed design implications for future browser interfaces that can better support managing these pressures.

Hundreds of millions of speakers of bidirectional (BiDi) languages rely on writing systems that mix the native right-to-left script with left-to-right strings. The global reach of interactive digital technologies requires special attention to these people, whose perception of interfaces is affected by this script mixture. However, empirical research on this topic is scarce. Although leading software vendors provide guidelines for BiDi design, bidirectional interfaces demonstrate inconsistent and incorrect directionality of UI elements, which may cause user confusion and errors.

Through a websites' review, we identified problematic UI items and considered reasons for their existence. In an online survey with 234 BiDi speakers, we observed that in many cases, users' direction preferences were inconsistent with the guidelines. The findings provide potential insights for design rules and empirical evidence for the problem's complexity, suggesting the need for further empirical research and greater attention by the HCI community to the BiDi design problem.

Accurate drone positioning is challenging because pilots only have a limited position and direction perception of a flying drone from their perspective. This makes conventional joystick-based speed control inaccurate and more complicated and significantly degrades piloting performance. We propose PinpointFly, an egocentric drone interface that allows pilots to arbitrarily position and rotate a drone using position-control direct interactions on a see-through mobile AR where the drone position and direction are visualized with a virtual cast shadow (i.e., the drone’s orthogonal projection onto the floor). Pilots can point to the next position or draw the drone’s flight trajectory by manipulating the virtual cast shadow and the direction/height slider bar on the touchscreen. We design and implement a prototype of PinpointFly for indoor and visual line of sight scenarios, which are comprised of real-time and predefined motion-control techniques. We conduct two user studies with simple positioning and inspection tasks. Our results demonstrate that PinpointFly makes the drone positioning and inspection operations faster, more accurate, simpler and fewer workload than a conventional joystick interface with a speed-control method.

AI systems collect our personal information in order to provide personalized services, raising privacy concerns and making users leery. As a result, systems have begun emphasizing overt over covert collection of information by directly asking users. This poses an important question for ethical interaction design, which is dedicated to improving user experience while promoting informed decision-making: Should the interface tout the benefits of information disclosure and frame itself as a help-provider? Or, should it appear as a help-seeker? We decided to find out by creating a mockup of a news recommendation system called Mindz and conducting an online user study (N=293) with the following four variations: AI system as help seeker vs. help provider vs. both vs. neither. Data showed that even though all participants received the same recommendations, power users tended to trust a help-seeking Mindz more whereas non-power users favored one that is both help-seeker and help-provider.

Online symptom checkers (OSC) are widely used intelligent systems in health contexts such as primary care, remote healthcare, and epidemic control. OSCs use algorithms such as machine learning to facilitate self-diagnosis and triage based on symptoms input by healthcare consumers. However, intelligent systems’ lack of transparency and comprehensibility could lead to unintended consequences such as misleading users, especially in high-stakes areas such as healthcare. In this paper, we attempt to enhance diagnostic transparency by augmenting OSCs with explanations. We first conducted an interview study (N=25) to specify user needs for explanations from users of existing OSCs. Then, we designed a COVID-19 OSC that was enhanced with three types of explanations. Our lab-controlled user study (N=20) found that explanations can significantly improve user experience in multiple aspects. We discuss how explanations are interwoven into conversation flow and present implications for future OSC designs.

We investigate the use of a miniaturized Near-Infrared Spectroscopy (NIRS) device in an assisted decision-making task. We consider the real-world scenario of determining whether food contains gluten, and we investigate how end-users interact with our NIRS detection device to ultimately make this judgment. In particular, we explore the effects of different nutrition labels and representations of confidence on participants’ perception and trust. Our results show that participants tend to be conservative in their judgment and are willing to trust the device in the absence of understandable label information. We further identify strategies to increase user trust in the system. Our work contributes to the growing body of knowledge on how NIRS can be mass-appropriated for everyday sensing tasks, and how to enhance the trustworthiness of assisted decision-making systems.

Recently, Artificial Intelligence (AI) has been used to enable efficient decision-making in managerial and organizational contexts, ranging from employment to dismissal. However, to avoid employees’ antipathy toward AI, it is important to understand what aspects of AI employees like and/or dislike. In this paper, we aim to identify how employees perceive current human resource (HR) teams and future algorithmic management. Specifically, we explored what factors negatively influence employees’ perceptions of AI making work performance evaluations. Through in-depth interviews with 21 workers, we found that 1) employees feel six types of burdens (i.e., emotional, mental, bias, manipulation, privacy, and social) toward AI's introduction to human resource management (HRM), and that 2) these burdens could be mitigated by incorporating transparency, interpretability, and human intervention to algorithmic decision-making. Based on our findings, we present design efforts to alleviate employees’ burdens. To leverage AI for HRM in fair and trustworthy ways, we call for the HCI community to design human-AI collaboration systems with various HR stakeholders.

Autonomous vehicles could improve mobility, safety, and inclusion in traffic. While this technology seems within reach, its successful introduction depends on the intended user’s acceptance. A substantial factor for this acceptance is trust in the autonomous vehicle’s capabilities. Visualizing internal information processed by an autonomous vehicle could calibrate this trust by enabling the perception of the vehicle’s detection capabilities (and its failures) while only inducing a low cognitive load. Additionally, the simultaneously raised situation awareness could benefit potential take-overs. We report the results of two comparative online studies on visualizing semantic segmentation information for the human user of autonomous vehicles. Effects on trust, cognitive load, and situation awareness were measured using a simulation (N=32) and state-of-the-art panoptic segmentation on a pre-recorded real-world video (N=41). Results show that the visualization using Augmented Reality increases situation awareness while remaining low cognitive load.

Automated Vehicles (AVs) are expected to radically disrupt our mobility. Whereas much is speculated about how AVs will actually be implemented in the future, we argue that their advent should be taken as an opportunity to enhance all people’s mobility and improve their lives. Thus, it is important to focus on both the environment and the needs of target groups that have not been sufficiently considered in the past. In this paper, we present the findings from a qualitative study (N=11) of public attitude on hypothetical implementation scenarios for AVs. Our results indicate that people are aware of the benefits of shared mobility for the environment and society, and are generally open to using it. However, 1) emotional factors mitigate this openness and 2) security concerns were expressed by female participants. We recommend that identified concerns must be addressed to allow AVs fully exploiting their benefits for society and environment.

Policymakers recommend that automated vehicles (AVs) display their automated driving status using an external human-machine interface (eHMI). However, previous studies suggest that a status eHMI is associated with overtrust, which might be overcome by an additional yielding intent message. We conducted a video-based laboratory study (N = 67) to investigate pedestrians’ trust and crossing behavior in repeated encounters with AVs. In a 2x2 between-subjects design, we investigated (1) the occurrence of a malfunction (AV failing to yield) and (2) system transparency (status eHMI vs. status+intent eHMI). Results show that during initial encounters, trust gradually increases and crossing onset time decreases. After a malfunction, trust declines but recovers quickly. In the status eHMI group, trust was reduced more, and participants showed 7.3 times higher odds of colliding with the AV as compared to the status+intent group. We conclude that a status eHMI can cause pedestrians to overtrust AVs and advocate additional intent messages.

Recent automotive research often focuses on automated driving, including the interaction between automated vehicles (AVs) and so-called “vulnerable road users” (VRUs). While road safety statistics and traffic psychology at least define VRUs as pedestrians, cyclists, and motorcyclists, many publications on human-vehicle interaction use the term without even defining it. The actual target group remains unclear. Since each group already poses a broad spectrum of research challenges, a one-fits-all solution seems unrealistic and inappropriate, and a much clearer differentiation is required. To foster clarity and comprehensibility, we propose a literature-based taxonomy providing a structured separation of (vulnerable) road users, designed to particularly (but not exclusively) support research on the communication between VRUs and AVs. It consists of two conceptual hierarchies and will help practitioners and researchers by providing a uniform and comparable set of terms needed for the design, implementation, and description of HCI applications.

Human-drone interaction is a growing topic of interest within HCI research. Researchers propose many innovative concepts for drone applications, but much of this research does not incorporate knowledge on existing applications already adopted by professionals. This limits the validity of said research. To address this limitation, we present our findings from an in-depth interview study with 10 professional drone pilots. Our participants were armed with significant experience and qualifications – pertinent to both drone operations and a set of applications covering diverse industries. Our findings have resulted in design recommendations that should inform both ends and means of human-drone interaction research. These include, but are not limited to: safety-related protocols, insights from domain-specific use cases, and relevant practices outside of hands-on flight.

In this paper, we investigate the human ability to distinguish political social bots from humans on Twitter. Following motivated reasoning theory from social and cognitive psychology, our central hypothesis is that especially those accounts which are opinion-incongruent are perceived as social bot accounts when the account is ambiguous about its nature. We also hypothesize that credibility ratings mediate this relationship. We asked N = 151 participants to evaluate 24 Twitter accounts and decide whether the accounts were humans or social bots. Findings support our motivated reasoning hypothesis for a sub-group of Twitter users (those who are more familiar with Twitter): Accounts that are opinion-incongruent are evaluated as relatively more bot-like than accounts that are opinion-congruent. Moreover, it does not matter whether the account is clearly social bot or human or ambiguous about its nature. This was mediated by perceived credibility in the sense that congruent profiles were evaluated to be more credible resulting in lower perceptions as bots.

In a fully autonomous driving situation, passengers hand over the steering control to a highly automated system. Autonomous driving behaviour may lead to confusion and negative user experience. When establishing such new technology, the user’s acceptance and understanding are crucial factors regarding success and failure. Using a driving simulator and a mobile application, we evaluated if system transparency during and after the interaction can increase the user experience and subjective feeling of safety and control. We contribute an initial guideline for autonomous driving experience design, bringing together the areas of user experience, explainable artificial intelligence and autonomous driving. The AVAM questionnaire, UEQ-S and interviews show that explanations during or after the ride help turn a negative user experience into a neutral one, which might be due to the increased feeling of control. However, we did not detect an effect for combining explanations during and after the ride.

Designing technologies to support mental health in children is a growing area of research. However less is known about how to design with therapists for children with serious emotional behaviour issues. We conducted a contextual enquiry at an organisation that provides therapy treatment for children of trauma backgrounds. A co-design with therapists of a reflective storytelling activity provided insight into how to integrate design research activities with therapy approaches. Our analysis produced a framework summarising the important elements of a facilitated reflective experience. The framework is intended to guide the design of technologies to support safety, connection, and reflection in scaffolding social emotional learning towards improved emotional behaviour for children. Future directions for applying the framework include: augmenting existing therapy activities with technology, technology to support children learning how to self-regulate their emotions outside the clinic, and technology to help parents in emotion coaching their child.

Expressivity is frequently recurring as a term in HCI, but it is often approached from different perspectives. Affective computing prompted research into emotional expressivity, and with technology becoming more ubiquitous and tangible, the opportunities for expressive behaviors towards systems as well as the expressivity of systems increases. By analyzing exemplar research-through-design cases and a literature survey on the use of expressivity in interaction, we discuss how different perspectives and concepts contribute to understand expressivity in interaction. We integrate these perspectives and make them operational for interaction design by creating a framework including design considerations such as freedom of interaction, action-perception loops, multimodality, subtlety, ambiguity, skill development and temporal form. The framework is a result of a mixed-method approach including a review of existing definitions and scholarly artefacts, and a systematic literature review to identify design cases including an analysis of these design cases. We finally illustrate how the framework has been used to inform the design of a shape-changing soft-robotic interface. As a result, we contribute an integrated framework on how to design for expressivity in interaction.

Although audiobooks are increasingly being used, people tend to perceive audiobook experiences as 'not real reading' due to its intangibility and ephemerality. In this paper, we developed ADIO, a device augmenting audiobook experience through representing personal listening state in the form of an interactive physical bookshelf. ADIO displays a user's listening progress through a pendant's changing length and the user's digital audiobook archive titles. The result of our four-week in-field study with six participants revealed that ADIO provided proof of the user's listening-to, which brought a sense of reading and gave a trigger for recalling the listened-to audiobook content. Additionally, audiobooks' improved visibility reminded participants to listen to them, and ADIO's physical interaction allowed participants to form personal patterns for listening to audiobooks. Our findings proposed new methods for augmenting the audiobook listening experience at three stages and further implications for designing physical curation on users’ digital archives.

Service robots are envisioned to be adaptive to their working environment based on situational knowledge. Recent research focused on designing visual representation of knowledge graphs for expert users. However, how to generate an understandable interface for non-expert users remains to be explored. In this paper, we use knowledge graphs (KGs) as a common ground for knowledge exchange and develop a pattern library for designing KG interfaces for non-expert users. After identifying the types of robotic situational knowledge from the literature, we present a formative study in which participants used cards to communicate the knowledge for given scenarios. We iteratively coded the results and identified patterns for representing various types of situational knowledge. To derive design recommendations for applying the patterns, we prototyped a lab service robot and conducted Wizard-of-Oz testing. The patterns and recommendations could provide useful guidance in designing knowledge-exchange interfaces for robots.

As the internet is increasingly embedded in the everyday things in our homes, we notice a need for greater focus on the role care plays in those relationships—and therefore an opportunity to realize unseen potential in reimagining home Internet of Things (IoT). In this paper we report on our inquiry of home dwellers’ relationships to caring for their everyday things and homes (referred to as thingcare). Findings from our design ethnography reveal four thematic qualities of their relationships to thingcare: Care Spectacle, Care Liminality, Ontological Braiding, and Care Condition. Using these themes as touchstones, we co-speculated to produce four speculative IoT concepts to explore what care as a design ethic might look like for IoT and reflect on nascent opportunities and challenges for domestic IoT design. We conclude by considering structures of power and privilege embedded within care practices that critically open new design imaginaries for IoT.

There is growing attention in the HCI community on how technology could be designed to support experiences of reminiscence on past life experiences. Yet, this research has largely overlooked people with blindness. We present a study that aims to understand everyday experiences of reminiscence for people with blindness. We conducted a qualitative study with 9 participants with blindness to understand their personal routines, wishes and desires, and challenges and tensions regarding the experience of reminiscence. Findings are interpreted to discuss new possibilities that offer starting points for future design initiatives and openings for collaboration aimed at creating technology to better support the practices of capturing, sharing, and reflecting on significant memories of the past.

Peer support through social media has been shown to have significant potential to improve health care outcomes. Despite this, very little is understood about how to design a social media-based peer support system. We use the model of unplatformed design to structure a multi-phase design process of a WhatsApp-based peer support system, with and for participants undergoing extreme weight loss as part of a health care intervention into diabetes management. From a mixed-methods evaluation of a three-month deployment of the system and reflections upon the design process we explore the value of the model in facilitating the expression of authentic peer support, and identify how the unique characteristics of unplatformed design allowed for the creation of a peer support system that was responsive to participants’ existing everyday use of social media technologies.

Digital data has become a key part of everyday life: people manage increasingly large and disparate collections of photos, documents, media, etc. But what happens after death? How can users select and prepare what data to leave behind before their eventual death? To explore how to support users, we first ran an ideation workshop to generate design ideas; then, we created a design workbook with 12 speculative concepts that explore diverging approaches and perspectives. We elicited reactions to the concepts from 20 participants (18-81, varied occupations). We found that participants anticipated different types of motivation at different life stages, wished for tools to feel personal and intimate, and preferred individual control on their post-death self-representation. They also found comprehensive data replicas creepy and saw smart assistants as potential aides for suggesting meaningful data. Based on the results, we discuss key directions for designing more personalized and respectful death-preparation tools.

Facilitation, or the craft and use of specific tools, methods, and practices to influence the way groups gather and converse to reach a goal, requires the nuanced craft of a facilitator to guide complex conversations. Online gathering spaces are often designed with neither the facilitator’s knowledge nor the range of tools their methods require, making online facilitation difficult. In this paper, we describe the design and evaluation of Keeper – an online video call extension environment – informed by the widespread, foundational facilitation method of Circle practice. We define three initial facilitator-driven goals of the platform: 1) enable tone setting, 2) ease the practice and smoothness of online conversation, and 3) improve social presence. After a qualitative study with 72 participants, semi-structured interviews with facilitators, and an RCT with 100 participants, we observe improved tone setting and ease of conversation and suggest several ways to further pursue improved online conversation.

Design research is important for understanding and interrogating how emerging technologies shape human experience. However, design research with Machine Learning (ML) is relatively underdeveloped. Crucially, designers have not found a grasp on ML uncertainty as a design opportunity rather than an obstacle. The technical literature points to data and model uncertainties as two main properties of ML. Through post-phenomenology, we position uncertainty as one defining material attribute of ML processes which mediate human experience. To understand ML uncertainty as a design material, we investigate four design research case studies involving ML. We derive three provocative concepts: thingly uncertainty: ML-driven artefacts have uncertain, variable relations to their environments; pattern leakage: ML uncertainty can lead to patterns shaping the world they are meant to represent; and futures creep: ML technologies texture human relations to time with uncertainty. Finally, we outline design research trajectories and sketch a post-phenomenological approach to human-ML relations.

User engagement with data privacy and security through consent banners has become a ubiquitous part of interacting with internet services. While previous work has addressed consent banners from either interaction design, legal, and ethics-focused perspectives, little research addresses the connections among multiple disciplinary approaches, including tensions and opportunities that transcend disciplinary boundaries. In this paper, we draw together perspectives and commentary from HCI, design, privacy and data protection, and legal research communities, using the language and strategies of “dark patterns” to perform an interaction criticism reading of three different types of consent banners. Our analysis builds upon designer, interface, user, and social context lenses to raise tensions and synergies that arise together in complex, contingent, and conflicting ways in the act of designing consent banners. We conclude with opportunities for transdisciplinary dialogue across legal, ethical, computer science, and interactive systems scholarship to translate matters of ethical concern into public policy.

As the role technology plays in relationships between people and their governments grows, developing a better understanding of how trust can inform designing civic technology with trust is urgent work for human computer-interaction researchers. This paper reports our efforts to design with trust through a two-year design-ethnography with the City of Atlanta Mayor's Office of Immigrant Affairs. We developed a sociotechnical system—Code Enforcer—to help this office guide immigrant residents through successfully engaging the city's code enforcement process. To inform the design process, we adapted our framework of trust-as-distance. While the framework was instrumental for integrating issues of trust throughout our design process, it also introduced tensions between how and by whom trust was enacted and interpreted. By reflecting on these tensions, we tease out the political and moral elements of designing with trust vital for HCI to navigate moving forward.

With AI on the boom, DeepFakes have emerged as a tool with a massive potential for abuse. The hyper-realistic imagery of these manipulated videos coupled with the expedited delivery models of social media platforms gives deception, propaganda, and disinformation an entirely new meaning. Hence, raising awareness about DeepFakes and how to accurately flag them has become imperative. However, given differences in human cognition and perception, this is not straightforward. In this paper, we perform an investigative user study and also analyze existing AI detection algorithms from the literature to demystify the unknowns that are at play behind the scenes when detecting DeepFakes. Based on our findings, we design a customized training program to improve detection and evaluate on a treatment group of low-literate population, which is most vulnerable to DeepFakes. Our results suggest that, while DeepFakes are becoming imperceptible, contextualized education and training can help raise awareness and improve detection.

Although the number of influencers is increasing and being an influencer is one of the most frequently mentioned career aspirations of young people, we still know very little about influencers’ motivations and actual practices from the HCI perspective. Driven by the emerging field of Human-Food Interaction and novel phenomena on social media such as Finstas, ASMR, Mukbang and live streaming, we would like to highlight the significance of food influencers as influential content creators and their social media practices. We have conducted a qualitative interview study and analyzed over 1,500 posts of food content creators on Instagram, focusing on practices of content creation, photography, staging, posting, and use of technology. Based on our findings, we have derived a process model that outlines the practices of this rather small, but influential user group. We contribute to the field of HCI by outlining the practices of food influencers as influential content creators within the social media sphere to open up design spaces for interaction researchers and practitioners.

Within the wider open science reform movement, HCI researchers are actively debating how to foster transparency in their own field. Publication venues play a crucial role in instituting open science practices, especially journals, whose procedures arguably lend themselves better to them than conferences. Yet we know little about how much HCI journals presently support open science practices. We identified the 51 most frequently published-in journals by recent CHI first authors and coded them according to the Transparency and Openness Promotion guidelines, a high-profile standard of evaluating editorial practices. Results indicate that journals in our sample currently do not set or specify clear openness and transparency standards. Out of a maximum of 29, the modal score was 0 (mean = 2.5, SD = 3.6, max = 15). We discuss potential reasons, the aptness of natural science-based guidelines for HCI, and next steps for the HCI community in furthering openness and transparency.

Electrodermal activity data is widely used in HCI to capture rich and unbiased signals. Results from related fields, however, have suggested several methodological issues that can arise when practices do not follow established standards. In this paper, we present a systematic methodological review of CHI papers involving the use of EDA data according to best practices from the field of psychophysiology, where standards are well-established and mature. We found severe issues in our sample at all stages of the research process. To ensure the validity of future research, we highlight pitfalls and offer directions for how to improve community standards.

A user's ownership perception of virtual objects, such as cloud files, is generally uncertain. Is this valid for streaming platforms featuring accounts designed for sharing (DS)? We observe sharing practices within DS accounts of streaming platforms and identify their ownership characteristics and unexpected complications through two mixed-method studies. Casual and Cost-splitting are the two sharing practices identified. The owner is the sole payer for the account in the former, whereas profile holders split the cost in the latter. We distinguish two types of ownership in each practice—Primary and Dual. In Primary ownership, the account owner has the power to allow others to use the account; in Dual ownership, Primary ownership appears in conjunction with joint ownership, notably displaying asymmetric ownership perceptions among users. Conflicts arise when the sharing agreements collapse. Therefore, we propose design recommendations that bridge ownership differences based on sharing practices of DS accounts.

Monitoring advertising around controversial issues is an important step in ensuring accountability and transparency of political processes. To that end, we use the Facebook Ads Library to collect 2312 migration-related advertising campaigns in Italy over one year. Our pro- and anti-immigration classifier (F1=0.85) reveals a partisan divide among the major Italian political parties, with anti-immigration ads accounting for nearly 15M impressions. Although composing 47.6% of all migration-related ads, anti-immigration ones receive 65.2% of impressions. We estimate that about two thirds of all captured campaigns use some kind of demographic targeting by location, gender, or age. We find sharp divides by age and gender: for instance, anti-immigration ads from major parties are 17% more likely to be seen by a male user than a female. Unlike pro-migration parties, we find that anti-immigration ones reach a similar demographic to their own voters. However their audience change with topic: an ad from anti-immigration parties is 24% more likely to be seen by a male user when the ad speaks about migration, than if it does not. Furthermore, the viewership of such campaigns tends to follow the volume of mainstream news around immigration, supporting the theory that political advertisers try to “ride the wave” of current news. We conclude with policy implications for political communication: since the Facebook Ads Library does not allow to distinguish between advertisers intentions and algorithmic targeting, we argue that more details should be shared by platforms regarding the targeting configuration of socio-political campaigns.

Maintenance of industrial equipment is done by fitters, electricians and other maintainers. For safety and quality control, maintainers must follow procedures; historically these have been paper-based. Asset-owning organisations seek to transition maintainers to digital platforms. However, there are limited studies on the potential impact of digitisation on maintenance work and the maintainers that perform it. Our challenge is to identify interface design considerations that support the safe and reliable execution of work. We looked specifically at maintenance procedures and conducted semi-structured interviews with process-plant maintainers. Thematic analysis identified eight factors influencing maintainers’ perceptions towards using digital technologies in their work. We map these factors to three categories, work identity, agency and community. These categories are consistent with concepts from the Job Characteristics Model (JCM). The contribution of this work is the relevance of job characteristics in guiding user interface design for maintainers, for which we make a number of recommendations.

Spreadsheet users routinely read, and misread, others' spreadsheets, but literature offers only a high-level understanding of users’ comprehension behaviors. This limits our ability to support millions of users in spreadsheet comprehension activities. Therefore, we conducted a think-aloud study of 15 spreadsheet users who read others’ spreadsheets as part of their work. With qualitative coding of participants’ comprehension needs, strategies and difficulties at 20-second granularity, our study provides the most detailed understanding of spreadsheet comprehension to date.

Participants comprehending spreadsheets spent around 40% of their time seeking additional information needed to understand the spreadsheet. These information seeking episodes were tedious: around 50% of participants reported feeling overwhelmed. Moreover, participants often failed to obtain the necessary information and worked instead with guesses about the spreadsheet. Eventually, 12 out of 15 participants decided to go back to the spreadsheet's author for clarifications. Our findings have design implications for reading as well as writing spreadsheets.

A prominent approach to combating online misinformation is to debunk false content. Here we investigate downstream consequences of social corrections on users’ subsequent sharing of other content. Being corrected might make users more attentive to accuracy, thus improving their subsequent sharing. Alternatively, corrections might not improve subsequent sharing - or even backfire - by making users feel defensive, or by shifting their attention away from accuracy (e.g., towards various social factors). We identified N=2,000 users who shared false political news on Twitter, and replied to their false tweets with links to fact-checking websites. We find causal evidence that being corrected decreases the quality, and increases the partisan slant and language toxicity, of the users’ subsequent retweets (but has no significant effect on primary tweets). This suggests that being publicly corrected by another user shifts one's attention away from accuracy - presenting an important challenge for social correction approaches.

New parents, defined as parents of children between the infant and preschooler stages, are increasingly turning to online media to exchange support and information to help with their life-changing transition. Understanding parents’ discussion online is crucial to the design and development of technologies that can better support their media interaction. This work studies how new parents use online media using a large-scale parenting corpus. To do so, we first employed a card-sorting methodology to identify a set of parenting topics, with which we trained BERT classifiers to automatically identify the topics of Reddit posts. We then investigate at scale what parenting topics were talked about most by new parents, how topics change over the course of their participation, and how interactions with different topics affect members’ engagement in the community. We conclude with implications of our research in designing future research and online parenting communities.

The amount of autonomy in software engineering tools is increasing as developers build increasingly complex systems. We study factors influencing software engineers’ trust in an autonomous tool situated in a high stakes workplace, because research in other contexts shows that too much or too little trust in autonomous tools can have negative consequences. We present the results of a ten week ethnographic case study of engineers collaborating with an autonomous tool to write control software at the National Aeronautics and Space Administration to support high stakes missions. We find that trust in an autonomous software engineering tool in this setting was influenced by four main factors: the tool’s transparency, usability, its social context, and the organization’s associated processes. Our observations lead us to frame trust as a quality the operator places in their collaboration with the automated system, and we outline implications of this framing and other results for researchers studying trust in autonomous systems, designers of software engineering tools, and organizations conducting high stakes work with these tools.

Vehicle manufacturers and government agencies are considering using vehicle-to-pedestrian (V2P) communication to improve pedestrian safety. However, there are unanswered questions about whether people will heed alerts and warnings presented through a smartphone. We conducted between-subject studies with younger and older adults where they physically crossed a virtual street. They received either permissive alerts (safe to cross), prohibitive warnings (not safe to cross), or no alerts or warnings (control). We found that both older and younger adults were highly likely to heed permissive alerts, even when this meant taking gaps between two vehicles that were smaller than they would typically take on their own. We also found that we could shift participants’ road-crossing behavior toward greater caution when they were only alerted to cross very large gaps between two vehicles. Participants stated that alerts and warnings were useful, but that prohibitive warnings were annoying. These findings give insights into V2P design and pedestrian behavior when smartphone assistance is provided.

There is a growing concern that e-commerce platforms are amplifying vaccine-misinformation. To investigate, we conduct two-sets of algorithmic audits for vaccine misinformation on the search and recommendation algorithms of Amazon—world’s leading e-retailer. First, we systematically audit search-results belonging to vaccine-related search-queries without logging into the platform—unpersonalized audits. We find 10.47% of search-results promote misinformative health products. We also observe ranking-bias, with Amazon ranking misinformative search-results higher than debunking search-results. Next, we analyze the effects of personalization due to account-history, where history is built progressively by performing various real-world user-actions, such as clicking a product. We find evidence of filter-bubble effect in Amazon’s recommendations; accounts performing actions on misinformative products are presented with more misinformation compared to accounts performing actions on neutral and debunking products. Interestingly, once user clicks on a misinformative product, homepage recommendations become more contaminated compared to when user shows an intention to buy that product.

With increasing interest in multisensory experiences in HCI there is a need to consider the potential impact of crossmodal correspondences (CCs) between sensory modalities on perception and interpretation. We investigated CCs between active haptic experiences of tangible 3D objects, visual colour and emotion using the “Bouba/Kiki” paradigm. We asked 30 participants to assign colours and emotional categories to 3D-printed objects with varying degrees of angularity and complexity. We found tendencies to associate high degrees of complexity and angularity with red colours, low brightness and high arousal levels. Less complex round shapes were associated with blue colours, high brightness and positive valence levels. These findings contrast previously reported crossmodal effects triggered by 2D shapes of similar angularity and complexity, suggesting that designers cannot simply extrapolate potential perceptual and interpretive experiences elicited by 2D shapes to seemingly similar 3D tangible objects. Instead, we propose a design space for creating tangible multisensory artefacts that can trigger specific emotional percepts and discuss implications for exploiting CCs in the design of interactive technology.

Static illustrations are ubiquitous means to represent interaction scenarios. Across papers and reports, these visuals demonstrate people’s use of devices, explain systems, or show design spaces. Creating such figures is challenging, and very little is known about the overarching strategies for visually representing interaction scenarios. To mitigate this task, we contribute a unified taxonomy of design elements that compose such figures. In particular, we provide a detailed classification of Structural and Interaction strategies, such as composition, visual techniques, dynamics, representation of users, and many others – all in context of the type of scenarios. This taxonomy can inform researchers’ choices when creating new figures, by providing a concise synthesis of visual strategies, and revealing approaches they were not aware of before. Furthermore, to support the community for creating further taxonomies, we also provide three open-source software facilitating the coding process and visual exploration of the coding scheme.

We present a novel, interactive interface for the integrated cleanup, neatening, structuring and vectorization of sketch imagery. Converting scanned raster drawings into vector illustrations is a well-researched set of problems. Our approach is based on a Delaunay subdivision of the raster drawing. We algorithmically generate a colored grouping of Delaunay regions that users interactively refine by dragging and dropping colors. Sketch strokes defined as marking boundaries of different colored regions are automatically neatened using Bézier curves, and turned into closed regions suitable for fills, textures, layering and animation. We show that minimal user interaction using our technique enables better sketch vectorization than state of art automated approaches. A user study, further shows our interface to be simple, fun and easy to use, yet effectively able to process messy images with a mix of construction lines, noisy and incomplete curves, sketched with arbitrary stroke style.

We present CASSIE, a conceptual modeling system in VR that leverages freehand mid-air sketching, and a novel 3D optimization framework to create connected curve network armatures, predictively surfaced using patches with C0 continuity. Our system provides a judicious balance of interactivity and automation, providing a homogeneous 3D drawing interface for a mix of freehand curves, curve networks, and surface patches. Our system encourages and aids users in drawing consistent networks of curves, easing the transition from freehand ideation to concept modeling. A comprehensive user study with professional designers as well as amateurs (N=12), and a diverse gallery of 3D models, show our armature and patch functionality to offer a user experience and expressivity on par with freehand ideation, while creating sophisticated concept models for downstream applications.

Task-irrelevant distractors affect visuo-motor control for target acquisition and studying such effects has already received much attention in human-computer interaction. However, there has been little research into distractor effects on crossing-based interaction. We thus conducted an empirical study on pen-based interfaces to investigate six crossing tasks with distractor interference in comparison to two tasks without it. The six distractor-related tasks differed in movement precision constraint (directional/amplitude), target size, target distance, distractor location and target-distractor spacing. We also developed and experimentally validated six quantitative models for the six tasks. Our results show that crossing targets with distractors had longer average times and similar accuracy than that without distractors. The effects of distractors varied depending on distractor location, target-distractor spacing and movement precision constraint. When spacing is smaller than 11.27 mm, crossing tasks with distractor interference can be regarded as pointing tasks or a combination of pointing and crossing tasks, which could be better fitted with our proposed models than Fitts’ law. According to these results, we provide practical implications to crossing-based user interface design.

Attentional tunneling describes a phenomenon in Augmented Reality (AR) where users excessively focus on virtual content while neglecting their physical surroundings. This leads to the concern that users could neglect hazardous situations when using AR applications. However, studies have often confounded the role of the virtual content with the role of the associated task in inducing attentional tunneling. In this paper, we disentangle the impact of the associated task and of the virtual content on the attentional tunneling effect by measuring reaction times to events in two user studies. We found that presenting virtual content did not significantly increase user reaction times to events, but adding a task to the content did. This work contributes towards our understanding of the attentional tunneling effect on handheld AR devices, and highlights the need to consider both task and context when evaluating AR application usage.

Abstract: Force feedback devices, such as motor-based exoskeletons or wearables based on electrical muscle stimulation (EMS), have the unique potential to accelerate users’ own reaction time (RT). However, this speedup has only been explored while the device is attached to the user. In fact, very little is known regarding whether this faster reaction time still occurs after the user removes the device from their bodies–this is precisely what we investigated by means of a simple reaction time (RT) experiment, in which participants were asked to tap as soon as they saw an LED flashing. Participants experienced this in three EMS conditions: (1) fast-EMS, the electrical impulses were synced with the LED; (2) agency-EMS, the electrical impulse was delivered 40ms faster than the participant’s own RT, which prior work has shown to preserve one’s sense of agency over this movement; and, (3) late-EMS: the impulse was delivered after the participant’s own RT. Our results revealed that the participants’ RT was significantly reduced by approximately 8ms (up to 20ms) only after training with the agency-EMS condition. This finding suggests that the prioritizing agency during EMS training is key to motor-adaptation, i.e., it enables a faster motor response even after the user has removed the EMS device from their body.

The overall pace of interaction combines the user’s pace and the system’s pace, and a pace mismatch could impair user preferences (e.g., animations or timeouts that are too fast or slow for the user). Motivated by studies of speech rate convergence, we conducted an experiment to examine whether user preferences for system pace are correlated with user pace. Subjects first completed a series of trials to determine their user pace. They then completed a series of hierarchical drag-and-drop trials in which folders automatically expanded when the cursor hovered for longer than a controlled timeout. Results showed that preferences for timeout values correlated with user pace – slow-paced users preferred long timeouts, and fast-paced users preferred short timeouts. Results indicate potential benefits in moving away from fixed or customisable settings for system pace. Instead, systems could improve preferences by automatically adapting their pace to converge towards that of the user.

Back-of-device interaction is a promising approach to interacting on smartphones. In this paper, we create a back-of-device command and text input technique called BackSwipe, which allows a user to hold a smartphone with one hand, and use the index finger of the same hand to draw a word-gesture anywhere at the back of the smartphone to enter commands and text. To support BackSwipe, we propose a back-of-device word-gesture decoding algorithm which infers the keyboard location from back-of-device gestures, and adjusts the keyboard size to suit the gesture scales; the inferred keyboard is then fed back into the system for decoding. Our user study shows BackSwipe is feasible and a promising input method, especially for command input in the one-hand holding posture: users can enter commands at an average accuracy of 92% with a speed of 5.32 seconds/command. The text entry performance varies across users. The average speed is 9.58 WPM with some users at 18.83 WPM; the average word error rate is 11.04% with some users at 2.85%. Overall, BackSwipe complements the extant smartphone interaction by leveraging the back of the device as a gestural input surface.

Decoding on phrase-level may afford more correction accuracy than on word-level according to previous research. However, how phrase-level input affects the user typing behavior, and how to design the interaction to make it practical remain under explored. We present PhraseFlow, a phrase-level input keyboard that is able to correct previous text based on the subsequently input sequences. Computational studies show that phrase-level input reduces the error rate of autocorrection by over 16%. We found that phrase-level input introduced extra cognitive load to the user that hindered their performance. Through an iterative design-implement-research process, we optimized the design of PhraseFlow that alleviated the cognitive load. An in-lab study shows that users could adopt PhraseFlow quickly, resulting in 19% fewer error without losing speed. In real-life settings, we conducted a six-day deployment study with 42 participants, showing that 78.6% of the users would like to have the phrase-level input feature in future keyboards.

Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants “eavesdropped’’ on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.

When users complete tasks on the computer, the knowledge they leverage and their intent is often lost because it is tedious or challenging to capture. This makes it harder to understand why a colleague designed a component a certain way or to remember requirements for software you wrote a year ago. We introduce think-aloud computing, a novel application of the think-aloud protocol where computer users are encouraged to speak while working to capture rich knowledge with relatively low effort. Through a formative study we find people shared information about design intent, work processes, problems encountered, to-do items, and other useful information. We developed a prototype that supports think-aloud computing by prompting users to speak and contextualizing speech with labels and application context. Our evaluation shows more subtle design decisions and process explanations were captured in think-aloud than via traditional documentation. Participants reported that think-aloud required similar effort as traditional documentation.

Various contact tracing approaches have been applied to help contain the spread of COVID-19, with technology-based tracing and human tracing among the most widely adopted. However, governments and communities worldwide vary in their adoption of digital contact tracing, with many instead choosing the human approach. We investigate how people perceive the respective benefits and risks of human and digital contact tracing through a mixed-methods survey with 291 respondents from the United States. Participants perceived digital contact tracing as more beneficial for protecting privacy, providing convenience, and ensuring data accuracy, and felt that human contact tracing could help provide security, emotional reassurance, advice, and accessibility. We explore the role of self-tracking technologies in public health crisis situations, highlighting how designs must adapt to promote societal benefit rather than just self-understanding. We discuss how future digital contact tracing can better balance the benefits of human tracers and technology amidst the complex contact tracing process and context.

The rise of ridesharing platforms has transformed traditional transportation, making it more accessible for getting to work and accessing grocery stores and healthcare providers, which are essential to physical and mental well-being. However, such technologies are not available everywhere. Additionally, there is a scarcity of HCI work that investigates how vulnerable populations such as rural-dwelling people with HIV face and overcome transportation barriers. To extend past research, we conducted 31 surveys and 18 interviews with people living with HIV (22 surveys, 14 interviews) and their case coordinators (9 surveys, 4 interviews) in rural areas. Contrary to past research, we found that the use of alternative vehicles, extensive support networks, and nonprofit health organizations facilitated transportation. However, distance, the lack of trust and infrastructure, stigma, and other cultural underpinnings made popular forms of urban transportation unappealing. We contextualize our findings with prior research and contribute implications for future research and design.

Polycystic Ovary Syndrome (PCOS) is a condition that causes hormonal imbalance and infertility in women and people with female reproductive organs. PCOS causes different symptoms for different people, with no singular or universal cure. Being a stigmatized and enigmatic condition, it is challenging to discover, diagnose, and manage PCOS. This work aims to inform the design of inclusive health technologies through an understanding of people’s lived experiences and challenges with PCOS. We conducted semi-structured interviews with 10 women diagnosed with PCOS and analyzed a PCOS-specific subreddit forum. We report people’s support-seeking, sense-making, and self-experimentation practices, and find uncertainty and stigma to be key in shaping their unique experiences of the condition. We further identify potential avenues for designing technology to support their diverse needs, such as personalized and contextual tracking, accelerated self-discovery, and co-management, contributing to a growing body of HCI literature on stigmatized topics in women’s health and well-being.

Physical activity (PA) is crucial for reducing the risk of obesity, an epidemic that disproportionately burdens families of low-socioeconomic status (SES). While fitness tracking tools can increase PA awareness, more work is needed to examine (1) how such tools can help people benefit from their social environment, and (2) how reflections can help enhance PA attitudes. We investigated how fitness tracking tools for families can support social modeling and self-modeling (through reflection), two critical processes in Social Cognitive Theory. We developed StoryMap, a novel fitness tracking app for families aimed at supporting both modes of modeling. Then, we conducted a five-week qualitative study evaluating StoryMap with 16 low-SES families. Our findings contribute an understanding of how social and self-modeling can be implemented in fitness tracking tools and how both modes of modeling can enhance key PA attitudes: self-efficacy and outcome expectations. Finally, we propose design recommendations for social personal informatics tools.

Fertility tracking and technology are characterized by logging varied health-related data potentially associated with female fertility cycles. Such data are often seen as private and restricted to the individual level. We conducted an interview study with 21 people (16 in the U.S.) facing challenges to conceive and 5 U.S. healthcare providers specialized in infertility to analyze (in)fertility experiences with data. Our findings suggest that although fertility data are considered personal and private, they are embedded in larger ecological systems of use, influencing and being influenced by different stakeholders, institutional contexts, and sociocultural factors. Leveraging the Ecological Systems Theory, we analyze the relationships and factors shaping individuals’ fertility trajectories, discussing how the different layers influence the work individuals have to engage and the burden imposed on them through various social, institutional, and cultural boundaries. We propose an ecological perspective on fertility data practices and discuss opportunities to counter-influence broader environmental systems through data tracking.

Transgender people face difficulties accessing healthcare from providers, and thus often turn to online sources to seek health information. However, online platforms may not properly support trans health information seeking, and health information found online may be limited in accuracy. To examine how online platforms can best support trans health information seeking, we conducted online focus groups with trans people (n = 26) about their experiences with online health information and their needs and desires for online health information seeking platforms. We found that trans people face both facilitators and barriers to finding accurate, actionable information online. Facilitators include online community discovery, group privacy features, and the dual synchronous and asynchronous nature of online content. Barriers include platform censorship, misinformation, hate speech, and lack of tools to flag inaccurate content. We provide recommendations for how platforms can support trans health information seeking by ensuring that medical information is accurate, accessible, easy to locate, and relevant to a diverse set of trans identities and experiences.

Self-tracking can help personalize self-management interventions for chronic conditions like type 2 diabetes (T2D), but reflecting on personal data requires motivation and literacy. Machine learning (ML) methods can identify patterns, but a key challenge is making actionable suggestions based on personal health data. We introduce GlucoGoalie, which combines ML with an expert system to translate ML output into personalized nutrition goal suggestions for individuals with T2D. In a controlled experiment, participants with T2D found that goal suggestions were understandable and actionable. A 4-week in-the-wild deployment study showed that receiving goal suggestions augmented participants’ self-discovery, choosing goals highlighted the multifaceted nature of personal preferences, and the experience of following goals demonstrated the importance of feedback and context. However, we identified tensions between abstract goals and concrete eating experiences and found static text too ambiguous for complex concepts. We discuss implications for ML-based interventions and the need for systems that offer more interactivity, feedback, and negotiation.

Recently, consumer-facing health technologies such as Artificial Intelligence (AI)-based symptom checkers (AISCs) have sprung up in everyday healthcare practice. AISCs solicit symptom information from users and provide medical suggestions and possible diagnoses, a responsibility that people usually entrust with real-person authorities such as physicians and expert patients. Thus, the advent of AISCs begs a question of whether and how they transform the notion of medical authority in people's everyday healthcare practice. To answer this question, we conducted an interview study with thirty AISC users. We found that users assess the medical authority of AISCs using various factors including AISCs’ automated decisions and interaction design patterns, associations with established medical authorities like hospitals, and comparisons with other health technologies. We reveal how AISCs are used in healthcare delivery, discuss how AI transforms conventional understandings of medical authority, and derive implications for designing AI-enabled health technology.

People with food hypersensitivities experience adverse reactions when eating certain foods and thus need to adapt their diet. When dining out, the challenge is greater as people entrust the care of their allergy, intolerance, or celiac disease, in the hands of staff who might not have enough knowledge to appropriately care for them. This interview study explored how people with food hypersensitivities avoid reactions while eating out, to inspire future digital technology design. Our findings show the social and emotional impact of food hypersensitivities and how people practically cope by investigating restaurants’ safety precautions, correcting orders, or even educating restaurants’ staff. We discuss our findings against the experiences of other people living with chronic conditions and offer design opportunities for digital technologies to enhance dining out experiences of people with food hypersensitivities.

Public controversies around the unethical use of personal data are increasing, spotlighting data ethics as an increasingly important field of study. MyData is a related emerging vision that emphasizes individuals’ control of their personal data. In this paper, we investigate people’s perceptions of various data management scenarios by measuring the perceived ethicality and level of felt concern concerning the scenarios. We deployed a set of 96 unique scenarios to an online crowdsourcing platform for assessment and invited a representative sample of the participants to a second-stage questionnaire about the MyData vision and its potential in the field of healthcare. Our results provide a timely investigation into how topical data-related practices affect the perceived ethicality and the felt concern. The questionnaire analysis reveals great potential in the MyData vision. Through the combined quantitative and qualitative results, we contribute to the field of data ethics.

Personal health informatics continues to grow in both research and practice, revealing many challenges of designing applications that address people’s needs in their health, everyday lives, and collaborations with clinicians. Research suggests strategies to address such challenges, but has struggled to translate these strategies into design practice. This study examines translation of insights from personal health informatics research into resources to support designers. Informed by a review of relevant literature, we present our development of a prototype set of design cards intended to support designers in re-thinking potential assumptions about personal health informatics. We examined our design cards in semi-structured interviews, first with 12 student designers and then with 12 health-focused professional designers and researchers. Our results and discussion reveal tensions and barriers designers encounter, the potential for translational resources to inform the design of health-related technologies, and a need to support designers in addressing challenges of knowledge, advocacy, and evidence in designing for health.

Breastfeeding brings benefits for newborns and parents, but can be a challenging process. In this paper, we leverage a mixed-methods approach that builds on the Integrated Behavioural Model (IBM) to explore parents’ perspectives toward breastfeeding along with their lived experiences, and examine the role of technology in this setting. Results of twelve semi-structured interviews and 175 online survey responses suggest generally positive attitudes toward breastfeeding and good theoretical knowledge. This is combined with a complex lived experience of breastfeeding where main challenges are situated in practical, emotional, and environmental/societal aspects, which are currently not sufficiently recognised by technology that seeks to support breastfeeding. Building upon our findings, we present points for reflection for the design of technology to support breastfeeding, focusing on the importance of drawing from the lived experience of parents, and ensuring that technology not only casts breastfeeding as an individual but also as a collective effort.

Beyond a simple notification of incoming calls or messages, more complex information such as alphabets and digits can be delivered through spatiotemporal tactile patterns (STPs) on a wrist-worn tactile display (WTD) with multiple tactors. However, owing to the limited skin area and spatial acuity of the wrist, frequent confusions occur between closely located tactors, resulting in a low recognition accuracy. Furthermore, the accuracies reported in previous studies have mostly been measured for a specific posture and could further decrease with free arm postures in real life. Herein, we present Heterogeneous Stroke, a design concept for improving the recognition accuracy of STPs on a WTD. By assigning unique vibrotactile stimuli to each tactor, the confusion between tactors can be reduced. Through our implementation of Heterogeneous Stroke, the alphanumeric characters could be delivered with high accuracy (93.8% for 26 alphabets and 92.4% for 10 digits) across different arm postures.

We propose a new type of haptic actuator, which we call MagnetIO, that is comprised of two parts: one battery-powered voice-coil worn on the user's fingernail and any number of interactive soft patches that can be attached onto any surface (everyday objects, user's body, appliances, etc.). When the user's finger wearing our voice-coil contacts any of the interactive patches it detects its magnetic signature via magnetometer and vibrates the patch, adding haptic feedback to otherwise input-only interactions. To allow these passive patches to vibrate, we make them from silicone with regions doped with polarized neodymium powder, resulting in soft and stretchable magnets. This stretchable form-factor allows them to be wrapped to the user's body or everyday objects of various shapes. We demonstrate how these add haptic output to many situations, such as adding haptic buttons to the walls of one's home. In our technical evaluation, we demonstrate that our interactive patches can be excited across a wide range of frequencies (0-500 Hz) and can be tuned to resonate at specific frequencies based on the patch's geometry. Furthermore, we demonstrate that MagnetIO's vibration intensity is as powerful as a typical linear resonant actuator (LRA); yet, unlike these rigid actuators, our passive patches operate as springs with multiple modes of vibration, which enables a wider band around its resonant frequency than an equivalent LRA.

We propose ThermoCaress, a haptic device to create a stroking sensation on the forearm using pressure force and present thermal feedback simultaneously. In our method, based on the phenomenon of thermal referral, by overlapping a stroke of pressure force, users feel as if the thermal stimulation moves although the position of temperature source is static. We designed the device to be compact and soft, using microblowers and inflatable pouches for presenting pressure force and water for presenting thermal feedback. Our user study showed that the device succeeded in generating thermal referrals and creating a moving thermal illusion. The results also suggested that cold temperature enhance the pleasantness of stroking. Our findings contribute to expanding the potential of thermal haptic devices.

In ad-hoc human-robot collaboration (HRC), humans and robots work on a task without pre-planning the robot's actions prior to execution; instead, task allocation occurs in real-time. However, prior research has largely focused on task allocations that are pre-planned - there has not been a comprehensive exploration or evaluation of techniques where task allocation is adjusted in real-time. Inspired by HCI research on territoriality and proxemics, we propose a design space of novel task allocation techniques including both explicit techniques, where the user maintains agency, and implicit techniques, where the efficiency of automation can be leveraged. The techniques were implemented and evaluated using a tabletop HRC simulation in VR. A 16-participant study, which presented variations of a collaborative block stacking task, showed that implicit techniques enable efficient task completion and task parallelization, and should be augmented with explicit mechanisms to provide users with fine-grained control.

We propose a technique that allows an unprecedented level of dexterity in electrical muscle stimulation (EMS), i.e., it allows interactive EMS-based devices to flex the user's fingers independently of each other. EMS is a promising technique for force feedback because of its small form factor when compared to mechanical actuators. However, the current EMS approach to flexing the user's fingers (i.e., attaching electrodes to the base of the forearm, where finger muscles anchor) is limited by its inability to flex a target finger's metacarpophalangeal (MCP) joint independently of the other fingers. In other words, current EMS devices cannot flex one finger alone, they always induce unwanted actuation to adjacent fingers. To tackle the lack of dexterity, we propose and validate a new electrode layout that places the electrodes on the back of the hand, where they stimulate the interossei/lumbricals muscles in the palm, which have never received attention with regards to EMS. In our user study, we found that our technique offers four key benefits when compared to existing EMS electrode layouts: our technique (1) flexes all four fingers around the MCP joint more independently; (2) has less unwanted flexion of other joints (such as the proximal interphalangeal joint); (3) is more robust to wrist rotations; and (4) reduces calibration time. Therefore, our EMS technique enables applications for interactive EMS systems that require a level of flexion dexterity not available until now. We demonstrate the improved dexterity with four example applications: three musical instrumental tutorials (piano, drum, and guitar) and a VR application that renders force feedback in individual fingers while manipulating a yo-yo.

Compared to grounded force feedback, providing tactile feedback via a wearable device can free the user and broaden the potential applications of simulated physical interactions. However, neither the limitations nor the full potential of tactile-only feedback have been precisely examined. Here we investigate how the dimensionality of cutaneous fingertip feedback affects user movements and virtual object recognition. We combine a recently invented 6-DOF fingertip device with motion tracking, a head-mounted display, and novel contact-rendering algorithms to enable a user to tactilely explore immersive virtual environments. We evaluate rudimentary 1-DOF, moderate 3-DOF, and complex 6-DOF tactile feedback during shape discrimination and mass discrimination, also comparing to interactions with real objects. Results from 20 naive study participants show that higher-dimensional tactile feedback may indeed allow completion of a wider range of virtual tasks, but that feedback dimensionality surprisingly does not greatly affect the exploratory techniques employed by the user.

Drone assisted navigation aids for supporting walking activities of visually impaired have been established in related work but fine-point object grasping tasks and the object localization in unknown environments still presents an open and complex challenge. We present a drone-based interface that provides fine-grain haptic feedback and thus physically guides them in hand-object localization tasks in unknown surroundings. Our research is built around community groups of blind or visually impaired (BVI) people, which provide in-depth insights during the development process and serve later as study participants. A pilot study infers users’ sensibility to applied guiding stimuli forces and the different human-drone tether interfacing possibilities. In a comparative follow-up study, we show that our drone-based approach achieves greater accuracy compared to a current audio-based hand guiding system and delivers overall a more intuitive and relatable fine-point guiding experience.

Today’s typical input device is flat, rigid and made of glass. However, advances in sensing technology and interaction design suggest thinking about input on other surface, including soft materials. While touching rigid and soft materials might feel similar, they clearly feel different when pressure is applied to them. Yet, to date, studies only investigated force input on rigid surfaces. We present a first systematic evaluation of the effects of compliance on force input. Results of a visual targeting task for three levels of softness indicate that high force levels appear more demanding for soft surfaces, but that performance is otherwise similar. Performance remained very high (∼ 5% for 20 force levels) regardless of the compliance, suggesting force input was underestimated so far. We infer implications for the design of force input on soft surfaces and conclude that interaction models used on rigid surfaces might be used on soft surfaces.

Providing haptic feedback when manipulating virtual objects is an essential part of immersive virtual reality experiences; however, it is challenging to replicate all of an object's properties and characteristics. We propose the use of visuo-haptic illusions alongside physical proxies to enhance the scope of proxy-based interactions with virtual objects. In this work, we focus on two manipulation techniques, linear translation and stretching across different distances, and investigate how much discrepancy between the physical proxy and the virtual object may be introduced without participants noticing. In a study with 24 participants, we found that manipulation technique and travel distance significantly affect the detection thresholds, and that visuo-haptic illusions impact performance and accuracy. We show that this technique can be used to enable functional proxy objects that act as stand-ins for multiple virtual objects, illustrating the technique through a showcase VR-DJ application.

Different form factors of wearable technology provide unique opportunities for output based on how they are connected to the human body. In this work, we investigate the idea of delivering notifications through devices worn on the underside of a user’s clothing. A wearable worn in such a manner is in direct contact with the user’s skin. We leverage this proximity to test the performance of 10 on-skin sensations (Press, Poke, Pinch, Heat, Cool, Blow, Suck, Vibrate, Moisture and Brush) as methods of notification delivery. We developed prototypes for each stimulus and conducted a user study to evaluate them across 6 locations commonly covered by upper body clothing. Results indicate significant differences in reaction time, error rates and comfort which may influence the design of future under-clothing wearables.

Traditional goal setting simply assumes a binary outcome for goal evaluation. This binary judgment does not consider a user’s effort, which may demotivate the user. This work explores the possibility of mitigating this negative impact with a slight modification on the goal evaluation criterion, by introducing a ‘margin’ that is widely used for quality control in the manufacturing fields. A margin represents a range near the goal where the user’s outcome will be regarded as ‘good enough’ even if the user fails to reach it. We explore users’ perceptions and behaviors through a large-scale survey study and a small-scale field experiment using a coaching system to promote physical activity. Our results provide positive evidence on the margin, such as lowering the burden of goal achievement and increasing motivation to make attempts. We discuss practical design implications on margin-enabled goal setting and evaluation for behavioral change support systems.

We present a qualitative study of a six-month pilot of WhatsApp-based facilitated peer support groups, serving youth living with human immunodeficiency virus (HIV) in an informal settlement in Nairobi, Kenya. Popular chat apps are increasingly being leveraged to make a combination of patient-provider communication and peer support more accessible beyond formal healthcare settings. However, how these interventions are experienced in Global South contexts with phone sharing and intermittent data access is understudied. The context of stigmatized illnesses like HIV further complicates privacy concerns. We draw on chat records and interviews with youth and the facilitator to describe their experience of the intervention. We find that despite tensions in group dynamics, intermittent participation, and contingencies around privacy, youth were motivated by newfound aspirations and community to manage their health. We use our findings to discuss implications for the design of chat-based peer interventions, negotiation of privacy in mobile health applications, and the role of aspirations in health interventions.

Health and well-being are integral parts of the human experience, and yet due to numerous factors are inaccessible for many communities. The transgender community is no exception and faces increased risk for both physical and mental illness. This population faces many unique challenges before, during, and after transition. To gain a deeper understanding of the trans community’s health and well-being needs, we conducted twenty-one interviews with transgender individuals to determine how they navigated their identities and transitions. From our interviews, we examine and highlight the unique needs of the trans population with respect to health, well-being, identity, and transition. We discuss how designers can better understand and accommodate the diversity of this community, give suggestions for the design of technologies for trans health and well-being, and contribute open areas of research.

Interfaces designed to elicit empathy provide an opportunity for HCI with important pro-social outcomes. Recent research has demonstrated that perceiving expressive biosignals can facilitate emotional understanding and connection with others, but this work has been largely limited to visual approaches. We propose that hearing these signals will also elicit empathy, and test this hypothesis with sounding heartbeats. In a lab-based within-subjects study, participants (N = 27) completed an emotion recognition task in different heartbeat conditions. We found that hearing heartbeats changed participants’ emotional perspective and increased their reported ability to “feel what the other was feeling.” From these results, we argue that auditory heartbeats are well-suited as an empathic intervention, and might be particularly useful for certain groups and use-contexts because of its musical and non-visual nature. This work establishes a baseline for empathic auditory interfaces, and offers a method to evaluate the effects of future designs.

Homesickness, which refers to feelings of distress caused by separation from home, is prevalent among university-aged students. Chronic homesickness can exacerbate mood problems, erode academic performance and lead to dropout from school. The present research examines how students use digital technologies to resolve the experience of missing home. Qualitative interviews and diaries with 50 students at major Australian universities revealed that technologies play a significant role in alleviating homesickness. Specifically, students use technologies to acquire social contact, find help and support, build co-presence to recreate their home, connect with culture, experience distant places, and regulate emotions. However, the use of technology sometimes led to increased emotional labour, frequent exposure to homesickness triggers, and heightened perceptions of distance. We conclude by discussing possible design implications for new technologies that allow students to alleviate homesickness by experiencing their home from afar.

The ubiquity of self-tracking devices and smartphone apps has empowered people to collect data about themselves and try to self-improve. However, people with little to no personal analytics experience may not be able to analyze data or run experiments on their own (self-experiments). To lower the barrier to intervention-based self-experimentation, we developed an app called Self-E, which guides users through the experiment. We conducted a 2-week diary study with 16 participants from the local population and a second study with a more advanced group of users to investigate how they perceive and carry out self-experiments with the help of Self-E, and what challenges they face. We find that users are influenced by their preconceived notions of how healthy a given behavior is, making it difficult to follow Self-E’s directions and trusting its results. We present suggestions to overcome this challenge, such as by incorporating empathy and scaffolding in the system.

We propose TiltChair, an actuated office chair that physically manipulates the user’s posture by actively inclining the chair’s seat to address problems associated with prolonged sitting. The system controls the inclination angle and motion speed with the aim of achieving manipulative but unobtrusive posture guidance. To demonstrate its potential, we first built a prototype of TiltChair with a seat that could be tilted by pneumatic control. We then investigated the effects of the seat’s inclination angle and motions on task performance and overall sitting experience through two experiments. The results show that the inclination angle mainly affects the difficulty of maintaining one’s posture, while the motion speed affected the conspicuousness and subjective acceptability of the motion. However, these seating conditions did not affect objective task performance. Based on these results, we propose a design space for facilitating effective seat-inclination behavior using the three dimensions of angle, speed, and continuity. Furthermore, we discuss promising applications.

How effectively do we adhere to nudges and interventions that help us control our online browsing habits? If we have a temporary lapse and disable the behavior change system, do we later resume our adherence, or has the dam broken? In this paper, we investigate these questions through log analyses of 8,000+ users on HabitLab, a behavior change platform that helps users reduce their time online. We find that, while users typically begin with high-challenge interventions, over time they allow themselves to slip into easier and easier interventions. Despite this, many still expect to return to the harder interventions imminently: they repeatedly choose to be asked to change difficulty again on the next visit, declining to have the system save their preference for easy interventions.

A commitment device, an attempt to bind oneself for a successful goal achievement, has been used as an effective strategy to promote behavior change. However, little is known about how commitment devices are used in the wild, and what aspects of commitment devices are related to goal achievements. In this paper, we explore a large-scale dataset from stickK, an online behavior change support system that provides both financial and social commitments. We characterize the patterns of behavior change goals (e.g., topics and commitment setting) and then perform a series of multilevel regression analyses on goal achievements. Our results reveal that successful goal achievements are largely dependent on the configuration of financial and social commitment devices, and a mixed commitment setting is considered beneficial. We discuss how our findings could inform the design of effective commitment devices, and how large-scale data can be leveraged to support data-driven goal elicitation and customization.

In India and other developing countries, Community Health Workers (CHWs) provide the first line of care in delivering necessary maternal and child health services. In this work, we assess the training and skill-building needs of CHWs, through a mobile-based training intervention deployed for six months to 500 CHWs for conducting 144 training sessions in rural India. We qualitatively probed 1178 questions, asked by CHWs, during training sessions, and conducted a content analysis of the learning material provided to CHWs. Further, we interviewed 48 CHWs to understand the rationale of information seeking and perceptions of training needs. We present our understanding of the knowledge gaps of CHWs and how the current learning material and training methods are ineffective in addressing it. Our study presents design implications for HCI4D researchers for mobile learning platforms targeted towards CHWs. We also provide policy-level suggestions to improve the training of CHWs in India or a similar context.

We present findings from a three country study exploring the intersection between female intimate health and religious beliefs. Through a qualitative study with Muslim female populations in Pakistan, Bangladesh and Malaysia, three different Muslim majority contexts, we examine the deep impact Islamic beliefs have on female intimate health and well-being. Our study investigates the perceptions, attitudes and behaviours of Muslim women to their own intimate and sexual bodies through their experiences of menarche, marriage and reproduction and menopause. The intersection of religion and female sexual bodies and health is a neglected area within HCI and we highlight how inextricably specific Islamic values are linked with women’s reproductive health in Muslim communities. We further discuss the opportunities and challenges of designing technologies for religious, non-secular beliefs and values with the aim to improve intimate health practices amongst Muslim women and to broaden the scope of health design within HCI.

Nurses frequently transfer patients as part of their daily work. However, manual patient transfers pose a major risk to nurses’ health. Although the Kinaesthetics care conception can help address this issue, existing support to learn the concept is low. We present KiTT, a tablet-based system, to promote the learning of ergonomic patient transfers based on the Kinaesthetics care conception. KiTT supports the training of Kinaesthetics-based patient transfers by two nurses. The nurses are guided by the phases (i) interactive instructions, (ii) training of transfer conduct, and (iii) feedback and reflection. We evaluated KiTT with 26 nursing-care students in a nursing-care school. Our results indicate that KiTT provides a good subjective support for the learning of Kinaesthetics. Our results also suggest that KiTT can promote the ergonomically correct conduct of patient transfers while providing a good user experience adequate to the nursing-school context, and reveal how KiTT can extend existing practices.

Studies show that users do not reliably click more often on headlines classified as clickbait by automated classifiers. Is this because the linguistic criteria (e.g., use of lists or questions) emphasized by the classifiers are not psychologically relevant in attracting interest, or because their classifications are confounded by other unknown factors associated with assumptions of the classifiers? We address these possibilities with three studies—a quasi-experiment using headlines classified as clickbait by three machine-learning models (Study 1), a controlled experiment varying the headline of an identical news story to contain only one clickbait characteristic (Study 2), and a computational analysis of four classifiers using real-world sharing data (Study 3). Studies 1 and 2 revealed that clickbait did not generate more curiosity than non-clickbait. Study 3 revealed that while some headlines generate more engagement, the detectors agreed on a classification only 47% of the time, raising fundamental questions about their validity.

How to attribute responsibility for autonomous artificial intelligence (AI) systems’ actions has been widely debated across the humanities and social science disciplines. This work presents two experiments (N=200 each) that measure people’s perceptions of eight different notions of moral responsibility concerning AI and human agents in the context of bail decision-making. Using real-life adapted vignettes, our experiments show that AI agents are held causally responsible and blamed similarly to human agents for an identical task. However, there was a meaningful difference in how people perceived these agents’ moral responsibility; human agents were ascribed to a higher degree of present-looking and forward-looking notions of responsibility than AI agents. We also found that people expect both AI and human decision-makers and advisors to justify their decisions regardless of their nature. We discuss policy and HCI implications of these findings, such as the need for explainable AI in high-stakes scenarios.

In order to support fairness-forward thinking by machine learning (ML) practitioners, fairness researchers have created toolkits that aim to transform state-of-the-art research contributions into easily-accessible APIs. Despite these efforts, recent research indicates a disconnect between the needs of practitioners and the tools offered by fairness research. By engaging 20 ML practitioners in a simulated scenario in which they utilize fairness toolkits to make critical decisions, this work aims to utilize practitioner feedback to inform recommendations for the design and creation of fair ML toolkits. Through the use of survey and interview data, our results indicate that though fair ML toolkits are incredibly impactful on users’ decision-making, there is much to be desired in the design and demonstration of fairness results. To support the future development and evaluation of toolkits, this work offers a rubric that can be used to identify critical components of Fair ML toolkits.

With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models. Although many supposedly interpretable models have been proposed, there have been relatively few experimental studies investigating whether these models achieve their intended effects, such as making people more closely follow a model’s predictions when it is beneficial for them to do so or enabling them to detect when a model has made a mistake. We present a sequence of pre-registered experiments (N = 3, 800) in which we showed participants functionally identical models that varied only in two factors commonly thought to make machine learning models more or less interpretable: the number of features and the transparency of the model (i.e., whether the model internals are clear or black box). Predictably, participants who saw a clear model with few features could better simulate the model’s predictions. However, we did not find that participants more closely followed its predictions. Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model’s sizable mistakes, seemingly due to information overload. These counterintuitive findings emphasize the importance of testing over intuition when developing interpretable models.

In Human-AI collaborative settings that are inherently interactive, direction of communication plays a role in how users perceive their AI partners. In an AI-driven cooperative game with partially observable information, players (be it the AI or the human player) require their actions to be interpreted accurately by the other player to yield a successful outcome. In this paper, we investigate social perceptions of AI agents with various directions of communication in a cooperative game setting. We measure subjective social perceptions (rapport, intelligence, and likeability) of participants towards their partners when participants believe they are playing with an AI or with a human and the nature of the communication (responsiveness and leading roles). We ran a large scale study on Mechanical Turk (n=199) of this collaborative game and find significant differences in gameplay outcome and social perception across different AI agents, different directions of communication and when the agent is perceived to be an AI/Human. We find that the bias against the AI that has been demonstrated in prior studies varies with the direction of the communication and with the AI agent.

Artificial Intelligence (AI) education is an increasingly popular topic area for K-12 teachers. However, little research has investigated how AI curriculum and tools can be designed to be more accessible to all teachers and learners. In this study, we take a Value-Sensitive Design approach to understanding the role of teacher values in the design of AI curriculum and tools, and identifying opportunities to integrate AI into core curriculum to leverage learners’ interests. We organized co-design workshops with 15 K-12 teachers, where teachers and researchers co-created lesson plans using AI tools and embedding AI concepts into various core subjects. We found that K-12 teachers need additional scaffolding in AI tools and curriculum to facilitate ethics and data discussions, and value supports for learner evaluation and engagement, peer-to-peer collaboration, and critical reflection. We present an exemplar lesson plan that shows entry points for teaching AI in non-computing subjects and reflect on co-designing with K-12 teachers in a remote setting.

In this paper, we report on perceptual experiments indicating that there are distinct and quantitatively measurable differences in the way we visually perceive genuine versus face-swapped videos.

Recent progress in deep learning has made face-swapping techniques a powerful tool for creative purposes, but also a means for unethical forgeries. Currently, it remains unclear why people are misled, and which indicators they use to recognize potential manipulations. Here, we conduct three perceptual experiments focusing on a wide range of aspects: the conspicuousness of artifacts, the viewing behavior using eye tracking, the recognition accuracy for different video lengths, and the assessment of emotions.

Our experiments show that responses differ distinctly when watching manipulated as opposed to original faces, from which we derive perceptual cues to recognize face swaps. By investigating physiologically measurable signals, our findings yield valuable insights that may also be useful for advanced algorithmic detection.

Domestic robots such as vacuum cleaners or lawnmowers are becoming popular consumer products in private homes, but while current HCI research on domestic robots has highlighted for example personalisation, long-term effects, or design guidelines, little attention has been paid to automation. To address this, we conducted a qualitative study with 24 participants in private households using interviews, contextual technology tours, and robot deployment. Through thematic analysis we identified three themes related to 1) work routines and automation, 2) domestic robot automation and the physical environment, as well as 3) interaction and breakdown intervention. We present an empirical understanding of how task automation using domestic robots can be implemented in the home. Lastly, we discuss our findings in relation to existing literature and highlight three opportunities for improved task automation using domestic robots for future research.

This work investigates the practices and challenges of voice user interface (VUI) designers. Existing VUI design guidelines recommend that designers strive for natural human-agent conversation. However, the literature leaves a critical gap regarding how designers pursue naturalness in VUIs and what their struggles are in doing so. Bridging this gap is necessary for identifying designers’ needs and supporting them. Our interviews with 20 VUI designers identified 12 ways that designers characterize and approach naturalness in VUIs. We categorized these characteristics into three groupings based on the types of conversational context that each characteristic contributes to: Social, Transactional, and Core. Our results contribute new findings on designers’ challenges, such as a design dilemma in augmenting task-oriented VUIs with social conversations, difficulties in writing for spoken language, lack of proper tool support for imbuing synthesized voice with expressivity, and implications for developing design tools and guidelines.

Pair programming has a documented history of benefits, such as increased code quality, productivity, self-efficacy, knowledge transfer, and reduced gender gap. Research uncovered problems with pair programming related to scheduling, collocating, role imbalance, and power dynamics. We investigated the trade-offs of substituting a human with an agent to simultaneously provide benefits and alleviate obstacles in pair programming. We conducted gender-balanced studies with human-human pairs in a remote lab with 18 programmers and Wizard-of-Oz studies with 14 programmers, then analyzed results quantitatively and qualitatively. Our comparative analysis of the two studies showed no significant differences in productivity, code quality, and self-efficacy. Further, agents facilitated knowledge transfer; however, unlike humans, agents were unable to provide logical explanations or discussions. Human partners trusted and showed humility towards agents. Our results demonstrate that agents can act as effective pair programming partners and open the way towards new research on conversational agents for programming.

Perceptions of system competence and communicative ability, termed partner models, play a significant role in speech interface interaction. Yet we do not know what the core dimensions of this concept are. Taking a psycholexical approach, our paper is the first to identify the key dimensions that define partner models in speech agent interaction. Through a repertory grid study (N=21), a review of key subjective questionnaires, an expert review of resulting word pairs and an online study of 356 users of speech interfaces, we identify three key dimensions that make up a users’ partner model: 1) perceptions towards partner competence and dependability; 2) assessment of human-likeness; and 3) a system’s perceived cognitive flexibility. We discuss the implications for partner modelling as a concept, emphasising the importance of salience and the dynamic nature of these perceptions.

The uptake of artificial intelligence-based applications raises concerns about the fairness and transparency of AI behaviour. Consequently, the Computer Science community calls for the involvement of the general public in the design and evaluation of AI systems. Assessing the fairness of individual predictors is an essential step in the development of equitable algorithms. In this study, we evaluate the effect of two common visualisation techniques (text-based and scatterplot) and the display of the outcome information (i.e., ground-truth) on the perceived fairness of predictors. Our results from an online crowdsourcing study (N = 80) show that the chosen visualisation technique significantly alters people’s fairness perception and that the presented scenario, as well as the participant’s gender and past education, influence perceived fairness. Based on these results we draw recommendations for future work that seeks to involve non-experts in AI fairness evaluations.

Current personal informatics models consider reflection as an important stage in users’ journeys with trackers. However, these models describe reflection from a meta perspective and it remains unclear what this stage entails. To design interactive technologies that support reflection, we need a more thorough understanding of how people reflect on their personal data in practice. To that end, we conducted semi-structured interviews with users of fitness trackers and an online survey to study practices in reflecting on fitness data. Our results show that users reported reflecting on data despite lacking reflection support from their tracking technology. Based on our results, we introduce the Technology-Mediated Reflection Model, which describes conditions and barriers for reflection on personal data. Our model consists of the temporal and conceptual cycles of reflection and helps designers identify the possible barriers a user might face when using a system for reflection.

The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N=175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N=117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.

There are many situations where using personal devices is not socially acceptable, or where nearby people present a privacy risk. For these situations, we explore the concept of hidden interaction techniques through two prototype applications. HiddenHaptics allows users to receive information through vibrotactile cues on a smartphone, and HideWrite allows users to write text messages by drawing on a dimmed smartwatch screen. We conducted three user studies to investigate whether, and how, these techniques can be used without being exposed. Our primary findings are (1) users can effectively hide their interactions while attending to a social situation, (2) users seek to interact when another person is speaking, and they also tend to hide the interaction using their body or furniture, and (3) users can sufficiently focus on the social situation despite their interaction, whereas non-users feel that observing the user hinders their ability to focus on the social activity.

Speech is now common in daily interactions with our devices, thanks to voice user interfaces (VUIs) like Alexa. Despite their seeming ubiquity, designs often do not match users’ expectations. Science fiction, which is known to influence design of new technologies, has included VUIs for decades. Star Trek: The Next Generation is a prime example of how people envisioned ideal VUIs. Understanding how current VUIs live up to Star Trek’s utopian technologies reveals mismatches between current designs and user expectations, as informed by popular fiction. Combining conversational analysis and VUI user analysis, we study voice interactions with the Enterprise’s computer and compare them to current interactions. Independent of futuristic computing power, we find key design-based differences: Star Trek interactions are brief and functional, not conversational, they are highly multimodal and context-driven, and there is often no spoken computer response. From this, we suggest paths to better align VUIs with user expectations.

Voice assistants are fundamentally changing the way we access information. However, voice assistants still leverage little about the web beyond simple search results. We introduce Firefox Voice, a novel voice assistant built on the open web ecosystem with an aim to expand access to information available via voice. Firefox Voice is a browser extension that enables users to use their voice to perform actions such as setting timers, navigating the web, and reading a webpage’s content aloud. Through an iterative development process and use by over 12,000 active users, we find that users see voice as a way to accomplish certain browsing tasks efficiently, but struggle with discovering functionality and frequently discontinue use. We conclude by describing how Firefox Voice enables the development of novel, open web-powered voice-driven experiences.

Silent speech input converts non-acoustic features like tongue and lip movements into text. It has been demonstrated as a promising input method on mobile devices and has been explored for a variety of audiences and contexts where the acoustic signal is unavailable (e.g., people with speech disorders) or unreliable (e.g., noisy environment). Though the method shows promise, very little is known about peoples’ perceptions regarding using it. In this work, first, we conduct two user studies to explore users’ attitudes towards the method with a particular focus on social acceptance and error tolerance. Results show that people perceive silent speech as more socially acceptable than speech input and are willing to tolerate more errors with it to uphold privacy and security. We then conduct a third study to identify a suitable method for providing real-time feedback on silent speech input. Results show users find an abstract feedback method effective and significantly more private and secure than a commonly used video feedback method.

Video-conferencing is essential for many companies, but its limitations in conveying social cues can lead to ineffective meetings. We present MeetingCoach, an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Through an exploratory survey (N=120), we identified important signals (e.g., turn taking, sentiment) and used these insights to create a wireframe dashboard. The design was evaluated with in situ participants (N=16) who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, we developed an AI system to quantify the meeting features and created personalized dashboards for each participant. Through interviews and surveys (N=23), we found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity. Based on our findings, we provide suggestions for future feedback system designs of video-conferencing meetings.

Virtual environments (VEs) can create collaborative and social spaces, which are increasingly important in the face of remote work and travel reduction. Recent advances, such as more open and widely available platforms, create new possibilities to observe and analyse interaction in VEs. Using a custom instrumented build of Mozilla Hubs to measure position and orientation, we conducted an academic workshop to facilitate a range of typical workshop activities. We analysed social interactions during a keynote, small group breakouts, and informal networking/hallway conversations. Our mixed-methods approach combined environment logging, observations, and semi-structured interviews. The results demonstrate how small and large spaces influenced group formation, shared attention, and personal space, where smaller rooms facilitated more cohesive groups while larger rooms made small group formation challenging but personal space more flexible. Beyond our findings, we show how the combination of data and insights can fuel collaborative spaces’ design and deliver more effective virtual workshops.

We present a dialogue elicitation study to assess how users envision conversations with a perfect voice assistant (VA). In an online survey, N=205 participants were prompted with everyday scenarios, and wrote the lines of both user and VA in dialogues that they imagined as perfect. We analysed the dialogues with text analytics and qualitative analysis, including number of words and turns, social aspects of conversation, implied VA capabilities, and the influence of user personality. The majority envisioned dialogues with a VA that is interactive and not purely functional; it is smart, proactive, and has knowledge about the user. Attitudes diverged regarding the assistant’s role as well as it expressing humour and opinions. An exploratory analysis suggested a relationship with personality for these aspects, but correlations were low overall. We discuss implications for research and design of future VAs, underlining the vision of enabling conversational UIs, rather than single command “Q&As”.

The use of virtual reality (VR) to simulate confrontational human behaviour has significant potential for use in training, where the recreation of uncomfortable feelings may help users to prepare for challenging real-life situations. In this paper we present a user study (n=68) in which participants experienced simulated confrontational behaviour performed by a virtual character either in immersive VR, or on a 2D display. Participants reported a higher elevation in anxiety in VR, which correlated positively with a perceived sense of physical space. Character believability was influenced negatively by visual elements of the simulation, and positively by behavioural elements, which complements findings from previous work. We recommend the use of VR for simulations of confrontational behaviour, where a realistic emotional response is part of the intended experience. We also discuss incorporation of domain knowledge of human behaviours, and carefully crafted motion-captured sequences, to increase users’ sense of believability.

Bringing positive experiences to users is one of the key goals when designing conversational agents (CAs). Yet we still lack an understanding of users’ underlying needs to achieve positive experiences and how to support them in design. This research first applies Self-Determination Theory in an interview study to explore how users’ needs of competence, autonomy and relatedness could be supported or undermined in CA experiences. Ten guidelines are then derived from the interview findings. The key findings demonstrate that: competence is affected by users’ knowledge of the CA capabilities and effectiveness of the conversation; autonomy is influenced by flexibility of the conversation, personalisation of the experiences, and control over user data; regarding relatedness, users still have concerns over integrating social features into CAs. The guidelines recommend how to inform users about the system capabilities, design effective and socially appropriate conversations, and support increased system intelligence, customisation, and data transparency.

Although social support is important for health and well-being, many young people are hesitant to reach out for support. The emerging uptake of chatbots for social and emotional purposes entails opportunities and concerns regarding non-human agents as sources of social support. To explore this, we invited 16 participants (16–21 years) to use and reflect on chatbots as sources of social support. Our participants first interacted with a chatbot for mental health (Woebot) for two weeks. Next, they participated in individual in-depth interviews. As part of the interview session, they were presented with a chatbot prototype providing information to young people. Two months later, the participants reported on their continued use of Woebot. Our findings provide in-depth knowledge about how young people may experience various types of social support—appraisal, informational, emotional, and instrumental support—from chatbots. We summarize implications for theory, practice, and future research.

This paper traces different conceptions of the body in HCI and identifies a narrative from user to body, body to bodies, and bodies to more-than-human bodies. Firstly, this paper aims to present a broader, updated, survey of work around the body in HCI. The overview shows how bodies are conceptualized as performative, sensing, datafied, intersectional and more-than-human. This paper then diverges from similar surveys of research addressing the body in HCI in that it is more disruptive and offers a critique of these approaches and pointers for where HCI might go next. We end our paper with recommendations drawn from across the different approaches to the body in HCI. In particular, that researchers working with the body have much to gain from the 4th wave HCI approach when designing with and for the body, where our relationships with technologies are understood as entangled and the body is always more-than-human.

Passwords have become a ubiquitous part of our everyday lives, needed for every web-service and system. However, it is challenging to create safe and diverse alphanumeric passwords, and to recall them, imposing a cognitive burden on the user. Through consecutive experiments, we explored the movement space, affordances and interaction, and memorability of a tangible, handheld, embodied password. In this context, we found that: (1) a movement space of 200 mm × 200 mm is preferred; (2) each context has a perceived level of safety, which—together with the affordances and link to familiarity—influences how the password is performed. Furthermore, the artefact’s dimensions should be balanced within the design itself, with the user, and the context, but there is a trade-off between the perceived safety and ergonomics; and (3) the designed embodied passwords can be recalled for at least a week, with participants creating unique passwords which were reproduced consistently.

Starting to menstruate can restrict adolescents’ movements due to physiological changes and societal stigma. We present a participatory soma design project advocating for young adolescents to listen to and care for their newly-menstruating bodies, specifically focusing on participation in sport. We designed Menarche Bits, an open-ended prototyping toolkit consisting of shape-changing actuators and heat pads, and used it in two design workshops with seven participants aged 16-18, as part of collaboration and menstrual advocacy in their sports clubs and high school. The participants designed menstrual technologies that respond to menstrual cramps and depressive, anxious feelings before menstruating. We contribute findings on designing menstrual technologies with adolescents using participatory soma design. We found that a toolkit approach to the design of menstrual technologies can allow for pluralist experiences of menstrual cycles. In addition, we found that participatory design with adolescents benefits from drawing on qualities of embodiment and participants’ own body literacy.

We introduce Azalea: a design to enrich remote dialog by diminishing externalities, created through a process informed by somaesthetics. Azalea is a tactile cushion that envelops a smartphone running a bespoke app. A pair of Azaleas mediate an embodied co-experience between remote interlocutors via a motion-driven soundscape and audio-driven visuals. While most designs for enriching remote communication increase dimensionality and fidelity of modalities, Azalea diminishes distractions and serves an abstract medium for co-experiencing embodied information. We present the theoretical foundations and design tactics of Azalea, and characterize the experience through a qualitative empirical study. Our findings culminated in 12 qualities, supporting 5 themes with design implications that contribute to (1) a design ethos of diminished reality and (2) an expansion of somaesthetic HCI towards expression and communication.

Designing computational support for dance is an emerging area of HCI research, incorporating the cultural, experiential, and embodied characteristics of the third-wave shift. The challenges of recognising the abstract qualities of body movement, and of mediating between the diverse parties involved in the idiosyncratic creative process, present important questions to HCI researchers: how can we effectively integrate computing with dance, to understand and cultivate the felt dimension of creativity, and to aid the dance-making process? In this work, we systematically review the past twenty years of dance literature in HCI. We discuss our findings, propose directions for future HCI works in dance, and distil lessons for related disciplines.

Deep Pressure Therapy relies on exerting firm touch to help individuals with sensory sensitivity. We performed first-person explorations of deep pressure enabled by shape-changing actuation driven by breathing sensing. This revealed a novel design space with rich, evocative, aesthetically interesting interactions that can help increase breathing awareness and appreciation through: (1) applying symmetrical as well as asymmetrical pressure on the torso; (2) using pressure to direct attention to muscles or bone structure involved in different breathing patterns; (3) apply synchronous as well as asynchronous feedback following or opposing the user’s breathing rhythm through applying rhythmic pressure. Taken together these explorations led us to design (4) breathing correspondence interactions – a balance point right between leading and following users’ breathing patterns by first applying deep pressure – almost to the point of being unpleasant – and then releasing in rhythmic flow.

Avatars, the users’ virtual representations, are becoming ubiquitous in virtual reality applications. In this context, the avatar becomes the medium which enables users to manipulate objects in the virtual environment. It also becomes the users’ main spatial reference, which can not only alter their interaction with the virtual environment, but also the perception of themselves. In this paper, we review and analyse the current state-of-the-art for 3D object manipulation and the sense of embodiment. Our analysis is twofold. First, we discuss the impact that the avatar can have on object manipulation. Second, we discuss how the different components of a manipulation technique (i.e. input, control and feedback) can influence the user’s sense of embodiment. Throughout the analysis, we crystallise our discussion with practical guidelines for VR application designers and we propose several research topics towards “avatar-friendly’’ manipulation techniques.

Meditation is a mind-body practice with considerable wellbeing benefits that can take different forms. Novices usually start with focused attention meditation that supports regulation of attention towards an inward focus or internal bodily sensations and away from external stimuli or distractors. Most meditation technologies employ metaphorical mappings of meditative states to visual or soundscape representations to support awareness of mind wandering and attention regulation, although the rationale for such mappings is seldom articulated. Moreover, such external modalities also take the focus attention away from the body. We advance the concept of interoceptive interaction and employed the embodied metaphor theory to explore the design of mappings to the interoceptive sense of thermoception. We illustrate this concept with WarmMind, an on-body interface integrating heat actuators for mapping meditation states. We report on an exploratory study with 10 participants comparing our novel thermal metaphors for mapping meditation states with comparable ones, albeit in aural modality, as provided by Muse meditation app. Findings indicate a tension between the highly discoverable soundscape's metaphors which however hinder attention regulation, and the ambiguous thermal metaphors experienced as coming from the body and supported attention regulation. We discuss the qualities of embodied metaphors underpinning this tension and propose an initial framework to inform the design of metaphorical mappings for meditation technologies.

Though seemingly straightforward and habitual, breathing is a complex bodily function. Problematising the space of designing for breathing as a non-habitual act pertaining to different bodies or situations, we conducted a soma design exploration together with a classical singer. Reflecting on how sensors could capture the impact and somatic experience of being sensed led us to develop a new sensing mechanism using shape-change technologies integrated in the Breathing Shell: a wearable that evokes a reciprocal experience of “feeling the sensor feeling you” when breathing. We contribute with two design implications: 1) Enabling reflections of the somatic impact of being sensed in tandem with the type of data captured, 2) creating a tactile impact of the sensor data on the body. Both implications aim to deepen one’s understanding of how the whole soma relates to or with biosensors and ultimately leading to designing for symbiotic experiences between biosensors and bodies.

The growing interest in wearables for sports and fitness calls for design knowledge and conceptualizations that can help shape future designs. Towards that end, we present and discuss a design space of wearables for these practices, based on a survey of previous work. Through a thematic analysis of 47 research publications in the domain, we surface core design decisions concerning wearability, technology design, and wearable use in practice. Building on these, we show how the design space takes into account the goals of introducing technology, that design decisions can be either directly designed, or left open for appropriation by end-users; and the social organization of the practice. We characterize prior work based on the design space elements, which yields trends and opportunities for design. Our contributions can help designers think about key design decisions, exploit trends and explore new areas in the domain of wearables for sports and fitness practices.

In this paper, we reflect on the applicability of the concept of trajectories to soma design. Soma design is a first-person design method which considers users’ subjective somatic or bodily experiences of a design. Due to bodily changes over time, soma experiences are inherently temporal. Current instruments for articulating soma experiences lack the power to express the effects of experiences on the body over time. To address this, we turn to trajectories, a well-known concept in the HCI community, as a way of mapping this aspect of soma experience. By showing trajectories through a range of dimensions, we can articulate individual experiences and differences in those experiences. Through analysis of a set of soma experience designs and a set of temporal dimensions within the experiences, this paper demonstrates how trajectories can provide a practical conceptual framing for articulating the temporal complexity of soma designs.

We examine a vocabulary of affective language, borrowed from Roland Barthes’ “A Lover’s Discourse: Fragments,” and its applicability to discourse describing smartphone attachment. This vocabulary, adopted from four of Barthes’ terms, waiting, dependency, anxiety, and absence, is used as a discursive lens to illustrate some of the many ways people understand and engage with their relationships to their smartphones. Based on this, a survey is conducted, and a speculative technology probe is created in the form of an instrumented pillow for people to lock away their smartphones during the night. The pillow is deployed in a diary study in which five people sleep with their phone locked away for multiple nights. The self-reported and observed behaviours are presented in a selection of vignettes. The results support the proposed discursive lens and suggest future interdisciplinary strategies to investigate how people relate to interactive technology, using a combined approach of literary theory and a technology probe supported by survey and study data.

Two-dimensional canvases are the core components of many digital productivity and creativity tools, with “artboards” containing objects rather than pixels. Unfortunately, the contents of artboards remain largely inaccessible to blind users relying on screen-readers, but the precise problems are not well understood. This study sought to understand how blind screen-reader users interact with artboards. Specifically, we conducted contextual interviews, observations, and task-based usability studies with 15 blind participants to understand their experiences of artboards found in Microsoft PowerPoint, Apple Keynote, and Google Slides. Participants expressed that the inaccessibility of these artboards contributes to significant educational and professional barriers. We found that the key problems faced were: (1) high cognitive loads from a lack of feedback about artboard contents and object state; (2) difficulty determining relationships among artboard objects; and (3) constant uncertainty about whether object manipulations were successful. We offer design remedies that improve feedback for object state, relationships, and manipulations.

The development of accurate machine learning models for sign languages like American Sign Language (ASL) has the potential to break down communication barriers for deaf signers. However, to date, no such models have been robust enough for real-world use. The primary barrier to enabling real-world applications is the lack of appropriate training data. Existing training sets suffer from several shortcomings: small size, limited signer diversity, lack of real-world settings, and missing or inaccurate labels. In this work, we present ASL Sea Battle, a sign language game designed to collect datasets that overcome these barriers, while also providing fun and education to users. We conduct a user study to explore the data quality that the game collects, and the user experience of playing the game. Our results suggest that ASL Sea Battle can reliably collect and label real-world sign language videos, and provides fun and education at the expense of data throughput.

User-generated videos are an increasingly important source of information online, yet most online videos are inaccessible to blind and visually impaired (BVI) people. To find videos that are accessible, or understandable without additional description of the visual content, BVI people in our formative studies reported that they used a time-consuming trial-and-error approach: clicking on a video, watching a portion, leaving the video, and repeating the process. BVI people also reported video accessibility heuristics that characterize accessible and inaccessible videos. We instantiate 7 of the identified heuristics (2 audio-related, 2 video-related, and 3 audio-visual) as automated metrics to assess video accessibility. We collected a dataset of accessibility ratings of videos by BVI people and found that our automatic video accessibility metrics correlated with the accessibility ratings (Adjusted R2 = 0.642). We augmented a video search interface with our video accessibility metrics and predictions. BVI people using our augmented video search interface selected an accessible video more efficiently than when using the original search interface. By integrating video accessibility metrics, video hosting platforms could help people surface accessible videos and encourage content creators to author more accessible products, improving video accessibility for all.

Despite recent improvements in online accessibility, the Internet remains an inhospitable place for users with photosensitive epilepsy, a chronic condition in which certain light stimuli can trigger seizures and even lead to death in extreme cases. In this paper, we explore how current risk detection systems have allowed attackers to take advantage of design oversights and target vulnerable users with photosensitivity on popular social media platforms. Through interviews with photosensitive individuals and a critical review of existing systems, we constructed design requirements for consumer-driven protective systems and developed a prototype browser extension for actively detecting and disarming potentially seizure-inducing GIFs and videos. We validate our system with a comprehensive dataset of simulated GIFs and GIFs collected from social media. Finally, we conduct a novel quantitative analysis of the prevalence of seizure-inducing GIFs across popular social media platforms and contribute recommendations for improving online accessibility for individuals with photosensitivity. All study materials are available at https://osf.io/5a3dy/.

For 15% of the world population with disabilities, accessibility is arguably the most critical software quality attribute. The ever-growing reliance of users with disability on mobile apps further underscores the need for accessible software in this domain. Existing automated accessibility assessment techniques primarily aim to detect violations of predefined guidelines, thereby produce a massive amount of accessibility warnings that often overlook the way software is actually used by users with disability. This paper presents a novel, high-fidelity form of accessibility testing for Android apps, called Latte, that automatically reuses tests written to evaluate an app’s functional correctness to assess its accessibility as well. Latte first extracts the use case corresponding to each test, and then executes each use case in the way disabled users would, i.e., using assistive services. Our empirical evaluation on real-world Android apps demonstrates Latte’s effectiveness in detecting substantially more useful defects than prior techniques.

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Presenters commonly use slides as visual aids for informative talks. When presenters fail to verbally describe the content on their slides, blind and visually impaired audience members lose access to necessary content, making the presentation difficult to follow. Our analysis of 90 presentation videos revealed that 72% of 610 visual elements (e.g., images, text) were insufficiently described. To help presenters create accessible presentations, we introduce Presentation A11y, a system that provides real-time and post-presentation accessibility feedback. Our system analyzes visual elements on the slide and the transcript of the verbal presentation to provide element-level feedback on what visual content needs to be further described or even removed. Presenters using our system with their own slide-based presentations described more of the content on their slides, and identified  3.26 times more accessibility problems to fix after the talk than when using a traditional slide-based presentation interface. Integrating accessibility feedback into content creation tools will improve the accessibility of informational content for all.

Video accessibility is essential for people with visual impairments. Audio descriptions describe what is happening on-screen, e.g., physical actions, facial expressions, and scene changes. Generating high-quality audio descriptions requires a lot of manual description generation [50]. To address this accessibility obstacle, we built a system that analyzes the audiovisual contents of a video and generates the audio descriptions. The system consisted of three modules: AD insertion time prediction, AD generation, and AD optimization. We evaluated the quality of our system on five types of videos by conducting qualitative studies with 20 sighted users and 12 users who were blind or visually impaired. Our findings revealed how audio description preferences varied with user types and video types. Based on our study’s analysis, we provided recommendations for the development of future audio description generation technologies.

This paper presents a systematic literature review of 292 publications from 97 unique venues on touch-based graphics for people who are blind or have low vision, from 2010 to mid-2020. It is the first review of its kind on touch-based accessible graphics. It is timely because it allows us to assess the impact of new technologies such as commodity 3D printing and low-cost electronics on the production and presentation of accessible graphics. As expected our review shows an increase in publications from 2014 that we can attribute to these developments. It also reveals the need to: broaden application areas, especially to the workplace; broaden end-user participation throughout the full design process; and conduct more in situ evaluation. This work is linked to an online living resource to be shared with the wider community.

Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.

The accessibility and affordability of tangible electronic toolkits are significant barriers to their uptake by people with disabilities. We present the design and evaluation of TapeBlocks, a low-cost, low-fidelity toolkit intended to be accessible for people with intellectual disabilities while promoting creativity and engagement. We evaluated TapeBlocks by interviewing makers, special educational needs teachers and support coaches. Analysis of these interviews informed the design of a series of maker workshops using TapeBlocks with young adults living with intellectual disabilities, led by support coaches with support from the research team. Participants were able to engage with TapeBlocks and making, eventually building their own TapeBlocks to make personal creations. Our evaluation reveals how TapeBlocks supports accessible making and playful discovery of electronics for people living with disabilities, and addresses a gap in existing toolkits by being tinkerable, affordable and having a low threshold for engagement.

The recently advanced robotics technology enables robots to assist users in their daily lives. Haptic guidance (HG) improves users’ task performance through physical interaction between robots and users. It can be classified into optimal action-based HG (OAHG), which assists users with an optimal action, and user prediction-based HG (UPHG), which assists users with their next predicted action. This study aims to understand the difference between OAHG and UPHG and propose a combined HG (CombHG) that achieves optimal performance by complementing each HG type, which has important implications for HG design. We propose implementation methods for each HG type using deep learning-based approaches. A user study (n=20) in a haptic task environment indicated that UPHG induces better subjective evaluations, such as naturalness and comfort, than OAHG. In addition, the CombHG that we proposed further decreases the disagreement between the user intention and HG, without reducing the objective and subjective scores.

To make off-screen interaction without specialized hardware practical, we investigate using deep learning methods to process the common built-in IMU sensor (accelerometers and gyroscopes) on mobile phones into a useful set of one-handed interaction events. We present the design, training, implementation and applications of TapNet, a multi-task network that detects tapping on the smartphone. With phone form factor as auxiliary information, TapNet can jointly learn from data across devices and simultaneously recognize multiple tap properties, including tap direction and tap location. We developed two datasets consisting of over 135K training samples, 38K testing samples, and 32 participants in total. Experimental evaluation demonstrated the effectiveness of the TapNet design and its significant improvement over the state of the art. Along with the datasets, codebase1, and extensive experiments, TapNet establishes a new technical foundation for off-screen mobile input.

Typing on wearables while situationally impaired, such as while walking, is challenging. However, while HCI research on wearable typing is diverse, existing work focuses on stationary scenarios and fine-grained input that will likely perform poorly when users are on-the-go. To address this issue we explore single-handed wearable typing using intra-hand touches between the thumb and fingers, a modality we argue will be robust to the physical disturbances inherent to input while mobile. We first examine the impact of walking on performance of these touches, noting no significant differences in accuracy or speed, then feed our study data into a multi-objective optimization process in order to design keyboard layouts (for both five and ten keys) capable of supporting rapid, accurate, comfortable, and unambiguous typing. A final study tests these layouts against QWERTY baselines and reports performance improvements of up to 10.45% WPM and 39.44% WER when users type while walking.

We present a computational approach to haptic design embedded in everyday tangible interaction with digital fabrication. To generate haptic feedback, the use of permanent magnets as the mechanism potentially contributes to simpleness and robustness; however, it is difficult to manually design how magnets should be embedded in the objects. Our approach enables the inverse design of magnetic force feedback; that is, we computationally solve an inverse problem to obtain an optimal arrangement of permanent magnets that renders the user-specified haptic sensation. To solve the inverse problem in a practical manner, we also present techniques on magnetic simulation and optimization. We demonstrate applications to explore the design possibility of augmenting digital fabrication for everyday use.

Motion correlation interfaces are those that present targets moving in different patterns, which the user can select by matching their motion. In this paper, we re-formulate the task of target selection as a probabilistic inference problem. We demonstrate that previous interaction techniques can be modelled using a Bayesian approach and that how modelling the selection task as transmission of information can help us make explicit the assumptions behind similarity measures. We propose ways of incorporating uncertainty into the decision-making process and demonstrate how the concept of entropy can illuminate the measurement of the quality of a design. We apply these techniques in a case study and suggest guidelines for future work.

We present a novel simulation model of point-and-click behaviour that is applicable both when a target is stationary or moving. To enable more realistic simulation than existing models, the model proposed in this study takes into account key features of the user and the external environment, such as intermittent motor control, click decision-making, visual perception, upper limb kinematics and the effect of input device. The simulated user’s point-and-click behaviour is formulated as a Markov decision process (MDP), and the user’s policy of action is optimised through deep reinforcement learning. As a result, our model successfully and accurately reproduced the trial completion time, distribution of click endpoints, and cursor trajectories of real users. Through an ablation study, we showed how the simulation results change when the model’s sub-modules are individually removed. The implemented model and dataset are publicly available.

Typing with ten fingers on a virtual keyboard in virtual or augmented reality exposes a challenging input interpretation problem. There are many sources of noise in this interaction context and these exacerbate the challenge of accurately translating human actions into text. A particularly challenging input noise source arises from the physiology of the hand. Intentional finger movements can produce unintentional coactivations in other fingers. On a physical keyboard, the resistance of the keys alleviates this issue. On a virtual keyboard, coactivations are likely to introduce spurious input events under a naïve solution to input detection. In this paper we examine the features that discriminate intentional activations from coactivations. Based on this analysis, we demonstrate three alternative coactivation detection strategies with high discrimination power. Finally, we integrate coactivation detection into a probabilistic decoder and demonstrate its ability to further reduce uncorrected character error rates by approximately 10% relative and 0.9% absolute.

Gaze-based selection has received significant academic attention over a number of years. While advances have been made, it is possible that further progress could be made if there were a deeper understanding of the adaptive nature of the mechanisms that guide eye movement and vision. Control of eye movement typically results in a sequence of movements (saccades) and fixations followed by a ‘dwell’ at a target and a selection. To shed light on how these sequences are planned, this paper presents a computational model of the control of eye movements in gaze-based selection. We formulate the model as an optimal sequential planning problem bounded by the limits of the human visual and motor systems and use reinforcement learning to approximate optimal solutions. The model accurately replicates earlier results on the effects of target size and distance and captures a number of other aspects of performance. The model can be used to predict number of fixations and duration required to make a gaze-based selection. The future development of the model is discussed.

Touch input is dominantly detected using mutual-capacitance sensing, which measures the proximity of close-by objects that change the electric field between the sensor lines. The exponential drop-off in intensities with growing distance enables software to detect touch events, but does not reveal true contact areas. In this paper, we introduce CapContact, a novel method to precisely infer the contact area between the user’s finger and the surface from a single capacitive image. At 8 × super-resolution, our convolutional neural network generates refined touch masks from 16-bit capacitive images as input, which can even discriminate adjacent touches that are not distinguishable with existing methods. We trained and evaluated our method using supervised learning on data from 10 participants who performed touch gestures. Our capture apparatus integrates optical touch sensing to obtain ground-truth contact through high-resolution frustrated total internal reflection. We compare our method with a baseline using bicubic upsampling as well as the ground truth from FTIR images. We separately evaluate our method’s performance in discriminating adjacent touches. CapContact successfully separated closely adjacent touch contacts in 494 of 570 cases (87%) compared to the baseline’s 43 of 570 cases (8%). Importantly, we demonstrate that our method accurately performs even at half of the sensing resolution at twice the grid-line pitch across the same surface area, challenging the current industry-wide standard of a ∼ 4 mm sensing pitch. We conclude this paper with implications for capacitive touch sensing in general and for touch-input accuracy in particular.

Arm discomfort is a common issue in Cross Reality applications involving prolonged mid-air interaction. Solving this problem is difficult because of the lack of tools and guidelines for 3D user interface design. Therefore, we propose a method to make existing ergonomic metrics available to creators during design by estimating the interaction cost at each reachable position in the user’s environment. We present XRgonomics, a toolkit to visualize the interaction cost and make it available at runtime, allowing creators to identify UI positions that optimize users’ comfort. Two scenarios show how the toolkit can support 3D UI design and dynamic adaptation of UIs based on spatial constraints. We present results from a walkthrough demonstration, which highlight the potential of XRgonomics to make ergonomics metrics accessible during the design and development of 3D UIs. Finally, we discuss how the toolkit may address design goals beyond ergonomics.

Hand gestures are a natural and expressive input method enabled by modern mixed reality headsets. However, it remains challenging for developers to create custom gestures for their applications. Conventional strategies to bespoke gesture recognition involve either hand-crafting or data-intensive deep-learning. Neither approach is well suited for rapid prototyping of new interactions. This paper introduces a flexible and efficient alternative approach for constructing hand gestures. We present Gesture Knitter: a design tool for creating custom gesture recognizers with minimal training data. Gesture Knitter allows the specification of gesture primitives that can then be combined to create more complex gestures using a visual declarative script. Designers can build custom recognizers by declaring them from scratch or by providing a demonstration that is automatically decoded into its primitive components. Our developer study shows that Gesture Knitter achieves high recognition accuracy despite minimal training data and delivers an expressive and creative design experience.

Millimeter wave (mmWave) Doppler radar is a new and promising sensing approach for human activity recognition, offering signal richness approaching that of microphones and cameras, but without many of the privacy-invading downsides. However, unlike audio and computer vision approaches that can draw from huge libraries of videos for training deep learning models, Doppler radar has no existing large datasets, holding back this otherwise promising sensing modality. In response, we set out to create a software pipeline that converts videos of human activities into realistic, synthetic Doppler radar data. We show how this cross-domain translation can be successful through a series of experimental results. Overall, we believe our approach is an important stepping stone towards significantly reducing the burden of training such as human sensing systems, and could help bootstrap uses in human-computer interaction.

Design research has recently turned to theoretical perspectives, including care ethics and posthumanism, to counter the industrial processes that have led to climate crisis. As design theorists and ethnographers of interaction, we researched experimental eco-farming in a community that shared many of these theoretical and ideological commitments. Our goal was not to offer an account of use and provide design implications in support of it. Instead, we chose to identify concrete practices and artifacts that embody the sorts of industrial transformations that we are seeking—even if they are manifest in an imperfect or partial form. We encountered practices focused on community building, local resilience to climate disruptions, experiments in eco-farming, economic survival, and attracting the next generation. One interlocutor translated these concerns into a simple binary, asking, “do we want to live here?” This paper contributes to a design research agenda that might (eventually) provide an affirmative answer.

Guqin is a plucked seven-string traditional Chinese musical instrument that exists for over 3,000 years. However, as an Intangible World Cultural Heritage, the inheritance of Guqin and its culture in modern society is in deep danger. According to our study with 1,006 Chinese worldwide, Guqin as an instrument is not well-known and barely accessible. To better promote Guqin, we developed two interactive systems: VirGuqin and MRGuqin. VirGuqin was developed using a low-cost motion tracking device and was tested in a museum. 89% of 308 participants expressed an increase in interest in learning Guqin after using our system. MRGuqin was developed as a mixed reality learning environment to reduce the entry barrier to Guqin, and was tested by 16 participants, allowing them to learn Guqin significantly faster and perform better than the current practice. Our study demonstrates how technology can be used to help the inheritance of this dying art.

HCI and STS researchers have previously described the ethical complexity of practice, drawing together aspects of organizational complexity, design knowledge, and ethical frameworks. Building on this work, we investigate the identity claims and beliefs that impact practitioners’ ability to recognize and act upon ethical concerns in a range of technology-focused disciplines. In this paper, we report results from an interview study with 12 practitioners, identifying and describing their identity claims related to ethical awareness and action. We conducted a critically-focused thematic analysis to identify eight distinct claims representing roles relating to learning, educating, following policies, feeling a sense of responsibility, being a member of a profession, a translator, an activist, and deliberative. Based on our findings, we demonstrate how the claims foreground building competence in relation to ethical practice. We highlight the dynamic interplay among these claims and point towards implications for identity work in socio-technical contexts.

As Extended Reality (XR) devices and applications become more mainstream, so too will XR advertising — advertising that takes place in XR mediums. Due to the defining features of XR devices, such as the immersivity of the medium and the ability of XR devices to simulate reality, there are fears that these features could be exploited to create manipulative XR ads that trick consumers into buying products they do not need or might harm them. Using scenario construction, we investigate potential future incarnations of manipulative XR advertising and their harms. We identify five key mechanisms of manipulative XR advertising: misleading experience marketing; inducing artificial emotions in consumers; sensing and targeting people when they are vulnerable; emotional manipulation through hyperpersonalization; and distortion of reality. We discuss research challenges and questions in order to address and mitigate manipulative XR advertising risks.

As crucial public functions are transferred to computer systems, emerging technologies have public implications that are often shaped beyond public influence and oversight. “Smart city” and “modernization” projects are just some examples of such transformations. This paper focuses on struggles over the acquisition, control, and maintenance of these public, digital infrastructures. We focus on the forms of HCI knowledge and practice that proved useful to a coalition of community organizations claiming rights of input into and political oversight over surveillance technology. Their claims were a response to their exclusion from decision-making about smart city implementation in San Diego. We offer tactics “from below” as a way to attune HCI to the needs and practices of those excluded from power over widespread technology infrastructures. Ultimately, we argue that HCI cultivates a variety of capacities beyond design and redesign that can strengthen struggles to shape real-world technologies from below.

We present a vision for conversational user interfaces (CUIs) as probes for speculating with, rather than as objects to speculate about. Popular CUIs, e.g., Alexa, are changing the way we converse, narrate, and imagine the world(s) to come. Yet, current conversational interactions normatively may promote non-desirable ends, delivering a restricted range of request-response interactions with sexist and digital colonialist tendencies. Our critical design approach envisions alternatives by considering how future voices can reside in CUIs as enabling probes. We present novel explorations that illustrate the potential of CUIs as critical design material, by critiquing present norms and conversing with imaginary species. As micro-level interventions, we show that conversations with diverse futures through CUIs can persuade us to critically shape our discourse on macro-scale concerns of the present, e.g., sustainability. We reflect on how conversational interactions with pluralistic, imagined futures can contribute to how being human stands to change.

There is a developing recognition of the social and economic costs entailed in global supply chains. In this paper, we report on efforts to provide alternative, more sustainable and resilient models of production. Community Supported Agricultures (CSAs) address this problem but require new means of exchange which, we suggest, offer a design opportunity for sustainable HCI research. This paper presents a two months participatory observation in a food movement, a German CSA which developed a distribution system involving their own currency. Based on our ethnographic observations, we focus our discussion on (1) the solidaristic principles upon which the movement is based and (2) techniques of mediating between consumers’ wishes and the constraints of local agricultural production. By relating to the continued development of CSAs, we identify three interrelated innovation gaps and discuss new software architectures aimed at resolving the problems which arise as the movement grows.

Sustainable HCI (SHCI) constitutes a relatively new research field within HCI. We have identified four literature reviews of the field conducted between 2009-2014. In this paper, we present and discuss the results of a systematic literature review of peer-reviewed conference and journal articles that have been published in the field during the last ten years (2010-2019). To this end, we apply the United Nations’ Sustainable Development Goals (SDGs) as a framework to classify and discern high-level goals SHCI researchers have worked towards during this period. This paper contributes to HCI by 1) identifying Sustainable Development Goals that SHCI researchers have worked towards, 2) discerning main research trends in the field during the last decade, 3) using the SDG framework generatively to enumerate and reflect on areas that this far have not been covered by SHCI research and 4) presenting takeaways and opportunities for further research by the larger HCI community.

This paper investigates the practices of organising face-to-face events of a volunteer-run food-sharing community in Denmark. The ethnographic fieldwork draws attention to the core values underlying the ways sharing events are organised, and how - through the work of volunteers - surplus food is transformed from a commodity to a gift. The findings illustrate the community’s activist agenda of food waste reduction, along with the volunteers’ concerns and practical labour of running events and organising the flow of attendees through various queuing mechanisms. The paper contributes to the area of Food and HCI by: i) outlining the role of queuing in organising activism and ii) reflecting on the role that values, such as collective care and commons, can play in structuring queuing at face-to-face events.

Within Participatory- and Co-Design projects, the issue of sustainability and maintenance of the co-designed artefacts is a crucial yet largely unresolved issue. In this paper, we look back on four years of work on co-designing tools that assist refugees and migrants in their efforts to settle in Germany, the last of which the project has been independently maintained by our community collaborators. We reflect on the role of pre-existing care practices amongst our community collaborators, and a continued openness throughout the project, that allowed a complex constellation of actors to be involved in its ongoing maintenance and our own, often mundane activities which have contributed to the sustainability of the results. Situating our account within an HCI for Social Justice agenda, we thereby contribute to an ongoing discussion about the sustainability of such activities.

In recent years HCI and CSCW work has increasingly begun to address complex social problems and issues of social justice worldwide. Such activist-leaning work is not without problems. Through the experiences and reflections of an activist becoming academic and an academic becoming an activist, we outline these difficulties such as (1) the risk of perpetuating violence, oppression and exploitation when working with marginalised communities, (2) the reception of activist-academic work within our academic communities, and (3) problems of social justice that exist within our academic communities. Building on our own experiences, practices and existing literature from a variety of disciplines we advocate for the possibility of an activist-academic practice, outline possible ways forward and formulate questions we need to answer for HCI to contribute to a more just world.

Working with emergent users in two of Mumbai’s slums, we explored the value and uses of photovoltaic (PV) self-powering digital materials. Through a series of co-design workshops, a diary study and responses by artists and craftspeople, we developed the PV-Pix concept for inter-home connections. Each PV-Pix element consists of a deformable energy harvesting material that, when actuated by a person in one home, changes its physical state both there and in a connected home. To explore the concept we considered two forms of PV-Pix: one uses rigid materials and the other flexible ones. We deployed two low-fidelity prototypes, each constructed of a grid of one PV-Pix type, in four slum homes over a four week period to further understand the usability and uses of the materials, eliciting interesting inter-family communication practices. Encouraged by these results we report on a first-step towards working prototypes and demonstrate the technical viability of the approach.

The relationships that constitute the global industrial food system tend towards two dominant values that are creating unsustainable social and environmental inequalities. The first is a human-centered perspective on food that privileges humans over all other species. The second is a view of food as a commodity to be traded for maximum economic value, rewarding a small number of shareholders. We present work that explores the unique algorithmic affordances of blockchain to create new types of value exchange and governance in the food system. We describe a project that used roleplay with urban agricultural communities to co-design blockchain-based food futures and explore the conditions for creating a thriving multispecies food commons. We discuss how the project helped rethink algorithmic food justice by reconfiguring more-than-human values and reconfiguring food as more-than-human commons. We also discuss some of the challenges and tensions arising from these explorations.

Non-expert annotators (who lack sufficient domain knowledge) are often recruited for manual image labeling tasks owing to the lack of expert annotators. In such a case, label quality may be relatively low. We propose leveraging the spatial layout for improving label quality in non-expert image annotation. In the proposed system, an annotator first spatially lays out the incoming images and labels them on an open space, placing related items together. This serves as a working space (spatial organization) for tentative labeling. During the process, the annotator observes and organizes the similarities and differences between the items. Finally, the annotator provides definitive labels to the images based on the results of the spatial layout. We ran a user study comparing the proposed method and a traditional non-spatial layout in an image labeling task. The results demonstrated that annotators can complete the labeling tasks more accurately using the spatial layout interface than the non-spatial layout interface.

Computational notebooks, which seamlessly interleave code with results, have become a popular tool for data scientists due to the iterative nature of exploratory tasks. However, notebooks provide a single execution state for users to manipulate through creating and manipulating variables. When exploring alternatives, data scientists must carefully create many-step manipulations in visually distant cells.

We conducted formative interviews with 6 professional data scientists, motivating design principles behind exposing multiple states. We introduce forking — creating a new interpreter session — and backtracking — navigating through previous states. We implement these interactions as an extension to notebooks that help data scientists more directly express and navigate through decision points a single notebook. In a qualitative evaluation, 11 professional data scientists found the tool would be useful for exploring alternatives and debugging code to create a predictive model. Their insights highlight further challenges to scaling this functionality.

Code search is an important and frequent activity for developers using computational notebooks (e.g., Jupyter). The flexibility of notebooks brings challenges for effective code search, where classic search interfaces for traditional software code may be limited. In this paper, we propose, NBSearch, a novel system that supports semantic code search in notebook collections and interactive visual exploration of search results. NBSearch leverages advanced machine learning models to enable natural language search queries and intuitive visualizations to present complicated intra- and inter-notebook relationships in the returned results. We developed NBSearch through an iterative participatory design process with two experts from a large software company. We evaluated the models with a series of experiments and the whole system with a controlled user study. The results indicate the feasibility of our analytical pipeline and the effectiveness of NBSearch to support code search in large notebook collections.

Researchers have explored several avenues to mitigate data scientists’ frustrations with computational notebooks, including: (1) live programming, to keep notebook results consistent and up to date; (2) supplementing scripting with graphical user interfaces (GUIs), to improve ease of use; and (3) providing domain-specific languages (DSLs), to raise a script’s level of abstraction. This paper introduces Glinda, which combines these three approaches by providing a live programming experience, with interactive results, for a domain-specific language for data science. The language’s compiler uses an open-ended set of “recipes” to execute steps in the user’s data science workflow. Each recipe is intended to combine the expressiveness of a written notation with the ease-of-use of a GUI. Live programming provides immediate feedback to a user’s input, whether in the form of program edits or GUI gestures. In a qualitative evaluation with 12 professional data scientists, participants highly rated the live programming and interactive results. They found the language productive and sufficiently expressive and suggested opportunities to extend it.

Training deep neural networks can generate non-descriptive error messages or produce unusual output without any explicit errors at all. While experts rely on tacit knowledge to apply debugging strategies, non-experts lack the experience required to interpret model output and correct Deep Learning (DL) programs. In this work, we identify DL debugging heuristics and strategies used by experts, andIn this work, we categorize the types of errors novices run into when writing ML code, and map them onto opportunities where tools could help. We use them to guide the design of Umlaut. Umlaut checks DL program structure and model behavior against these heuristics; provides human-readable error messages to users; and annotates erroneous model output to facilitate error correction. Umlaut links code, model output, and tutorial-driven error messages in a single interface. We evaluated Umlaut in a study with 15 participants to determine its effectiveness in helping developers find and fix errors in their DL programs. Participants using Umlaut found and fixed significantly more bugs and were able to implement fixes for more bugs compared to a baseline condition.

End-user programmers opportunistically copy-and-paste code snippets from colleagues or the web to accomplish their tasks. Unfortunately, these snippets often don’t work verbatim, so these people—who are non-specialists in the programming language—make guesses and tweak the code to understand and apply it successfully. To support their desired workflow and facilitate tweaking and understanding, we built a prototype tool, TweakIt, that provides users with a familiar live interaction to help them understand, introspect, and reify how different code snippets would transform their data. Through a usability study with 14 data analysts, participants found the tool to be useful to understand the function of otherwise unfamiliar code, to increase their confidence about what the code does, to identify relevant parts of code specific to their task, and to proactively explore and evaluate code. Overall, our participants were enthusiastic about incorporating TweakIt in their own day-to-day work.

Trigger-action programming (if-this-then-that rules) empowers non-technical users to automate services and smart devices. As a user’s set of trigger-action programs evolves, the user must reason about behavior differences between similar programs, such as between an original program and several modification candidates, to select programs that meet their goals. To facilitate this process, we co-designed user interfaces and underlying algorithms to highlight differences between trigger-action programs. Our novel approaches leverage formal methods to efficiently identify and visualize differences in program outcomes or abstract properties. We also implemented a traditional interface that shows only syntax differences in the rules themselves. In a between-subjects online experiment with 107 participants, the novel interfaces better enabled participants to select trigger-action programs matching intended goals in complex, yet realistic, situations that proved very difficult when using traditional interfaces showing syntax differences.

Many programmers want to use deep learning due to its superior accuracy in many challenging domains. Yet our formative study with ten programmers indicated that, when constructing their own deep neural networks (DNNs), they often had a difficult time choosing appropriate model structures and hyperparameter values. This paper presents ExampleNet—a novel interactive visualization system for exploring common and uncommon design choices in a large collection of open-source DNN projects. ExampleNet provides a holistic view of the distribution over model structures and hyperparameter settings in the corpus of DNNs, so users can easily filter the corpus down to projects tackling similar tasks and compare design choices made by others. We evaluated ExampleNet in a within-subjects study with sixteen participants. Compared with the control condition (i.e., online search), participants using ExampleNet were able to inspect more online examples, make more data-driven design decisions, and make fewer design mistakes.

This paper explores software’s role in visual art production by examining how artists use and develop software. We conducted interviews with professional artists who were collaborating with software developers, learning software development, and building and maintaining software. We found artists were motivated to learn software development for intellectual growth and access to technical communities. Artists valued efficient workflows through skilled manual execution and personal software development, but avoided high-level forms of software automation. Artists identified conflicts between their priorities and those of professional developers and computational art communities, which influenced how they used computational aesthetics in their work. These findings contribute to efforts in systems engineering research to integrate end-user programming and creativity support across software and physical media, suggesting opportunities for artists as collaborators. Artists’ experiences writing software can guide technical implementations of domain-specific representations, and their experiences in interdisciplinary production can aid inclusive community building around computational tools.

User interface design is a complex task that involves designers examining a wide range of options. We present Spacewalker, a tool that allows designers to rapidly search a large design space for an optimal web UI with integrated support. Designers first annotate each attribute they want to explore in a typical HTML page, using a simple markup extension we designed. Spacewalker then parses the annotated HTML specification, and intelligently generates and distributes various configurations of the web UI to crowd workers for evaluation. We enhanced a genetic algorithm to accommodate crowd worker responses from pairwise comparison of UI designs, which is crucial for obtaining reliable feedback. Based on our experiments, Spacewalker allows designers to effectively search a large design space of a UI, using the language they are familiar with, and improve their design rapidly at a minimal cost.

Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.

This paper presents GestureMap, a visual analytics tool for gesture elicitation which directly visualises the space of gestures. Concretely, a Variational Autoencoder embeds gestures recorded as 3D skeletons on an interactive 2D map. GestureMap further integrates three computational capabilities to connect exploration to quantitative measures: Leveraging DTW Barycenter Averaging (DBA), we compute average gestures to 1) represent gesture groups at a glance; 2) compute a new consensus measure (variance around average gesture); and 3) cluster gestures with k-means. We evaluate GestureMap and its concepts with eight experts and an in-depth analysis of published data. Our findings show how GestureMap facilitates exploring large datasets and helps researchers to gain a visual understanding of elicited gesture spaces. It further opens new directions, such as comparing elicitations across studies. We discuss implications for elicitation studies and research, and opportunities to extend our approach to additional tasks in gesture elicitation.

Due to advances in deep learning, gestures have become a more common tool for human-computer interaction. When implementing a large amount of training data, deep learning models show remarkable performance in gesture recognition. Since it is expensive and time consuming to collect gesture data from people, we are often confronted with a practicality issue when managing the quantity and quality of training data. It is a well-known fact that increasing training data variability can help to improve the generalization performance of machine learning models. Thus, we directly intervene in the collection of gesture data to increase human gesture variability by adding some words (called styling words) into the data collection instructions, e.g., giving the instruction "perform gesture #1 faster" as opposed to "perform gesture #1." Through an in-depth analysis of gesture features and video-based gesture recognition, we have confirmed the advantageous use of styling words in gesture training data collection.

This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We performed a perceptual study with this dataset and found out that UI/UX designers can recognize the UI element sketches with ~96% accuracy. To compare human performance against computational recognition methods, we trained the state-of-the-art DNN-based image classification models to recognize the UI elements sketches. This study revealed that the ResNet-152 model outperforms other classification networks and detects unknown UI element sketches with 91.77% accuracy (chance is 4.76%). We have open-sourced the entire dataset of UI element sketches to the community intending to pave the way for further research in utilizing AI to assist the conversion of lo-fi UI sketches to higher fidelities.

In recent times, millions of people enjoy watching video gameplays at an eSports stadium or home. We seek a method that improves gameplay spectator or viewer experiences by presenting multisensory stimuli. Using a motion chair, we provide the motion effects automatically generated from the audiovisual stream to the viewers watching a first-person shooter (FPS) gameplay. The motion effects express the game character’s movement and gunfire action. We describe algorithms for the computation of such motion effects developed using computer vision techniques and deep learning. By a user study, we demonstrate that our method of providing motion effects significantly improves the viewing experiences of FPS gameplay. The contributions of this paper are with the motion synthesis algorithms integrated for FPS games and the empirical evidence for the benefits of experiencing multisensory gameplays.

Tilt as a gaming term is associated with frustration, rage, and deterioration of gameplay ability. In this exploratory study, we surveyed 95 esports players in a high school esports league on their definitions of tilt, triggers for tilting, responses to tilt, and perception of its malleability. We found that players are tilted most commonly by their own teammates rather than opponents, with their most negative tilt responses reserved for themselves. The majority surveyed believe that they can change how easily they are tilted and believing so was found to lead players to choose more positive responses to tilt. In contrast, perceiving tilt as malleable was found to increase the probability that participants respond with positive strategies. Implications for efforts to improve esports culture and community are discussed.

Competitive VR gaming has emerged as a new trend in recent years, due to the availability of consumer grade VR technologies and the rise of esports as a billion dollar industry. Despite the considerable attention to competitive VR gaming, there is a lack of research on attitudes and experiences that players have with these games. In this qualitative study with a pre-post interview design, we recruited eight competitive Counter-Strike: Global Offensive players from a university esports club. We aimed to understand their attitudes towards VR esports and their experiences playing a representative location based VR esports game. Findings showed that players had visceral and positive affective experiences in the game, such as how players map physical movements to the game. These findings can help design future competitive VR esports, while also further contributing to HCI as the first exploration on player experiences with VR esports, laying groundwork for future studies.

Esports are rapidly growing within academia. In HCI, earlier work explored how problematic behaviors emerging from gender biases (e.g., toxicity) negatively impact female participation in esports. Here, we further explore gender biases in esports by interviewing 19 self-identified female and male professional gamers and event organizers. We inquire our interviewees about personal experiences with gender biases in esports and their perspective on how these biases impact participation, inclusivity, and career prospects. Our interviewees see gender biases in esports as a consequence of stereotypical gender roles in gaming tout-court (e.g., girls do not like violence, boys are competitive by nature). The rationale for separating male and female in esports, however, seems rooted in the need for female gamers to create role-models and grow in self-confidence. We scrutinize the considerations emerging from our interviews under a Feminist HCI lens and discuss how HCI research can help design equitable environments in esports.

Esports is a rapidly growing industry, generating interest from research disciplines including marketing, social sciences, and human-computer interaction. Despite its continued growth, there is a lack of studies surrounding the health of esports players. Previous work on the subject is limited, as research has only recently begun to explore the potential factors affecting physical and psychological wellness. Using an exploratory mixed-methods approach, a series of semi-structured interviews (n = 10) and an online survey (n = 68) were used to identify the biggest health concerns among esports players. The results demonstrate a better understanding of issues regarding physical and psychological wellness according to the players. Moving forward, we suggest the HCI community adapts mindfulness, ergonomics, and social-emotional learning as methods for supporting player’s health concerns.

Game industry professionals are frequently implementing new methods of addressing ethical issues related to in-game toxicity and disruptive player behaviours associated with online multiplayer games. However, academic work on these behaviours tends to focus on the perspectives of players rather than the industry. To fully understand the ethics of multiplayer games and promote ethical design, we must examine the challenges facing those designing multiplayer games through an ethical lens. To this end, this paper presents a reflexive thematic analysis of 21 in-depth interviews with games industry professionals on their ethical views and experiences in game design and community management. We identify a number of tensions involved in making ethics-related design decisions for divided player communities alongside current game design practices that are concerned with functionality, revenue and entertainment. We then put forward a set of design considerations for integrating ethics into multiplayer game design.

Video game play is among the most popular forms of entertainment in the world and eSports is a multi-billion dollar industry. Esports gamers, and competitive gamers more broadly, want fast game systems to maximize their chances of winning. In general, the faster the game system, the lower the latency between a player’s action and the intended outcome. But how much small reductions in local latencies benefit competitive players is not known. This paper presents results from a 43-person user study that evaluates the impact of system latencies for high-end gaming systems (below 125 ms) on experienced Counter-strike: Global Offensive (CS:GO) players. Analysis of the results show pronounced benefits to CS:GO player performance (accuracy and score) for even small reductions in latency, with subjective opinions on Quality of Experience following suit.

In 2020, the rapid spread of COVID-19 forced many people to self-isolate, resulting in struggles with mental health [60], and an increase in gaming [3]. In this paper, we seek to examine how individuals used digital games during the quarantine. We conducted a two-tier qualitative study where we used a thematic analysis of tweets to derive questions for an online survey that we distributed. Results of thematic analysis of survey responses identified 15 themes. Some themes confirm previous works’ findings, particularly how games are used to increase social connection or distract oneself from unpleasant situations. We also found new themes unique to the quarantine, such as interactions with non-player characters used as a surrogate for real-world interaction and using in-game routines as a substitute to real-world routines lost due to the pandemic. This work discusses the use of games during the pandemic and can be seeds for future studies.

Whereas past CHI-related work has paid attention to various elements of the esports industry, there exists a scarcity of such research on the increasing convergence between esports and gambling. This study, therefore, aimed to shed light on the emerging phenomenon of esports betting in two ways. First, we theorized about its characteristics according to the 5W1H framework. Second, using semi-structured laddering interviews and Means-End Chain Theory, we assessed the esports betting motivations of young, male and recreational gamblers who interact with monetary and skin betting websites. Our results show that gamblers follow various motivational pathways when using an esports betting platform and construct their motives in relation to platform-specific properties and personal values. As such, we demonstrate that a holistic understanding of esports betting is of utmost importance if we want to explain and investigate its appeal to a worldwide audience.

Live streaming sites such as Twitch offer new ways for remote audiences to engage with and affect gameplay. While research has considered how audiences interact with games, HCI lacks clear demarcations of the potential design spaces for audience participation. This paper introduces and validates a theme map of audience participation in game live streaming for student designers. This map is a lens that reveals relationships among themes and sub-themes of Agency, Pacing, and Community—to explore, reflect upon, describe, and make sense of emerging, complex design spaces. We are the first to articulate such a lens, and to provide a reflective tool to support future research and education. To create the map, we perform a thematic analysis of design process documents of a course on audience participation for Twitch, using this analysis to visually coordinate relationships between important themes. To help student designers analyze and reflect on existing experiences, we supplement the theme map with a set of mapping procedures. We validate the applicability of our map with a second set of student designers, who found the map useful as a comparative and reflective tool.

With the rise of competitive online gaming and esports, players’ ability to review, reflect upon, and improve their in-game performance has become important. Post-play visualizations are key for such improvements. Despite the increased interest in visualizations of gameplay, research specifically informing the design of player-centric visualizations is currently limited. As with all visualizations, their design should, however, be guided by a thorough understanding of the goals to be achieved and which information is important and why.

This paper reports on a mixed-methods study exploring the information demands posed by players on post-play visualizations and the goals they pursue with such visualizations. We focused on three genres that enjoy great popularity within the competitive gaming scene. Our results provide useful guideposts on which data to focus on by offering an overview of the relevance of different in-game metrics across genres. Lastly, we outline high-level implications for the design of post-play visualizations.

In first-person shooters (FPS), professional players (a.k.a., Gosu) outperform amateur players. The secrets behind the performance of professional FPS players have been debated in online communities with many conjectures; however, attempts of scientific verification have been limited. We addressed this conundrum through a data-collection study of the gameplay of eight professional and eight amateur players in the commercial FPS Counter-Strike: Global Offensive. The collected data cover behavioral data from six sensors (motion capture, eye tracker, mouse, keyboard, electromyography armband, and pulse sensor) and in-game data (player data and event logs). We examined conjectures in four categories: aiming, character movement, physicality, and device and settings. Only 6 out of 13 conjectures were supported with statistically sufficient evidence.

We present a system for connecting partners in long-distance relationships in bedrooms and at bedtime, a space and time that most couples share. Unlike communications explicitly initiated by users, our system is always-on, staying in the background and enabling remote presence without constant use. We present findings from a field study in which the system was deployed into participants’ bedrooms. The system includes an automated photo-stream (rather than video), which was found to provide a balance between the feeling of presence and privacy, and to remove the pressure to communicate. The system also includes a real-time shared inking canvas with disappearing ink, which was found to provide a rich versatile medium allowing for new patterns of communication, live interventions, and collaborative drawing. Learnings from how our system balances privacy and remote connectedness may also have relevance for other domains such as remote healthcare and education.

With the growing ubiquity of wearable devices, sensed physiological responses provide new means to connect with others. While recent research demonstrates the expressive potential for biosignals, the value of sharing these personal data remains unclear. To understand their role in communication, we created Significant Otter, an Apple Watch/iPhone app that enables romantic partners to share and respond to each other’s biosignals in the form of animated otter avatars. In a one-month study with 20 couples, participants used Significant Otter with biosignals sensing OFF and ON. We found that while sensing OFF enabled couples to keep in touch, sensing ON enabled easier and more authentic communication that fostered social connection. However, the addition of biosignals introduced concerns about autonomy and agency over the messages they sent. We discuss design implications and future directions for communication systems that recommend messages based on biosignals.

Social media platforms continue to evolve as archival platforms, where important milestones in an individual’s life are socially disclosed for support, solidarity, maintaining and gaining social capital, or to meet therapeutic needs. However, a limited understanding of how and what life events are disclosed (or not) prevents designing platforms to be sensitive to life events. We ask what life events individuals disclose on a 256 participants’ year-long Facebook dataset of 14K posts against their self-reported life events. We contribute a codebook to identify life event disclosures and build regression models on factors explaining life events’ disclosures. Positive and anticipated events are more likely, whereas significant, recent, and intimate events are less likely to be disclosed on social media. While all life events may not be disclosed, online disclosures can reflect complementary information to self-reports. Our work bears practical and platform design implications in providing support and sensitivity to life events.

Music fans strategically support their artists. Their collective efforts can extend to social causes as well: In 2020 for example, ARMY—the fandom of the music group BTS—successfully organized the #MatchAMillion campaign to raise over one million USD to support Black Lives Matter. To better understand factors of fandoms’ collaborative success for arguably unrelated social goals, we conducted a survey focusing on ARMYs’ perceptions of their fandom and their social effort. Most ARMYs viewed the fandom as a community, loosely structured with pillar accounts. They reported trust in each other as well as high team composition, which mediated the relationship between their neutral psychological safety and high efficacy. Respondents attributed their success in #MatchAMillion to shared values, good teamwork, and established infrastructure. Our findings elucidate contextual factors that contribute to ARMY’s collaborative success and highlight themes that may be applied to studying other fandoms and their collaborative efforts.

Live streaming is a rapidly growing industry, with millions of content creators using platforms like Twitch to share games, art, and other activities. However, with this rise in popularity, most streamers often fail to attract viewers and grow their platforms. Analytic tools—which have shown success in other business and learning contexts—may be one potential solution, but their use in streaming settings remains unexplored. In this study, we focused on game streaming and interviewed 18 game streamers on Twitch and Mixer about their information needs and current use of tools, supplemented by explorations into their Discord communities. We find that streamers have a range of content, marketing, and community information needs, many of which are not being met by available tools. We conclude with design implications for developing more streamer-centered analytics for video game streamers.

Loneliness threatens public mental wellbeing during COVID-19. In response, YouTube creators participated in the #StayHome #WithMe movement (SHWM) and made myriad videos for people experiencing loneliness or boredom at home. User-shared videos generate parasocial attachment and virtual connectedness. However, there is limited knowledge of how creators contributed videos during disasters to provide social provisions as disaster-relief. Grounded on Weiss’s loneliness theory, this work analyzed 1488 SHWM videos to examine video sharing as a pathway to social provisions. Findings suggested that skill and knowledge sharing, entertaining arts, homelife activities, live chatting, and gameplay were the most popular video styles. YouTubers utilized parasocial relationships to form a space for staying away from the disaster. SHWM YouTubers provided friend-like, mentor-like, and family-like provisions through videos in different styles. Family-like provisions led to the highest overall viewer engagement. Based on the findings, design implications for supporting viewers’ mental wellbeing in disasters are discussed.

The growing number of video chat users includes socially anxious people, but it is not known how video chat interfaces affect their interpersonal interactions. In our first study, we use a get-to-know-you task to show that when video feedback of oneself is disabled, higher social anxiety is associated with more public self-awareness, use of 2nd person pronouns, and experienced anxiety. Higher social anxiety was linked to discussing more topics, but discussing more topics only elicited higher self-disclosure and trust when social anxiety was low. In our second study, we assess these same effects using a presentation layout video chat interface and observe no effects of social anxiety on public self-awareness, 2nd person pronoun use, or number of topics discussed; no effect of feedback on experienced anxiety; and no link between number of topics and self-disclosure. Video chat adopters and designers should consider how feedback and interface layout affect conversations.

Even though today’s videoconferencing systems are often very useful, these systems do not provide support for one of the most important aspects of in-person meetings: the ad hoc, private conversations that happen before, after, and during the breaks of scheduled events–the proverbial hallway conversations. Here we describe our design of a simple system, called Minglr, which supports this kind of interaction by facilitating the matching of conversational partners. We describe two studies of this system’s use at two virtual conferences with over 450 total participants. Our results provide evidence for the usefulness of this capability, showing that, for example, 81% of people who used the system successfully thought that future virtual conferences should include a tool with similar functionality. We believe that similar functionality is likely to be widely implemented in many videoconferencing systems and to increase the feasibility and desirability of many kinds of remote work and socializing.

An estimated 100,000 people work today as commercial content moderators. These moderators are often exposed to disturbing content, which can lead to lasting psychological and emotional distress. This literature review investigates moderators’ psychological symptomatology, drawing on other occupations involving trauma exposure to further guide understanding of both symptoms and support mechanisms. We then introduce wellness interventions and review both programmatic and technological approaches to improving wellness. Additionally, we review methods for evaluating intervention efficacy. Finally, we recommend best practices and important directions for future research. Content Warning: we discuss the intense labor and psychological effects of CCM, including graphic descriptions of mental distress and illness.

Co-customizations are collaborative customizations in messaging apps that all conversation members can view and change, e.g. the color of chat bubbles on Facebook Messenger. Co-customizations grant new opportunities for expressing intimacy; however, most apps offer private customizations only. To investigate how people in close relationships integrate co-customizations into their established communication app ecosystems, we built DearBoard: an Android keyboard that allows two people to co-customize its color theme and a toolbar of expression shortcuts (emojis and GIFs). In a 5-week field study with 18 pairs of couples, friends, and relatives, participants expressed their shared interests, history, and knowledge of each other through co-customizations that served as meaningful decorations, interface optimizations, conversation themes, and non-verbal channels for playful, affectionate interactions. The co-ownership of the co-customizations invited participants to negotiate who customizes what and for whom they customize. We discuss how co-customizations mediate intimacy through place-making efforts and suggest design opportunities.

Message deletion in mobile messaging apps allows people to “unsay” things they have said. This paper explores how and why people use (or do not use) this feature within remediation strategies after a communication error is identified. We present findings from a multi-stage survey designed to explore people’s general experiences of the message deletion feature (N = 401), peoples’ experiences of using this feature during the remediation of an error (N = 70), and receivers’ perceptions around recent message deletions (N = 68). While people are typically aware of the deletion feature, it is infrequently used. When used, it is primarily done so to improve conversations by reducing confusion between conversation partners. We found people being aware of message deletions creating information-gaps which can provoke curiosity in recipients, causing them to develop narratives to help address the uncertainty. We found concerns amongst senders that these narratives would be of a negative nature, having an undesirable impact on how others perceive them. We use our findings to suggest ways in which mobile messaging apps could improve conversational experiences around erroneous and regrettable messages.

We present a video-analysis study of museum visitors’ interactions at two tangible interactive exhibits in a transport museum. Our focus is on groups’ social and shared interactions, in particular how exhibit setup and structure influence collaboration patterns. Behaviors at the exhibits included individuals focusing beyond their personal activity towards companions’ interaction, adults participating via physical interaction, and visitors taking opportunities to interact when companions moved between sections of the exhibit or stepped back from interaction. We demonstrate how exhibits’ physical configuration and interactive control engendered behavioral patterns. Systematic analysis reveals how different configurations (concerning physical-spatial hardware and interactive software) distribute control differently amongst visitors. We present four mechanisms for how control can be distributed at an interactive installation: functional, temporal, physical and indirect verbal. In summary, our work explores how mechanisms that distribute control influence patterns of shared interaction with the exhibits and social interaction between museum visitor companions.

Co-designing with children in an online environment is increasingly important due to external factors, such as the COVID-19 pandemic, and the diversification and inclusion of youth participants. Many prior studies about co-design with youth focus on co-located or asynchronous online sessions. However, conducting synchronous online co-design sessions adds layers of complexity and uncertainty to collaboration. This paper introduces a model explicating factors to consider when co-designing with children synchronously in an online space. We examined ten consecutive intergenerational participatory design sessions online where children (ages 7-11) and adults designed new technologies. Along with highlighting unexpected moments and interactions, we use theories of improvisation to guide our understanding of dynamic situations that are out of the control of researchers. This work contributes to improving theoretical understanding of improvisation as a method of inquiry for co-designing with youth, and offers practical suggestions for suitable online co-design techniques and implementation.

When schools and families form a good partnership, children benefit. With the recent flourishing of communication apps, families and schools in China have shifted their primary communication channels to chat groups hosted on popular instant-messenger(IM) tools such as WeChat and QQ. With an interview study consisting of 18 parents and 9 teachers, followed by a survey study with 210 teachers, we found that IM group chat has become the most popular way that the majority of parents and teachers communicate, from among the many different channels available. While there are definite advantages to this kind of group chat, we also found a number of problematic issues, including a lack of privacy and repeated negative feedback shared by both parents and teachers. We discuss our results on how IM-based group chat could affect Chinese teachers’ authoritative figures, affect Chinese teacher’s work-life balance and potentially compromise Chinese students’ privacy.

The COVID-19 pandemic and subsequent closure of schools forced families across the globe to transition to school at home. This unprecedented context is likely to have a lasting impact on the practice of schooling and the role of online, digital platforms within school contexts. In this paper we present a contextual inquiry of an ‘emergency home school context’, detailing how nine young families in Melbourne, Australia adapted to the unexpected introduction of school to the home following the government-directed closure of schools. Through an online interview and photo-journal study, we develop an emplaced understanding of the context detailing how the relations between people and places around the home evolved over time. We present five design considerations for digital platforms to support the emergency home school context, placing focus on the fluid roles, relationships and evolving sense of place.

The COVID-19 global pandemic and resulted lockdown policies have forced education in nearly every country to switch from a traditional co-located paradigm to a pure online “distance learning from home” paradigm. Lying in the center of this learning paradigm shift is the emergence and wide adoption of distance communication tools and live streaming platforms for education. Here, we present a mixed-methods study on live streaming based education experience during the COVID-19 pandemic. We focus our analysis on Chinese higher education, carried out semi-structured interviews on 30 students, and 7 instructors from diverse colleges and disciplines, meanwhile launched a large-scale survey covering 6291 students and 1160 instructors in one leading Chinese university. Our study not only reveals important design guidelines and insights to better support current remote learning experience during the pandemic, but also provides valuable implications towards constructing future collaborative education supporting systems and experience after pandemic.

Today, preschool-aged children have an abundance of digital content to choose from, with some more desirable than others from a developmental perspective. We aim to describe and better understand the interplay of factors that influence children’s selection of media content using a year-long, multi-case ethnography of 13 diverse families in Southern California. We found that young children’s media content selection may be best understood as an ecologically situated process involving the interplay between the content, the child, their family, community, and societal spheres. Children do not make media selections on their own. Rather, these choices are supported or constrained by a range of resource, culture, and policy factors specific to family and community background. We argue that policy makers and technology designers are better served by an ecological perspective if they wish to understand how digital content is selected and used by children in sociocultural context.

Digital fabrication courses that relied on physical makerspaces were severely disrupted by COVID-19. As universities shut down in Spring 2020, instructors developed new models for digital fabrication at a distance. Through interviews with faculty and students and examination of course materials, we recount the experiences of eight remote digital fabrication courses. We found that learning with hobbyist equipment and online social networks could emulate using industrial equipment in shared workshops. Furthermore, at-home digital fabrication offered unique learning opportunities including more iteration, machine tuning, and maintenance. These opportunities depended on new forms of labor and varied based on student living situations. Our findings have implications for remote and in-person digital fabrication instruction. They indicate how access to tools was important, but not as critical as providing opportunities for iteration; they show how remote fabrication exacerbated student inequities; and they suggest strategies for evaluating trade-offs in remote fabrication models with respect to learning objectives.

This paper investigates in-class interactions in synchronous online classrooms when the choice of modality is discretionary, such that students choose when and if they turn on their cameras and microphones. Instructor interviews (N = 7) revealed that most students preferred not to share videos and verbally participate. This hindered instructors’ ability to read their classrooms and make deeper connections with students. Survey results (N = 102) suggested that students felt a lacking sense of community in online vs. in-person lectures. Some students felt uncomfortable broadcasting their appearances to everyone in the class, and some were unaware of the benefits for instructors. Most students favored using the text chat to participate. Considering the needs of both instructors and students, we propose recommendations to mitigate the loss of classroom interactions by collecting and presenting less invasive social cues in an aggregated format, and incorporating opportunities for informal exchanges and individual control to spark peer bonding.

We investigated changes in and factors affecting American adolescents’ subjective wellbeing during the early months (April – August 2020) of the coronavirus pandemic in the United States. Twenty-one teens (14-19 years) participated in interviews at the start and end of the study and completed ecological momentary assessments three times per week between the interviews. There was an aggregate trend toward increased wellbeing, with considerable variation within and across participants. Teens reported greater reliance on networked technologies as their unstructured time increased during lockdown. Using multilevel growth modeling, we found that how much total time teens spent with technology had less bearing on daily fluctuations in wellbeing than the satisfaction and meaning they derived from their technology use. Ultimately, teens felt online communication could not replace face-to-face interactions. We conducted two follow-up participatory design sessions with nine teens to explore these insights in greater depth and reflect on general implications for design to support teens’ meaningful technology experiences and wellbeing during disruptive life events.

Social class contexts shape parents’ guiding principles around teens’ smartphone use. These contexts can affect how parents coach and censor their teens’ smartphone use and can create tensions in the home. Through 174 interviews (87 parent-teen dyads), we find that upper-middle-class families generally adopt an orientation toward scaffolded achievements and working-class families tend to embrace an orientation toward empowered self-sufficiency. We further find that these class-based orientations contribute to parent-teen tensions. For upper-middle-class families, tensions arise when parents insist that teens should use smartphones to get help with academic and enrichment activities and teens disagree about whether their phone-related activities align with this goal. In contrast, we find that conflict can occur in working-class families when teens use their smartphones to get assistance and parents interpret such activity as teens being lazy or not self-sufficient. These findings highlight the role of social class contexts in shaping families’ orientations toward teens’ smartphone use and phone-related tensions.

Researchers and designers have incorporated social media affordances into learning technologies to engage young people and support personally relevant learning, but youth may reject these attempts because they do not meet user expectations. Through in-depth case studies, we explore the sociotechnical ecosystems of six teens (ages 15-18) working at a science center that had recently introduced a digital badge system to track and recognize their learning. By analyzing interviews, observations, ecological momentary assessments, and system data, we examined tensions in how badges as connected learning technologies operate in teens' sociotechnical ecosystems. We found that, due to issues of unwanted context collapse and incongruent identity representations, youth only used certain affordances of the system and did so sporadically. Additionally, we noted that some features seemed to prioritize values of adult stakeholders over youth. Using badges as a lens, we reveal critical tensions and offer design recommendations for networked learning technologies.

“Study with me” videos contain footage of people studying for hours, in which social components like conversations or informational content like instructions are absent. Recently, they became increasingly popular on video-sharing platforms. This paper provides the first broad look into what “study with me” videos are and how people use them. We analyzed 30 “study with me” videos and conducted 12 interviews with their viewers to understand their motivation and viewing practices. We identified a three-factor model that explains the mechanism for shaping a satisfactory studying experience in general. One of the factors, a well-suited ambience, was difficult to achieve because of two common challenges: external conditions that prevent studying in study-friendly places and extra cost needed to create a personally desired ambience. We found that the viewers used “study with me” videos to create a personalized ambience at a lower cost, to find controllable peer pressure, and to get emotional support. These findings suggest that the viewers self-regulate their learning through watching “study with me” videos to improve efficiency even when studying alone at home.

Personal informatics (PI) technologies allow users to collect data about aspects of their lifestyle like mood or step count. Though teens increasingly encounter and use such technologies, little is known about how they ascribe meaning to their own PI activities. We report a qualitative study of the PI experiences of eighteen teens (aged 14 – 17). Following a learning phase focused on interpreting PI data, participants chose a personal goal that interested them and a PI tool to track it for 4-8 weeks in everyday contexts. Participants proved to be competent, flexible users of PI tools, tracking a range of meaningful life factors, from ‘worries’ to ‘exercise’; they valued learning about ‘natural patterns’ in their lives and were motivated to manage their emotions and evaluate whether they were doing the right thing. Our findings contribute to understanding how young people can engage in appropriation and interpretation of PI data – suggesting opportunities for educational interventions and design.

Bullying is a challenge concerning us all, and particularly our children. This has already been acknowledged by CHI, among others. Despite the interest, there is a lack of comprehensive understanding of the state of the art – a critical review is needed, addressing bullying in the lives of children, in the context of and/or by the means of design and technology, covering CHI as well as related computing fields, being inspired by the strong body of knowledge within human sciences. We report on a comprehensive literature review on the topic, with the aim to understand what and how has been done so far to handle this troublesome and widespread phenomenon as well as to indicate how to move the field forward. We report how the topic has been examined and with what kind of means tackled, revealing interesting underlying assumptions about design, technology and human agency.

Subtle patterns in users’ think-aloud (TA) verbalizations and speech features are shown to be telltale signs of User Experience (UX) problems. However, such patterns were uncovered among young adults. Whether such patterns apply for older adults remains unknown. We conducted TA usability testing with older adults using physical and digital products. We analyzed their verbalizations, extracted speech features, identified UX problems, and uncovered the patterns that indicate UX problems. Our results show that when older adults encounter problems, their verbalizations tend to include observations (remarks), negations, question words and words with negative sentiments; and their voices tend to include high loudness, high pitch and high speech rate. We compare these subtle patterns with those of young adults uncovered in recent studies and discuss the implications of these patterns for the design of Human-AI collaborative UX analysis tools to better pinpoint UX problems.

Priming is used as a way of increasing the diversity of proposals in end-user elicitation studies, but priming has not been investigated thoroughly in this context. We conduct a distributed end-user elicitation study with 167 participants, which had three priming groups: a no-priming control group, sci-fi priming, and a creative mindset group. We evaluated the gestures proposed by these groups in a distributed learnability and memorability study with 18 participants. We found that the user-elicited gestures from the sci-fi group were significantly faster to learn, requiring an average of 1.22 viewings to learn compared to 1.60 viewings required to learn the control gestures, and 1.56 viewings to learn the gestures elicited from the creative mindset group. In addition, both primed gesture groups had higher memorability with 80% of the sci-fi-primed gestures and 73% of the creative mindset group gestures were recalled correctly after one week without practice compared to 43% of the control group gestures.

There is a rapidly growing literature on dark patterns, user interface designs—typically related to shopping or privacy—that researchers deem problematic. Recent work has been predominantly descriptive, documenting and categorizing objectionable user interfaces. These contributions have been invaluable in highlighting specific designs for researchers and policymakers. But the current literature lacks a conceptual foundation: What makes a user interface a dark pattern? Why are certain designs problematic for users or society?

We review recent work on dark patterns and demonstrate that the literature does not reflect a singular concern or consistent definition, but rather, a set of thematically related considerations. Drawing from scholarship in psychology, economics, ethics, philosophy, and law, we articulate a set of normative perspectives for analyzing dark patterns and their effects on individuals and society. We then show how future research on dark patterns can go beyond subjective criticism of user interface designs and apply empirical methods grounded in normative perspectives.

Online display advertising on websites is widely disliked by users, with many turning to ad blockers to avoid “bad” ads. Recent evidence suggests that today’s ads contain potentially problematic content, in addition to well-studied concerns about the privacy and intrusiveness of ads. However, we lack knowledge of which types of ad content users consider problematic and detrimental to their browsing experience. Our work bridges this gap: first, we create a taxonomy of 15 positive and negative user reactions to online advertising from a survey of 60 participants. Second, we characterize classes of online ad content that users dislike or find problematic, using a dataset of 500 ads crawled from popular websites, labeled by 1000 participants using our taxonomy. Among our findings, we report that users consider a substantial amount of ads on the web today to be clickbait, untrustworthy, or distasteful, including ads for software downloads, listicles, and health & supplements.

Prior work in cross-cultural psychology and neuroscience has shown robust variations in visual attention patterns. People from East Asian societies, in which a holistic thinking style predominates, have been found to attend to contextual information in scenes more than Westerners, whose tendency to think analytically expresses itself in greater attention to foreground objects. This paper applies these findings to website design, using an online study to evaluate whether Japanese (N=65) remember more and are faster at finding contextual website information than US Americans (N=84). Our results do not support this hypothesis. Instead, Japanese overall took significantly longer to find information than US participants—a difference that was exacerbated by an increase in website complexity—suggesting that Japanese may holistically take in a website before engaging with detailed information. We discuss implications of these findings for website design and cross-cultural research.

Spatial computing devices that blend virtual and real worlds have the potential to soon become ubiquitous. Yet, creating experiences for spatial computing is non-trivial and needs skills in programming and 3D content creation, rendering them inaccessible to a wider group of users. We present SpatialProto, an in-situ spatial prototyping system for lowering the barrier to engage in spatial prototyping. With a depth-sensing capable mixed reality headset, SpatialProto lets users record animated objects of the real-world environment (e.g. paper, clay, people, or any other prop), extract only the relevant parts, and directly place and transform these recordings in their physical environment. We describe the design and implementation of SpatialProto, a user study evaluating the system’s prototype with non-expert users (n = 9), and demonstrate applications where multiple captures are fused for compelling augmented reality experiences.

Interactive machine learning (iML) tools help to make ML accessible to users with limited ML expertise. However, gathering necessary training data and expertise for model-building remains challenging. Transfer learning, a process where learned representations from a model trained on potentially terabytes of data can be transferred to a new, related task, offers the possibility of providing ”building blocks” for non-expert users to quickly and effectively apply ML in their work. However, transfer learning largely remains an expert tool due to its high complexity. In this paper, we design a prototype to understand non-expert user behavior in an interactive environment that supports transfer learning. Our findings reveal a series of data- and perception-driven decision-making strategies non-expert users employ, to (in)effectively transfer elements using their domain expertise. Finally, we synthesize design implications which might inform future interactive transfer learning environments.

The advancement of computational design and fabrication technologies has allowed combining physical and digital processes in architecture. Existing methods for physical-digital integration offer limited support for explorations with folded non-linear surfaces. This paper introduces a feedback-oriented design approach linking physical models with digital tools to enhance ideation processes in architecture. We employ paper as a medium for translating simple mock-up ideas to more elaborate digital design models. We explain the physical exploration, 3D scanning, digital simulation, and fabrication processes. Then, we discuss the results, observations, and limitations of this design approach.

Reflection is an often addressed design goal in Human-Computer Interaction (HCI) research. An increasing number of artefacts for reflection have been developed in recent years. However, evaluating if and how an interactive technology helps a user reflect is still complex. This makes it difficult to compare artefacts (or prototypes) for reflection, impeding future design efforts. To address this issue, we developed the Technology-Supported Reflection Inventory (TSRI), which is a scale that evaluates how effectively a system supports reflection. We first created a list of possible scale items based on past work in defining reflection. The items were then reviewed by experts. Next, we performed exploratory factor analysis to reduce the scale to its final length of nine items. Subsequently, we confirmed test-retest validity of our instrument, as well as its construct validity. The TSRI enables researchers and practitioners to compare prototypes designed to support reflection.

Visual and physical representations of historical personal data have been discussed as artifacts that can lead to self-reflection through meaning-making. However, it is yet unclear how those two concepts relate to each other. We focus on meaningfulness, a part of meaning-making that relates to feelings. In this paper, we present three projects where mundane objects, our data agents, are combined in meaningful ways with personal data with the aim to trigger reflection by placing a person’s individual experience of data in relation to others’. To identify relationships between self-reflection and meaningfulness we use Fleck and Fitzpatrick’s framework to describe the levels of reflection that we found in our projects and Mekler and Hornbæk’s meaning framework to define the depth of reflection. We conclude with a discussion on four themes in which we outline how data agents informed the intersections between our central concepts. This paper, constitutes a first step towards unpacking those relationships and invites for further explorations by the HCI community.

In the attention economy, video apps employ design mechanisms like autoplay that exploit psychological vulnerabilities to maximize watch time. Consequently, many people feel a lack of agency over their app use, which is linked to negative life effects such as loss of sleep. Prior design research has innovated external mechanisms that police multiple apps, such as lockout timers. In this work, we shift the focus to how the internal mechanisms of an app can support user agency, taking the popular YouTube mobile app as a test case. From a survey of 120 U.S. users, we find that autoplay and recommendations primarily undermine sense of agency, while playlists and search support it. From 13 co-design sessions, we find that when users have a specific intention for how they want to use YouTube they prefer interfaces that support greater agency. We discuss implications for how designers can help users reclaim a sense of agency over their media use.

This paper presents and evaluates Aiki, a simple browser extension designed to redirect a user to a learning platform for a fixed amount of time before accessing websites defined as ’procrastination’ or ’time-wasting’ websites. The goal of the extension is to enable the user to exchange time spent on pages they believe contribute less to their own productivity for microlearning activities, defined as small or short-term learning activities. The paper describes the design and development of Aiki and evaluates the extension with a group of n = 10 participants studying the Danish language. Based on a two-week study, we conclude that this type of extension, even in its preliminary version, has the potential to improve language skills in a lightweight manner and that redirection is an important alternative to blocking for procrastination management.

The rise of autonomous systems in cities, such as automated vehicles (AVs), requires new approaches for prototyping and evaluating how people interact with those systems through context-based user interfaces, such as external human-machine interfaces (eHMIs). In this paper, we present a comparative study of three prototype representations (real-world VR, computer-generated VR, real-world video) of an eHMI in a mixed-methods study with 42 participants. Quantitative results show that while the real-world VR representation results in higher sense of presence, no significant differences in user experience and trust towards the AV itself were found. However, interview data shows that participants focused on different experiential and perceptual aspects in each of the prototype representations. These differences are linked to spatial awareness and perceived realism of the AV behaviour and its context, affecting in turn how participants assess trust and the eHMI. The paper offers guidelines for prototyping and evaluating context-based interfaces through simulations.

Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS’ founding in 1994. We qualitatively coded areas of focus and methodological decisions for the past 10 years (2010-2019, N=506 papers), and analyzed paper counts and keywords over the full 26 years (N=836 papers). Our findings highlight areas that have received disproportionate attention and those that are underserved—for example, over 43% of papers in the past 10 years are on accessibility for blind and low vision people. We also capture common study characteristics, such as the roles of disabled and nondisabled participants as well as sample sizes (e.g., a median of 13 for participant groups with disabilities and older adults). We close by critically reflecting on gaps in the literature and offering guidance for future work in the field.

Despite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world’s largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs’ social acceptability in China.

In the US, abuse of individuals with intellectual and developmental disabilities (I/DD) is at epidemic proportions. Further, abuse incidents of individuals with I/DD are woefully under-reported. We surveyed practitioners who help individuals with I/DD post-abuse to get a broader context on the problem. We found that abuse of individuals with I/DD was often reported by someone other than the survivor, as survivors faced impediments in reporting. Consequently, we argue for developing a mobile-computing-based reporting tool for empowering individuals with I/DD to self-report abuse. Next, we conducted focus groups of individuals with I/DD to evaluate the tool’s viability, with respect to their ability to recognize/report abuse and use mobile-computing devices. We found individuals with I/DD could recognize/report abuse well when they received appropriate training. We also found individuals with I/DD could independently use their devices, though they shared access to them with family. Based on these findings, we call for several lines of accessibility research in designing an abuse self-reporting tool for individuals with I/DD.

Best practices for conducting HCI research on dementia care increasingly involve multiple stakeholders and incorporate diverse viewpoints. When done effectively, involving proxy stakeholders such as family members and professionals can help bring forward the voices of people with dementia. However, concrete practical guidance for navigating the challenges of integrating different perspectives is lacking. We critically reflect on our own recent qualitative fieldwork involving participants with dementia, family caregivers, and facilitators at a local social program for people with dementia, re-examining our interview transcripts and observation notes through content analysis. We illustrate practical approaches to prioritizing participants’ voices through concrete excerpts that demonstrate strategies for better managing dynamics, intervening effectively, and engaging all stakeholders in the research process. Our reflections and proposed guidelines can benefit HCI researchers and practitioners working with vulnerable populations. We hope this work will spur further discussion and critique to strengthen and improve research practices in this domain.

Content creators are instructed to write textual descriptions of visual content to make it accessible; yet existing guidelines lack specifics on how to write about people’s appearance, particularly while remaining mindful of consequences of (mis)representation. In this paper, we report on interviews with screen reader users who were also Black, Indigenous, People of Color, Non-binary, and/or Transgender on their current image description practices and preferences, and experiences negotiating theirs and others’ appearances non-visually. We discuss these perspectives, and the ethics of humans and AI describing appearance characteristics that may convey the race, gender, and disabilities of those photographed. In turn, we share considerations for more carefully describing appearance, and contexts in which such information is perceived salient. Finally, we offer tensions and questions for accessibility research to equitably consider politics and ecosystems in which technologies will embed, such as potential risks of human and AI biases amplifying through image descriptions.

Despite efforts to support students with disabilities in higher education, few continue to pursue doctoral degrees in computing. We conducted an interview study with 12 blind and low vision, and 7 deaf and hard of hearing current and former doctoral students in computing to understand how graduate students adjust to inaccessibility and ineffective accommodations. We asked participants how they worked around inaccessibility, managed ineffective accommodations, and advocated for tools and services. Employing a lens of ableism in our analysis, we found that participants’ extra effort to address accessibility gaps gave rise to a burden of survival, which they sustained to meet expectations of graduate-level productivity. We recommend equitable solutions that acknowledge taken-for-granted workarounds and that actively address inaccessibility in the graduate school context.

There is considerable effort within the HCI community to explore, document, and advocate for the lived experiences of persons with disabilities (PWDs). However, PWDs from the Global South, particularly Africa, are underrepresented in this scholarship. We contribute to closing this gap by investigating the unmet transit needs and characterization of technology within the disability community in Kampala, Uganda. We investigated transportation due to the increase in ride-share solutions created by widespread mobile computing and the resulting disruption of transportation worldwide. We hosted co-design sessions with disability advocates and adapted the stakeholder tokens method from the value-sensitive design framework to map the stakeholder ecosystem. Our key insight is the identification of a new group of non-traditional core stakeholders who highlight the values of inclusion, mobility, and safety within the ecosystem. Finally, we discuss how our findings engage with concepts of disability justice and perceptions of power.

Designers of digital content have access to various resources that they use to help them meet disabled people’s accessibility needs. Disability simulations are one resource, but often criticized for failing to guide digital designers appropriately, and it is unclear if digital designers are aware of the issues surrounding disability simulations. I surveyed 92 digital designers to understand their perspectives toward disability simulations (both perceived advantages and disadvantages). I then shared work process challenges faced by digital designers and their reasons for using disability simulations with 17 people with vision impairments to facilitate a discussion on this topic. The interviewees discussed ideas that suggest many paths can be explored to connect digital designers and disabled people, in general, to reduce reliance on simulations, and a change is needed within workplace processes, culture, and staffing to further support positive change. There are research opportunities to investigate establishing avenues for connecting digital designers and disabled people in a way that is beneficial to both groups.

Authentication has become increasingly ubiquitous for controlling access to personal computing devices (e.g., laptops, tablets, and smartphones). In this paper, we aim to understand the authentication process used by people with upper extremity impairment (UEI). A person with UEI lacks range of motion, strength, endurance, speed, and/or accuracy associated with arms, hands, or fingers. To this end, we conducted semi-structured interviews with eight (8) adults with UEI about their use of authentication for their personal computing devices. We found that our participants primarily use passwords and PINs as a verification credential during authentication. We found the process of authentication to have several accessibility issues for our participants. Consequently, our participants implemented a variety of workarounds that prioritized usability over security throughout the authentication process. Based on these findings, we present six broad subareas of research that should be explored in order to create more accessible authentication for people with UEI.

We conceptualize Digital Design Marginalization (DDM) as the process in which a digital interface design excludes certain users and contributes to marginalization in other areas of their lives. Due to non-inclusive designs, many underrepresented users face barriers in accessing essential services that are moving increasingly, sometimes exclusively, online – services such as personal finance, healthcare, social connectivity, and shopping. This can further perpetuate the “digital divide,” a technology-based form of social inequality that has offline consequences. We introduce the term Marginalizing Design to describe designs that contribute to DDM. In this paper, we focus on the impact of Marginalizing Design on older adults through examples from our research and discussions of services that may have marginalizing designs for older adults. Our aim is to provide a conceptual lens for designers, service providers, and policy makers through which they can use to purposely lessen or avoid digitally marginalizing groups of users.

Crowdwork can enable invaluable opportunities for people with disabilities, not least the work flexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how engagement in crowdwork tasks is affected by individual disabilities and the resulting implications for HCI. We first surveyed 1,000 Amazon Mechanical Turk (AMT) workers to identify demographics of crowdworkers who identify as having various disabilities within the AMT ecosystem—including vision, hearing, cognition/mental, mobility, reading and motor impairments. Through a second focused survey and follow-up interviews, we provide insights into how respondents cope with crowdwork tasks. We found that standard task factors, such as task completion time and presentation, often do not account for the needs of users with disabilities, resulting in anxiety and a feeling of depression on occasion. We discuss how to alleviate barriers to enable effective interaction for crowdworkers with disabilities.

The Trail Making Test (TMT) is a frequently used neuropsychological test for assessing cognitive performance. The subject connects a sequence of numbered nodes by using a pen on normal paper. We present an automatic cognitive assessment tool that analyzes samples of the TMT which we record using a digital pen. This enables us to analyze digital pen features that are difficult or impossible to evaluate manually. Our system automatically measures several pen features, including the completion time which is the main performance indicator used by clinicians to score the TMT in practice. In addition, our system provides a structured report of the analysis of the test, for example indicating missed or erroneously connected nodes, thereby offering more objective, transparent and explainable results to the clinician. We evaluate our system with 40 elderly subjects from a geriatrics daycare clinic of a large hospital.

Alerting customers on suspected online-payment fraud and persuade them to terminate transactions is increasingly requested with the rapid growth of digital finance worldwide. We explored the feasibility of using a conversational agent (CA) to fulfill this request. Shing, a voice-based CA, proactively initializes and repairs the conversation with empathetical communication skills in order to alert customers when a suspected online-payment fraud is detected, collects important information for fraud scrutiny and persuades customers to terminate the transaction once the fraud is confirmed. We evaluated our system by comparing it with a rule-based CA with regards to customer response and perceptions in a real-world context where our systems took 144,795 phone calls in total in which 83,019 (57.3%) natural breakdowns happened. Results showed that more customers stopped risky transactions after conversing with Shing. They seemed more willing to converse with Shing for more dialogue turns and provide transaction details. Our work presents practical implications for the design of proactive CA.

Building conversational agents that can conduct natural and prolonged conversations has been a major technical and design challenge, especially for community-facing conversational agents. We posit Mutual Theory of Mind as a theoretical framework to design for natural long-term human-AI interactions. From this perspective, we explore a community’s perception of a question-answering conversational agent through self-reported surveys and computational linguistic approach in the context of online education. We first examine long-term temporal changes in students’ perception of Jill Watson (JW), a virtual teaching assistant deployed in an online class discussion forum. We then explore the feasibility of inferring students’ perceptions of JW through linguistic features extracted from student-JW dialogues. We find that students’ perception of JW’s anthropomorphism and intelligence changed significantly over time. Regression analyses reveal that linguistic verbosity, readability, sentiment, diversity, and adaptability reflect student perception of JW. We discuss implications for building adaptive community-facing conversational agents as long-term companions and designing towards Mutual Theory of Mind in human-AI interaction.

Designing reliable Speech Emotion Recognition systems is a complex task that inevitably requires sufficient data for training purposes. Such extensive datasets are currently available in only a few languages, including English, German, and Italian. In this paper, we present SEMOUR, the first scripted database of emotion-tagged speech in the Urdu language, to design an Urdu Speech Recognition System. Our gender-balanced dataset contains 15,040 unique instances recorded by eight professional actors eliciting a syntactically complex script. The dataset is phonetically balanced, and reliably exhibits a varied set of emotions as marked by the high agreement scores among human raters in experiments. We also provide various baseline speech emotion prediction scores on the database, which could be used for various applications like personalized robot assistants, diagnosis of psychological disorders, and getting feedback from a low-tech-enabled population, etc. On a random test sample, our model correctly predicts an emotion with a state-of-the-art 92% accuracy.

Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.

In recent years, mobile accessibility has become an important trend with the goal of allowing all users the possibility of using any app without many limitations. User reviews include insights that are useful for app evolution. However, with the increase in the amount of received reviews, manually analyzing them is tedious and time-consuming, especially when searching for accessibility reviews. The goal of this paper is to support the automated identification of accessibility in user reviews, to help technology professionals in prioritizing their handling, and thus, creating more inclusive apps. Particularly, we design a model that takes as input accessibility user reviews, learns their keyword-based features, in order to make a binary decision, for a given review, on whether it is about accessibility or not. The model is evaluated using a total of 5,326 mobile app reviews. The findings show that (1) our model can accurately identify accessibility reviews, outperforming two baselines, namely keyword-based detector and a random classifier; (2) our model achieves an accuracy of 85% with relatively small training dataset; however, the accuracy improves as we increase the size of the training dataset.

Machine learning classifiers for human-facing tasks such as comment toxicity and misinformation often score highly on metrics such as ROC AUC but are received poorly in practice. Why this gap? Today, metrics such as ROC AUC, precision, and recall are used to measure technical performance; however, human-computer interaction observes that evaluation of human-facing systems should account for people’s reactions to the system. In this paper, we introduce a transformation that more closely aligns machine learning classification metrics with the values and methods of user-facing performance measures. The disagreement deconvolution takes in any multi-annotator (e.g., crowdsourced) dataset, disentangles stable opinions from noise by estimating intra-annotator consistency, and compares each test set prediction to the individual stable opinions from each annotator. Applying the disagreement deconvolution to existing social computing datasets, we find that current metrics dramatically overstate the performance of many human-facing machine learning tasks: for example, performance on a comment toxicity task is corrected from .95 to .73 ROC AUC.

Recent studies show the effectiveness of interview chatbots for information elicitation. However, designing an effective interview chatbot is non-trivial. Few tools exist to help designers design, evaluate, and improve an interview chatbot iteratively. Based on a formative study and literature reviews, we propose a computational framework for quantifying the performance of interview chatbots. Incorporating the framework, we have developed iChatProfile, an assistive chatbot design tool that can automatically generate a profile of an interview chatbot with quantified performance metrics and offer design suggestions for improving the chatbot based on such metrics. To validate the effectiveness of iChatProfile, we designed and conducted a between-subject study that compared the performance of 10 interview chatbots designed with or without using iChatProfile. Based on the live chats between the 10 chatbots and 1349 users, our results show that iChatProfile helped the designers build significantly more effective interview chatbots, improving both interview quality and user experience.

Recent work in fair machine learning has proposed dozens of technical definitions of algorithmic fairness and methods for enforcing these definitions. However, we still lack an understanding of how to develop machine learning systems with fairness criteria that reflect relevant stakeholders’ nuanced viewpoints in real-world contexts. To address this gap, we propose a framework for eliciting stakeholders’ subjective fairness notions. Combining a user interface that allows stakeholders to examine the data and the algorithm’s predictions with an interview protocol to probe stakeholders’ thoughts while they are interacting with the interface, we can identify stakeholders’ fairness beliefs and principles. We conduct a user study to evaluate our framework in the setting of a child maltreatment predictive system. Our evaluations show that the framework allows stakeholders to comprehensively convey their fairness viewpoints. We also discuss how our results can inform the design of predictive systems.

In single-target video object tracking, an initial bounding box is drawn around a target object and propagated through a video. When this bounding box is provided by a careful human expert, it is expected to yield strong overall tracking performance that can be mimicked at scale by novice crowd workers with the help of advanced quality control methods. However, we show through an investigation of 900 crowdsourced initializations that such quality control strategies are inadequate for this task in two major ways: first, the high level of redundancy in these methods (e.g., averaging multiple responses to reduce error) is unnecessary, as 23% of crowdsourced initializations perform just as well as the gold-standard initialization. Second, even nearly perfect initializations can lead to degraded long-term performance due to the complexity of object tracking. Considering these findings, we evaluate novel approaches for automatically selecting bounding boxes to re-query, and introduce Smart Replacement, an efficient method that decides whether to use the crowdsourced replacement initialization.

Advances in artificial intelligence (AI) have made it increasingly applicable to supplement expert’s decision-making in the form of a decision support system on various tasks. For instance, an AI-based system can provide therapists quantitative analysis on patient’s status to improve practices of rehabilitation assessment. However, there is limited knowledge on the potential of these systems. In this paper, we present the development and evaluation of an interactive AI-based system that supports collaborative decision making with therapists for rehabilitation assessment. This system automatically identifies salient features of assessment to generate patient-specific analysis for therapists, and tunes with their feedback. In two evaluations with therapists, we found that our system supports therapists significantly higher agreement on assessment (0.71 average F1-score) than a traditional system without analysis (0.66 average F1-score, p < 0.05). After tuning with therapist’s feedback, our system significantly improves its performance from 0.8377 to 0.9116 average F1-scores (p < 0.01). This work discusses the potential of a human-AI collaborative system to support more accurate decision making while learning from each other’s strengths.

Crowdsourcing can collect many diverse ideas by prompting ideators individually, but this can generate redundant ideas. Prior methods reduce redundancy by presenting peers’ ideas or peer-proposed prompts, but these require much human coordination. We introduce Directed Diversity, an automatic prompt selection approach that leverages language model embedding distances to maximize diversity. Ideators can be directed towards diverse prompts and away from prior ideas, thus improving their collective creativity. Since there are diverse metrics of diversity, we present a Diversity Prompting Evaluation Framework consolidating metrics from several research disciplines to analyze along the ideation chain — prompt selection, prompt creativity, prompt-ideation mediation, and ideation creativity. Using this framework, we evaluated Directed Diversity in a series of a simulation study and four user studies for the use case of crowdsourcing motivational messages to encourage physical activity. We show that automated diverse prompting can variously improve collective creativity across many nuanced metrics of diversity.

Qualitative research can produce a rich understanding of a phenomenon but requires an essential and strenuous data annotation process known as coding. Coding can be repetitive and time-consuming, particularly for large datasets. Existing AI-based approaches for partially automating coding, like supervised machine learning (ML) or explicit knowledge represented in code rules, require high technical literacy and lack transparency. Further, little is known about the interaction of researchers with AI-based coding assistance. We introduce Cody, an AI-based system that semi-automates coding through code rules and supervised ML. Cody supports researchers with interactively (re)defining code rules and uses ML to extend coding to unseen data. In two studies with qualitative researchers, we found that (1) code rules provide structure and transparency, (2) explanations are commonly desired but rarely used, (3) suggestions benefit coding quality rather than coding speed, increasing the intercoder reliability, calculated with Krippendorff’s Alpha, from 0.085 (MAXQDA) to 0.33 (Cody).

Developing human-like conversational agents is a prime area in HCI research and subsumes many tasks. Predicting listener backchannels is one such actively-researched task. While many studies have used different approaches for backchannel prediction, they all have depended on manual annotations for a large dataset. This is a bottleneck impacting the scalability of development. To this end, we propose using semi-supervised techniques to automate the process of identifying backchannels, thereby easing the annotation process. To analyze our identification module’s feasibility, we compared the backchannel prediction models trained on (a) manually-annotated and (b) semi-supervised labels. Quantitative analysis revealed that the proposed semi-supervised approach could attain 95% of the former’s performance. Our user-study findings revealed that almost 60% of the participants found the backchannel responses predicted by the proposed model more natural. Finally, we also analyzed the impact of personality on the type of backchannel signals and validated our findings in the user-study.

AI technologies are often used to aid people in performing discrete tasks with well-defined goals (e.g., recognising faces in images). Emerging technologies that provide continuous, real-time information enable more open-ended AI experiences. In partnership with a blind child, we explore the challenges and opportunities of designing human-AI interaction for a system intended to support social sensemaking. Adopting a research-through-design perspective, we reflect upon working with the uncertain capabilities of AI systems in the design of this experience. We contribute: (i) a concrete example of an open-ended AI system that enabled a blind child to extend his own capabilities; (ii) an illustration of the delta between imagined and actual use, highlighting how capabilities derive from the human-AI interaction and not the AI system alone; and (iii) a discussion of design choices to craft an ongoing human-AI interaction that addresses the challenge of uncertain outputs of AI systems.

The question of who gets to contribute to design futures and technology innovation has become a topic of conversation across HCI, CSCW, and other computing communities. This conversation has grave implications for communities that often find themselves an afterthought in technology design, and who coincidentally could benefit most from technological interventions in response to societal oppression. To explore this topic, we examined “futuring” through co-designed speculative design fictions as methods to envision utopian and dystopian futures. In a case study, we examined technology’s role in the imagined futures of youth participants of a Chicago summer design program. We highlight emerging themes and contribute an analysis of remote co-design through an Afrofuturism lens. Our analysis shows that concepts of utopian futures and technologies to support those futures are still heavily laden with dystopian realities of racism and poverty. We discuss ways that speculative design fictions and futuring can serve to address inclusivity in concept generation for new technologies, and we provide recommendations for conducting design techniques remotely with historically excluded populations.

Ranging from subtle to overt, unintentional to systemic, navigating racism is additional everyday work for many people. Yet the needs of people who experience racism have been overlooked as a fertile ground for better technology. Through a series of workshops we call Foundational Fiction, we engaged BIPOC (Black, Indigenous, People of Color) in participatory design to identify qualities of technology that can support people coping before, during, and after a racist interaction. Participants developed storyboards for digital tools that offer advice, predict consequences, identify racist remarks and intervene, educate both targets and perpetrators about interpersonal and systemic racism, and more. In the paper we present our workshop method utilizing interactive fiction, participants’ design concepts, prevalent themes (reducing uncertainty and offering comfort), and we provide critical analysis of the complexity of technology in these contexts. This work identifies specific opportunities for exploring anti-racist social tools.

In human-computer interaction (HCI) studies, bias in the gender representation of participants can jeopardize the generalizability of findings, perpetuate bias in data driven practices, and make new technologies dangerous for underrepresented groups. Key to progress towards inclusive and equitable gender practices is diagnosing the current status of bias and identifying where it comes from. In this mixed-methods study, we interviewed 13 HCI researchers to identify the potential bias factors, defined a systematic data collection procedure for meta-analysis of participant gender data, and created a participant gender dataset from 1,147 CHI papers. Our analysis provided empirical evidence for the underrepresentation of women, the invisibility of non-binary participants, deteriorating representation of women in MTurk studies, and characteristics of research topics prone to bias. Based on these findings, we make concrete suggestions for promoting inclusive community culture and equitable research practices in HCI.

Gender input forms act as gates to accessing information, websites, and services online. Non-binary people regularly have to interact with them, though many do not offer non-binary gender options. This results in non-binary individuals having to either choose an incorrect gender category or refrain from using a site or service—which is occasionally infeasible (e.g., when accessing health services). We tested five different forms through a survey with binary and non-binary participants (n = 350) in three contexts—a digital health form, a social media website, and a dating app. Our results indicate that the majority of participants found binary “male or female” forms exclusive and uncomfortable to fill out across all contexts. We conclude with design considerations for improving gender input forms and consequently their underlying gender model in databases. Our work aims to sensitize designers of (online) gender web forms to the needs and desires of non-binary people.

While multiple studies suggest that female-identified participants are more likely to experience cybersickness in virtual reality (VR), our systematic review of 71 eligible VR publications (59 studies and 12 surveys) pertaining to gender and cybersickness reveals a number of confounding factors in study design (e.g., a variety of technical specifications, tasks, content), a lack of demographic data, and a bias in participant recruitment. Our review shows an ongoing need within VR research to more consistently include and report on women’s experiences in VR to better understand the gendered possibility of cybersickness. Based on the gaps identified in our systematic review, we contribute study design recommendations for future work, arguing that gender considerations are necessary at every stage of VR study design, even when the study is not ‘about’ gender.

While makerspaces are often discussed in terms of a utopian vision of democratization and empowerment, many have shown how these narratives are problematic. There remains optimism for the future of makerspaces, but there is a gap in knowledge of how to articulate their promise and how to pursue it. We present a reflexive and critical reflection of our efforts as leaders of a university makerspace to articulate a vision, as well as our experience running a maker fashion show that aimed to address some specific critiques. We analyze interviews of participants from the fashion show using feminist utopianism as a lens to help us understand an alternate utopian narrative for making. Our contributions include insights about how a particular making context embodies feminist utopianism, insights about the applicability of feminist utopianism to makerspace research and visioning efforts, and a discussion about how our results can guide makerspace leaders and HCI researchers.

Affirmative consent is the idea that someone must ask for, and earn, enthusiastic approval before interacting with someone else. For decades, feminist activists and scholars have used affirmative consent to theorize and prevent sexual assault. In this paper, we ask: Can affirmative consent help to theorize online interaction? Drawing from feminist, legal, and HCI literature, we introduce the feminist theory of affirmative consent and use it to analyze social computing systems. We present affirmative consent’s five core concepts: it is voluntary, informed, revertible, specific, and unburdensome. Using these principles, this paper argues that affirmative consent is both an explanatory and generative theoretical framework. First, affirmative consent is a theoretical abstraction for explaining various problematic phenomena in social platforms—including mass online harassment, revenge porn, and problems with content feeds. Finally, we argue that affirmative consent is a generative theoretical foundation from which to imagine new design ideas for consentful socio-technical systems.

Self-care apps offer a wide variety of different therapy paradigms, pedagogies and concepts for people to maintain and make sense of their mental health. However, as human-made artefacts, these apps are being imbued with their designers’ interests, opinions, biases and assumptions about self-care. This paper is interested in making these (often) implicit notions visible. After selecting 69 apps from the Google Play Store, we use Feminist Content Analysis to investigate the store descriptions of these apps: Inductively through thematic analysis and deductively through charting concepts found within the descriptions. Our findings indicate that commercial self-care apps portray themselves as “future creating” tools for individual self-discovery, but they also create narratives that propagate an overly simplistic, individualistic and potentially harmful view of mental distress. We conclude this paper by sketching out alternative design considerations for how self-care apps can portray themselves and communicate in a more transparent, plurality-embracing fashion.

Sexual consent has undergone a transformation toward an “enthusiastic” feminist model that emphasizes consent as an ongoing and voluntary process of negotiation and affirmation. This paper considers how such a model can advance understandings of consent in HCI research and design in relation to embodied interactions with emerging technologies that also occur outside of sexual interactions. We apply the popular “FRIES” model of sexual consent (Freely given, Reversible, Informed, Enthusiastic and Specific) to three areas of embodied interaction: 1) bodily-play interactions, 2) persuasive interactions with smart technologies, and 3) intimate interactions with anthropomorphized devices. Based on erotic play practices, we contribute a “TEASE” process guideline (Traffic lights, Establish ongoing dialogue, Aftercare, Safewords, and Explicate soft/hard limits) to advance consensual practice in HCI and develop implementation scenarios.

Data-centered participatory design research projects—wherein researchers collaborate with community members for the purpose of gathering, generating, or communicating data about the community or their causes—can place epistemic burdens on minoritized or racialized groups, even in projects focused on social justice outcomes. Analysis of epistemic burden encourages researchers to rethink the purpose and value of data in community organizing and activism more generally. This paper describes three varieties of epistemic burden drawn from two case studies based on the authors’ previous work with anti-police brutality community organizations. The authors conclude with a discussion of ways to alleviate and avoid these issues through a series of questions about participatory research design. Ultimately, we call for a reorientation of knowledge production away from putative design solutions to community problems and toward a more robust interrogation of the power dynamics of research itself.

Smart technology turns the home into an active agent, which shifts the power structure within the household. This paper examines how initiators of smart technology insert their vision of the good life into households, and how these technologies exert power over the residents. Through a thematic analysis of interviews with five households, we consider Foucault’s theory on disciplinary power to examine how smart home technologies shape the experience of the home by shifting the flow of information and thereby reify power structures. Results indicate that the implementation of smart technology can affect access to shared spaces, constrain interactions, and predefine practices thereby establishing hierarchies within the household. We turn the discussion towards ethical challenges concerning control, whose problems the smart home is concerned with, and how the smart home embeds itself in the household. We conclude with design considerations and future work.

The menopause transition involves bodily-rooted, socially-shaped changes, often in a context of medicalisation that marginalises people based on their age and gender. With the goal of addressing this social justice matter with a participatory design approach, we started to cultivate partnerships with people going through menopause. This paper reports on interviews with 12 women and a design workshop with three. Our data analysis highlights their experiences from a holistic perspective that reclaims the primacy of the body and acknowledges the entanglement of the physical and the psychosocial. Participants’ design concepts show how design can come close the body to make space for menopause experiences, recognising and transforming them. We discuss how HCI can actively engage with the body to promote appreciation for it during menopause, and call for design that accompanies people in resisting the medicalisation of menopause as an enactment of social justice in everyday life.

Fertility tracking applications are technologies that collect sensitive information about their users i.e. reproductive potential. For many, these apps are an affordable solution when trying to conceive or managing their pregnancy. However, intimate data are not only collected but also shared beyond users knowledge or consent. In this paper, we explore the privacy risks that can originate from the mismanagement, misuse, and misappropriation of intimate data, which are entwined in individual life events and in public health issues such as abortion and (in)fertility. We look at differential vulnerabilities to enquire data’s vulnerability and that of ‘data subjects’. We introduce the General Data Protection Regulation (GDPR) and how it addresses fertility data. We evaluate the privacy of 30 top ‘fertility apps’ through their privacy notices and tracking practices. Lastly, we discuss the regulations and fertility data as critical to the future design of tracking technologies and privacy rights.

Today’s smart cities use thousands of physical sensors distributed across the urban landscape to support decision making in areas such as infrastructure monitoring, public health, and resource management. These weather-hardened devices require power and connectivity, and often cost thousands just to install, let alone maintain. In this paper, we show how long-range laser vibrometry can be used for low-cost, city-scale sensing. Although typically limited to just a few meters of sensing range, the use of retroreflective markers can boost this to 1km or more. Fortuitously, cities already make extensive use of retroreflective materials for street signs, construction barriers, road studs, license plates, and many other markings. We describe how our prototype system can co-opt these existing markers at very long ranges and use them as unpowered accelerometers for use in a wide variety of sensing applications.

Most augmented reality (AR) authoring tools only support the author’s current environment, but designers often need to create site-specific experiences for a different environment. We propose DistanciAR, a novel tablet-based workflow for remote AR authoring. Our baseline solution involves three steps. A remote environment is captured by a camera with LiDAR; then, the author creates an AR experience from a different location using AR interactions; finally, a remote viewer consumes the AR content on site. A formative study revealed understanding and navigating the remote space as key challenges with this solution. We improved the authoring interface by adding two novel modes: Dollhouse, which renders a bird’s-eye view, and Peek, which creates photorealistic composite images using captured images. A second study compared this improved system with the baseline, and participants reported that the new modes made it easier to understand and navigate the remote scene.

3D sketching in virtual reality (VR) enables users to create 3D virtual objects intuitively and immersively. However, previous studies showed that mid-air drawing may lead to inaccurate sketches. To address this issue, we propose to use one hand as a canvas proxy and the index finger of the other hand as a 3D pen. To this end, we first perform a formative study to compare two-handed interaction with tablet-pen interaction for VR sketching. Based on the findings of this study, we design HandPainter, a VR sketching system which focuses on the direct use of two hands for 3D sketching without requesting any tablet, pen, or VR controller. Our implementation is based on a pair of VR gloves, which provide hand tracking and gesture capture. We devise a set of intuitive gestures to control various functionalities required during 3D sketching, such as canvas panning and drawing positioning. We show the effectiveness of HandPainter by presenting a number of sketching results and discussing the outcomes of a user study-based comparison with mid-air drawing and tablet-based sketching tools.

Despite the central importance of research papers to scientific progress, they can be difficult to read. Comprehension is often stymied when the information needed to understand a passage resides somewhere else—in another section, or in another paper. In this work, we envision how interfaces can bring definitions of technical terms and symbols to readers when and where they need them most. We introduce ScholarPhi, an augmented reading interface with four novel features: (1) tooltips that surface position-sensitive definitions from elsewhere in a paper, (2) a filter over the paper that “declutters” it to reveal how the term or symbol is used across the paper, (3) automatic equation diagrams that expose multiple definitions in parallel, and (4) an automatically generated glossary of important terms and symbols. A usability study showed that the tool helps researchers of all experience levels read papers. Furthermore, researchers were eager to have ScholarPhi’s definitions available to support their everyday reading.

Human-robot interaction designers and developers navigate a complex design space, which creates a need for tools that support intuitive design processes and harness the programming capacity of state-of-the-art authoring environments. We introduce Figaro, an expressive tabletop authoring environment for mobile robots, inspired by shadow puppetry, that provides designers with a natural, situated representation of human-robot interactions while exploiting the intuitiveness of tabletop and tangible programming interfaces. On the tabletop, Figaro projects a representation of an environment. Users demonstrate sequences of behaviors, or scenes, of an interaction by manipulating instrumented figurines that represent the robot and the human. During a scene, Figaro records the movement of figurines on the tabletop and narrations uttered by users. Subsequently, Figaro employs real-time program synthesis to assemble a complete robot program from all scenes provided. Through a user study, we demonstrate the ability of Figaro to support design exploration and development for human-robot interaction.

Prototyping electronic devices that meet today’s consumer standards is a time-consuming task that requires multi-domain expertise. Consumers expect electronic devices to have visually appealing interfaces with both tactile and screen-based interfaces. Appliancizer, our interactive computational design tool, exploits the similarities between graphical and tangible interfaces, allowing web pages to be rapidly transformed into physical electronic devices. Using a novel technique we call essential interface mapping, our tool converts graphical user interface elements (e.g., an HTML button) into tangible interface components (e.g., a physical button) without changing the application source code. Appliancizer automatically generates the PCB and low-level code from web-based prototypes and HTML mock-ups. This makes the prototyping of mixed graphical-tangible interactions as easy as modifying a web page and allows designers to leverage the well-developed ecosystem of web technologies. We demonstrate how our technique simplifies and accelerates prototyping by developing two devices with Appliancizer.

User Interface (UI) mockups are commonly used as shared context during interface development collaboration. In practice, UI designers often use screenshots and sketches to create mockups of desired UI behaviors for communication. However, in the later stages of UI development, interfaces can be arbitrarily complex, making it labor-intensive to sketch, and static screenshots are limited in the types of interactive and dynamic behaviors they can express. We introduce CoCapture, a system that allows designers to easily create UI behavior mockups on existing web interfaces by demonstrating and remixing, and to accurately describe their requests to helpers by referencing the resulting mockups using hypertext. We showed that participants could more accurately describe UI behaviors with CoCapture than with existing sketch and communication tools and that the resulting descriptions were clear and easy to follow. Our approach can help teams develop UIs efficiently by bridging communication gaps with more accurate visual context.

Modern manufacturing processes are in a state of flux, as they adapt to increasing demand for flexible and self-configuring production. This poses challenges for training workers to rapidly master new machine operations and processes, i.e. machine tasks. Conventional in-person training is effective but requires time and effort of experts for each worker trained and not scalable. Recorded tutorials, such as video-based or augmented reality (AR), permit more efficient scaling. However, unlike in-person tutoring, existing recorded tutorials lack the ability to adapt to workers’ diverse experiences and learning behaviors. We present AdapTutAR, an adaptive task tutoring system that enables experts to record machine task tutorials via embodied demonstration and train learners with different AR tutoring contents adapting to each user’s characteristics. The adaptation is achieved by continually monitoring learners’ tutorial-following status and adjusting the tutoring content on-the-fly and in-situ. The results of our user study evaluation have demonstrated that our adaptive system is more effective and preferable than the non-adaptive one.

Haptic feedback significantly enhances virtual experiences. However, supporting haptics currently requires modifying the codebase, making it impractical to add haptics to popular, high-quality experiences such as best selling games, which are typically closed-source. We present HapticSeer, a multi-channel, black-box, platform-agnostic approach to detecting game events for real-time haptic feedback. The approach is based on two key insights: 1) all games have 3 types of data streams: video, audio, and controller I/O, that can be analyzed in real-time to detect game events, and 2) a small number of user interface design patterns are reused across most games, so that event detectors can be reused effectively. We developed an open-source HapticSeer framework and implemented several real-time event detectors for commercial PC and VR games. We validated system correctness and real-time performance, and discuss feedback from several haptics developers that used the HapticSeer framework to integrate research and commercial haptic devices.

Tangibles on capacitive touchscreens are a promising approach to overcome the limited expressiveness of touch input. While research has suggested many approaches to detect tangibles, the corresponding tangibles are either costly or have a considerable minimal size. This makes them bulky and unattractive for many applications. At the same time, they obscure valuable display space for interaction.

To address these shortcomings, we contribute Itsy-Bits: a fabrication pipeline for 3D printing and recognition of tangibles on capacitive touchscreens with a footprint as small as a fingertip. Each Itsy-Bit consists of an enclosing 3D object and a unique conductive 2D shape on its bottom. Using only raw data of commodity capacitive touchscreens, Itsy-Bits reliably identifies and locates a variety of shapes in different sizes and estimates their orientation. Through example applications and a technical evaluation, we demonstrate the feasibility and applicability of Itsy-Bits for tangibles with small footprints.

3D printing has revolutionized rapid prototyping by speeding up the creation of custom-shaped objects. With the rise of multi-material 3D printers, these custom-shaped objects can now be made interactive in a single pass through passive conductive structures. However, connecting conventional electronics to these conductive structures often still requires time-consuming manual assembly involving many wires, soldering or gluing.

To alleviate these shortcomings, we propose : a fabrication pipeline and interfacing concept to magnetically connect a 3D-printed object equipped with passive sensing structures to conventional sensing electronics. To this end, utilizes ferromagnetic and conductive 3D-printed structures, printable in a single pass on standard printers. We further present a proof-of-concept capacitive sensing board that enables easy and robust magnetic assembly to quickly create interactive 3D-printed objects. We evaluate by assessing the robustness and quality of the connection and demonstrate its broad applicability by a series of example applications.

Tangibles can enrich interaction with digital surfaces. Among others, they support eyes-free control or increase awareness of other users’ actions. Tangibles have been studied in combination with horizontal surfaces such as tabletops, but not with vertical screens such as wall displays. The obvious obstacle is gravity: tangibles cannot be placed on such surfaces without falling. We present WallTokens, easy-to-fabricate tangibles to interact with a vertical surface. A WallToken is a passive token whose footprint is recognized on a tactile surface. It is equipped with a push-handle that controls a suction cup. This makes it easy for users to switch between sliding the token or attaching it to the wall. We describe how to build such tokens and how to recognize them on a tactile surface. We report on a study showing the benefits of WallTokens for manipulating virtual objects over multi-touch gestures. This project is a step towards enabling tangible interaction in a wall display context.

Acoustic levitation is gaining popularity as an approach to create physicalized mid-air content by levitating different types of levitation primitives. Such primitives can be independent particles or particles that are physically connected via threads or pieces of cloth to form shapes in mid-air. However, initialization (i.e., placement of such primitives in their mid-air target locations) currently relies on either manual placement or specialized ad-hoc implementations, which limits their practical usage. We present ArticuLev, an integrated pipeline that deals with the identification, assembly and mid-air placement of levitated shape primitives. We designed ArticuLev with the physical properties of commonly used levitation primitives in mind. It enables experiences that seamlessly combine different primitives into meaningful structures (including fully articulated animated shapes) and supports various levitation display approaches (e.g., particles moving at high speed). In this paper, we describe our pipeline and demonstrate it with heterogeneous combinations of levitation primitives.

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface’s view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.

Documentation for DIY tasks serve as codified project knowledge and help makers reach new understandings and appreciations for the artifact. Engaging in reflective processes using the documentation can be challenging when it comes to physical objects as the documentation and the artifact exist separately. We hypothesize that spatially associating the documentation information with the artifact can provide richer contextualization to reflect upon the artifact and design process. We implemented and evaluated Documented, a web application that helps makers associate documentation to 3D printed objects. Information can be embedded using printed tags spatially placed on the model and accessed using mobile AR. Our study highlights the different strategies participants had for organizing, embedding, and retrieving information. Informed by our results, we discuss how the coupling of the documentation and the artifact can support reflection and identify potential barriers that need further investigation.

Flower jellies, a delicate dessert in which a flower-shaped jelly floats inside another clear jelly, fascinate people with both their beauty and elaborate construction. In efforts to simplify the challenging fabrication and enrich the design space of this dessert, we present Flower Jelly Printer: a printing device and design software for digitally fabricating flower jellies. Our design software lets users play with parameters and preview the resulting forms until achieving their desired shapes. We also developed slit injection printing that directly injects colored jelly into a base jelly, and shared several design examples to show the breadth of design possibilities. Finally, the user study with novice and experienced users demonstrates that our system benefits creators of all experience levels by iterative design and precise fabrication. We hope to enable more people to design and create their own flower jellies while expanding access and the design space for digitally fabricated foods.

Unprecedented maker efforts arose in response to COVID-19 medical supply gaps worldwide. Makers in the U.S., participated in peer-production activities to manufacture personal protective equipment (PPE). Whereas, medical makers, who innovate exclusively for points of care, pivoted towards safer, reliable PPE. What were their efforts to pivot medical maker infrastructure towards reliable production of safe equipment at higher volumes? We interviewed 13 medical makers as links between institutions, maker communities, and wider regional industry networks. These medical makers organized stopgap manufacturing in institutional spaces to resolve acute shortages (March–May) and chronic shortages (May–July). They act as intermediaries in efforts to prototype and produce devices under regulatory, material, and human constraints of a pandemic. We re-frame their making efforts as repair work to offer an alternate critical view of optimism around making for crisis. We contribute an understanding of these efforts to inform infrastructure design for making with purpose and safety leading to opportunities for community production of safe devices at scale.

Visual metaphors, which are widely used in graphic design, can deliver messages in creative ways by fusing different objects. The keys to creating visual metaphors are diverse exploration and creative combinations, which is challenging with conventional methods like image searching. To streamline this ideation process, we propose to use a mind-map-like structure to recommend and assist users to explore materials. We present MetaMap, a supporting tool which inspires visual metaphor ideation through multi-dimensional example-based exploration. To facilitate the divergence and convergence of the ideation process, MetaMap provides 1) sample images based on keyword association and color filtering; 2) example-based exploration in semantics, color, and shape dimensions; and 3) thinking path tracking and idea recording. We conduct a within-subject study with 24 design enthusiasts by taking a Pinterest-like interface as the baseline. Our evaluation results suggest that MetaMap provides an engaging ideation process and helps participants create diverse and creative ideas.

Mathematical models and expressions traditionally evolved as symbolic representations, with cognitively arbitrary rules of symbol manipulation. The embodied mathematics philosophy posits that abstract math concepts are layers of metaphors grounded in our intuitive arithmetic capabilities, such as categorizing objects and part-whole analysis. We introduce a design framework that facilitates the construction and exploration of embodied representations for algebraic expressions, using interactions inspired by innate arithmetic capabilities. We instantiated our design in a sketch interface that enables construction of visually interpretable compositions that are directly mappable to algebraic expressions and explorable through a ladder of abstraction [47]. The emphasis is on bottom-up construction, with the user sketching pictures while the system generates corresponding algebra. We present diverse examples created by our prototype system. A coverage of the US Common Core curriculum and playtesting studies with children point to the future direction and potential for a sketch-based design paradigm for mathematics.

The access and growing ubiquity of digital fabrication has ushered in a celebration of creativity and “making.” However, the focus is often on the resulting static artifact or the creative process and tools to design it. We envision a post-making process that extends past these final static objects — not just in their making but in their “unmaking.” By drawing from artistic movements such as Auto-Destructive Art, intentionally inverting well-established engineering principles of structurally sound designs, and safely misusing unstable materials, we demonstrate an important extension to making — unmaking. In this paper, we provide designers with a new vocabulary of unmaking operations within standard 3D modeling tools. We demonstrate how such designs can be realized using a novel multi-material 3D printing process. Finally, we detail how unmaking allows designs to change over time, is an ally to sustainability and re-usability, and captures themes of “aura,” emotionality, and personalization.

Though photographs of real people are typically used to portray personas, there is little research into the potential advantages or disadvantages of using such images, relative to other image styles. We conducted an experiment with 149 participants, testing the effects of six different image styles on user perceptions and personality traits that are attributed to personas by the participants. Results show that perceptions of clarity, completeness, consistency, credibility, and empathy for a persona increase with picture realism. Personas with more realistic pictures are also perceived as more agreeable, open, and emotionally stable, with higher confidence in these assessments. We also find evidence of the uncanny valley effect, with realistic cartoon personas experiencing a decrease in the user perception scores.

Digital tools that support creative activities are ubiquitous in the design industry, yet practitioners appear to prefer pen and paper for design ideation. To better understand this exception, we conducted a comparative study between analog and digital tools and their impact on the divergent and convergent thinking patterns of groups of designers. We analysed how 24 participants solved comparable design ideation tasks in two conditions using linkographic protocol analysis – a notation method that focuses on identifying and linking small steps in the design process called moves. Our findings suggest that digital ideation tools yield more convergent thinking compared to analog tools, with no discernible impact on general productivity or divergent thinking.

3D printing, as a rapid prototyping technique, usually fabricates objects that are difficult to modify physically. This paper presents FlexTruss, a design and construction pipeline based on the assembly of modularized truss-shaped objects fabricated with conventional 3D printers and assembled by threading. To create an end-to-end system, a parametric design tool with an optimal Euler path calculation method is developed, which can support both inverse and forward design workflow and multi-material construction of modular parts. In addition, the assembly of truss modules by threading is evaluated with a series of application cases to demonstrate the affordance of FlexTruss. We believe that FlexTruss extends the design space of 3D printing beyond typically hard and fixed forms, and it will provide new capabilities for designers and researchers to explore the use of such flexible truss structures in human-object interaction.

3D printing technology makes Do-It-Yourself and reforming everyday objects a reality. However, designing and fabricating attachments that can seamlessly adapt existing objects to extended functionality is a laborious process, which requires accurate measuring, modeling, manufacturing, and assembly. This paper presents ShrinCage, a 4D printing system that allows novices to easily create shrinkable adaptations to fit and fasten existing objects. Specifically, the design tool presented in this work aid in the design of attachment that adapts to irregular morphologies, which accommodates the variations in measurements and fabrication, subsequently simplifying the modeling and assembly processes. We further conduct mechanical tests and user studies to evaluate the availability and feasibility of this method. Numerous application examples created by ShrinCage prove that it can be adopted by aesthetic modification, assistive technology, repair, upcycling, and augmented 3D printing.

Digital fabrication and craftsmanship is entering into a new phase with increasing levels of complexity and a renewed desire for composites and cross-material experimentation. However, allowing work to travel from machine to machine, remains a challenge in terms of workflow, communication, orientation and material. Based on an exploration to combine embroidery and 3D printing in the pursuit of inflatable solutions, we propose the metaphor of the drawing game Exquisite Corpse to outline the three emerging concerns: turn taking, orientation and trade-offs. We propose a set of guidelines that suggest ways in which, we may allow different digital fabrication machines to be used in sequence, as a method for adding complexity to the things we make and the ways our machines may talk to one another.

Interest in combining interactive play and the human body, using “bodily play” systems, is increasing. While these systems primarily prioritize a player's control over their bodily actions, we see intriguing possibilities in the pursuit of “limited control over the body” as an intriguing design resource for bodily play systems. In this paper, we use three of our bodily play systems to illustrate how designers can engage with limited control over the body by varying the player's degree of indirect control (for instance, via other bodily activity and external triggers). We also propose four strategies for employing limited control over the body: Exploration, Reflection, Learning and Embracement. We hope our own work and the strategies developed from it will assist designers to employ limited control over the body, ultimately helping people benefit from engaging their bodies through play.

Prior research has highlighted opportunities for technology to better support the tabletop game experience in offline and online settings, but little work has focused on the social aspect of tabletop gaming. We investigated the social and collaborative aspects of tabletop gaming in the unique context of “social distancing” during the 2020 COVID-19 pandemic to shed light on the experience of remote tabletop gaming. With a multi-method qualitative approach (including digital ethnography and in-depth interviews), we empirically studied how people appropriate existing technologies and adapt their offline practices to play tabletop games remotely. We identify three themes that describe people's game and social experience during remote play: creating a shared tabletop environment (shared space), enabling a collective understanding (shared information and awareness), and facilitating a communal temporal experience (shared time). We reflect on challenges and design opportunities for a better experience in the age of remote collaboration.

Online platforms rely upon users or automated tools to flag toxic behaviors, the very first step in online moderation. While much recent research has examined online moderation, the role of flag remains poorly understood. This question becomes even more urgent in automated moderation, where flagging becomes a primary source of human judgment. We conducted a qualitative study of flagging practices in League of Legends (LoL), a popular eSports game. We found stark differences between how flag is designed to identify toxicity, and flaggability, or how players use and appropriate flag. Players distrust flag, but also appropriate flag for instrumental purposes. Thus, flaggability diverges decidedly from the conception of toxicity, and must be understood within the highly competitive gaming context of LoL. These findings help shed light on the situated nature of flaggability, the role of flag in online moderation, as well as implications for designing flag and moderation.

Video game toxicity, endemic to online play, represents a pervasive and complex problem. Antisocial behaviours in online play directly harm player wellbeing, enjoyment, and retention—but research has also revealed that some players normalize toxicity as an inextricable and acceptable element of the competitive video game experience. In this work, we explore perceptions of toxicity and how they are predicted by player traits, demonstrating that participants reporting a higher tendency towards Conduct Reconstrual, Distorting Consequences, Dehumanization, and Toxic Online Disinhibition perceive online game interactions as less toxic. Through a thematic analysis on willingness to report, we also demonstrate that players abstain from reporting toxic content because they view it as acceptable, typical of games, as banter, or as not their concern. We propose that these traits and themes represent contributing factors to the cyclical normalization of toxicity. These findings further highlight the multifaceted nature of toxicity in online video games.

Uncertainty is widely acknowledged as an engaging gameplay element but rarely used in exergames. In this research, we explore the role of uncertainty in exergames and introduce three uncertain elements (false-attacks, misses, and critical hits) to an exergame. We conducted a study under two conditions (uncertain and certain), with two display types (virtual reality and large display) and across young and middle-aged adults to measure their effect on game performance, experience, and exertion. Results show that (1) our designed uncertain elements are instrumental in increasing exertion levels; (2) when playing a motion-based first-person perspective exergame, virtual reality can improve performance, while maintaining the same motion sickness level as a large display; and (3) exergames for middle-aged adults should be designed with age-related declines in mind, similar to designing for elderly adults. We also framed two design guidelines for exergames that have similar features to the game used in this research.

Warm-up games are widespread practices in multiple activities across domains, yet little scholarly work can be found about their role in physical training. Here, we study potential goals and benefits of warm-up games, and explore opportunities for technology inclusion through investigating a collection of warm-up games gathered: online, from a survey of online warm-up games curated, described, and used by Physical Education teachers; and in person, from an ongoing design research work as part of a technology-supported circus training course. Further, in the context of the latter, we conducted explorative design interventions, augmenting a range of the warm-up games with wearable technology. Our work surfaces major goals and benefits of warm-up games, which can be broadly classified as preparing participants physically, socially, and mentally. We also show how the inclusion of open-ended technology can support these goals and discuss broader opportunities for technology inclusion in warm-up games.

This paper presents restraints - directly imposed restrictions on players' bodily movements, as a mechanic for bodily play in HCI. While this is a familiar mechanic in non-digital movement-based games, its potential in designing bodily play experiences in HCI has been scarcely explored. Three types of restraints observed in non-digital movement-based games, are explored here: fixating body parts, excluding body parts and depriving/manipulating bodily senses. Then, we investigate the experiential dynamics of restraints as a bodily play mechanic bridging a phenomenological perspective on bodily movement with theories on play. These investigations form the theoretical framework for the subsequent analysis of five digital body game examples. Building on this analysis and theoretical framework, we formulate five design strategies for implementing restraints as a mechanic for bodily play in HCI. We propose restraints as a generative resource for researchers and designers interested in understanding and designing bodily play experiences in HCI.

The design space of social exergames remains narrow despite the many benefits of playing and exercising together. Towards opening this design space, we followed a Research through Design (RtD) approach focused on exergames that can be fun and immersive social training experiences. Through embodied sketching activities with designers and 10 pairs of players, we explored future games for the ExerCube, an immersive exergame platform. Our work contributes with forms of intermediate-level knowledge: a design space model (the Immersive Social Fitness—ImSoFit—Games model); and a novel design vocabulary including new bodily orientations in co-located physical interaction. We illustrate their use and value scrutinizing three of our games and applying three analytical lenses to 1) understand how design choices impact how players move together; 2) evaluate design expectations and analyze players’ behavior in relation to design choices; and 3) potentially extend the design space of immersive co-located social fitness games.

Virtual reality (VR) multiplayer games increasingly use asymmetry (e.g., differences in a person’s capability or the user interface) and resulting interdependence between players to create engagement even when one player has no access to a head-mounted display (HMD). Previous work shows this enhances player experience (PX). Until now, it remains unclear whether and how an asymmetric game design with interdependences creates comparably enjoyable PX for both an HMD and a non-HMD player. In this work, we designed and implemented an asymmetric VR game (different in its user interface) with two types of interdependence: strategic (difference in game information/player capability) and biometric (difference in player’s biometric influence). Our mixed-methods user study (N=30) shows that asymmetries positively impact PX for both player roles, that interdependence strongly affects players’ perception of agency, and that biometric feedback—while subjective—is a valuable game mechanic.

Recent studies on gaming wearables show that wearables can contribute to the gaming experience by bolstering performativity, facilitating social interaction, and accommodating distinct interaction modalities. Still, these studies focused on contexts such as role-playing, casual, or festival games. Stakeholder-oriented research that explores the integration of wearables for mainstream gaming platforms such as game consoles is scarce. To fill this gap, we have conducted an exploratory study through 6 participatory design workshops focusing on different aspects of wearables with 33 participants from different stakeholders. As a result, we have created fifteen design themes and three gaming wearable concepts that led to seven actionable design implications which can be adopted by designers and researchers for designing gaming wearables.

The spectatorship experience for virtual reality (VR) games differs strongly from its non-VR precursor. When watching non-VR games on platforms such as Twitch, spectators just see what the player sees, as the physical interaction is mostly unimportant for the overall impression. In VR, the immersive full-body interaction is a crucial part of the player experience. Hence, content creators, such as streamers, often rely on green screens or similar solutions to offer a mixed-reality third-person view to disclose their full-body actions. Our work compares the most popular realizations of the first-person and the third-person perspective in an online survey (N = 217) with three different VR games. Contrary to the current trend to stream in third-person, our key result is that most viewers prefer the first-person version, which they attribute mostly to the better focus on in-game actions and higher involvement. Based on the study insights, we provide design recommendations for both perspectives.

While virtual character embodiment has been studied as a mitigator of singular societal biases in fully immersive VR and empathy games, there have been no major studies on representation featuring standard game play or intersectional identities. In our study, participants played a short 2D video game with racial and gender character manipulations. They then rated a LinkedIn profile application to examine interactions of racial and gender biases. White male participants showed bias against black and female applicants, with the black female applicant experiencing both racial and gender bias. However, participants who embodied certain underrepresented characters in the game displayed reduced biases. Participants’ perceived identification with the characters moderated this effect. The study highlights a lack of homogeneity in the prevalence and potential reduction of different societal biases and incorporates intersectionality to illustrate how multiple parts of a player character’s identity can be used to combat biases.

Many journalists and newsrooms now incorporate audience contributions in their sourcing practices by leveraging user-generated content (UGC). However, their sourcing needs and practices as they seek information from UGCs are still not deeply understood by researchers or well-supported in tools. This paper first reports the results of a qualitative interview study with nine professional journalists about their UGC sourcing practices, detailing what journalists typically look for in UGCs and elaborating on two UGC sourcing approaches: deep reporting and wide reporting. These findings then inform a human-centered design approach to prototype a UGC sourcing tool for journalists, which enables journalists to interactively filter and rank UGCs based on users’ example content. We evaluate the prototype with nine professional journalists who source UGCs in their daily routines to understand how UGC sourcing practices are enabled and transformed, while also uncovering opportunities for future research and design to support journalistic sourcing practices and sensemaking processes.

Virtual meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In-meeting multitasking is closely linked to people’s productivity and wellbeing. However, we currently have limited understanding of multitasking in remote meetings and its potential impact. In this paper, we present what we believe is the most comprehensive study of remote meeting multitasking behavior through an analysis of a large-scale telemetry dataset collected from February to May 2020 of U.S. Microsoft employees and a 715-person diary study. Our results demonstrate that intrinsic meeting characteristics such as size, length, time, and type, significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. Our findings suggest important best-practice guidelines for remote meetings (e.g., avoid important meetings in the morning) and design implications for productivity tools (e.g., support positive remote multitasking).

Targeting the right group of workers for crowdsourcing often achieves better quality results. One unique example of targeted crowdsourcing is seeking community-situated workers whose familiarity with the background and the norms of a particular group can help produce better outcome or accuracy. These community-situated crowd workers can be recruited in different ways from generic online crowdsourcing platforms or from online recovery communities. We evaluate three different approaches to recruit generic and community-situated crowd in terms of the time and the cost of recruitment, and the accuracy of task completion. We consider the context of Alcoholics Anonymous (AA), the largest peer support group for recovering alcoholics, and the task of identifying and validating AA meeting information. We discuss the benefits and trade-offs of recruiting paid vs. unpaid community-situated workers and provide implications for future research in the recovery context and relevant domains of HCI, and for the design of crowdsourcing ICT systems.

Drones are increasingly used to support humanitarian crises and events that involve dangerous or costly tasks. While drones have great potential for remote collaborative work and aerial telepresence, existing drone technology is limited in its support for synchronous collaboration among multiple remote users. Through three design iterations and evaluations, we prototyped Squadrone, a novel aerial telepresence platform that supports synchronous mid-air collaboration among multiple remote users. We present our design and report results from evaluating our iterations with 13 participants in 3 different collaboration configurations. Our first design iteration validates the basic functionality of the platform. Then, we establish the effectiveness of collaboration using a 360-degree shared aerial display. Finally, we simulate a type of search task in an open environment to see if collaborative telepresence impacts members’ participation. The results validate some initial goals for Squadrone and are used to reflect back on a recent telepresence design framework.

In hybrid meetings, multiple co-located participants communicate with remote participants through video. But video communication inhibits non-verbal cues, and this often causes remote participants to feel excluded. To address this issue, we built MirrorBlender: a What-You-See-Is-What-I-See video-conferencing system for blending, repositioning, and resizing mirrors. Mirrors here denote shared video feeds of people and screens. In a qualitative study of MirrorBlender with three hybrid meeting sessions, we found that the shared control of mirrors supported users in negotiating a blended interpersonal space. Moreover, it enabled diverse acts of inclusion of remote participants. In particular, remote participants brought attention to themselves by manipulating the position, scale, and translucency of their camera and screen feeds. Participants also embodied and leveraged their mirror images for deictic gestures and playful interactions. Based on these findings, we discuss new opportunities for supporting video-mediated collaboration.

Social media has become an effective recruitment tool for higher-waged and white-collar professionals. Yet, past studies have questioned its effectiveness for the recruitment of lower-waged workers. It is also unclear whether or how employers leverage social media in their recruitment of low-wage job seekers, and how social media could better support the needs of both stakeholders. Therefore, we conducted 15 semi-structured interviews with employers of low-wage workers in the U.S. We found that employers: use social media, primarily Facebook, to access large pools of active low-wage job seekers; and recognize indirect signals about low-wage job seekers’ commitment and job readiness. Our work suggests that there remains a visible, yet unaddressed power imbalance between low-wage workers and employers in the use of social media, which risks further destabilizing the precarious labor market.

Team members commonly collaborate on visual documents remotely and asynchronously. Particularly, students are frequently restricted to this setting as they often do not share work schedules or physical workspaces. As communication in this setting has delays and limits the main modality to text, members exert more effort to reference document objects and understand others’ intentions. We propose Winder, a Figma plugin that addresses these challenges through linked tapes—multimodal comments of clicks and voice. Bidirectional links between the clicked-on objects and voice recordings facilitate understanding tapes: selecting objects retrieves relevant recordings, and playing recordings highlights related objects. By periodically prompting users to produce tapes, Winder preemptively obtains information to satisfy potential communication needs. Through a five-day study with eight teams of three, we evaluated the system’s impact on teams asynchronously designing graphical user interfaces. Our findings revealed that producing linked tapes could be as lightweight as face-to-face (F2F) interactions while transmitting intentions more precisely than text. Furthermore, with preempted tapes, teammates coordinated tasks and invited members to build on each others’ work.

Augmented/Virtual Reality (AR/VR) is still a fragmented space to design for due to the rapidly evolving hardware, the interdisciplinarity of teams, and a lack of standards and best practices. We interviewed 26 professional AR/VR designers and developers to shed light on their tasks, approaches, tools, and challenges. Based on their work and the artifacts they generated, we found that AR/VR application creators fulfill four roles: concept developers, interaction designers, content authors, and technical developers. One person often incorporates multiple roles and faces a variety of challenges during the design process from the initial contextual analysis to the deployment. From analysis of their tool sets, methods, and artifacts, we describe critical key challenges. Finally, we discuss the importance of prototyping for the communication in AR/VR development teams and highlight design implications for future tools to create a more usable AR/VR tool chain.

Freelancing platforms, such as Upwork, represent an expansion of the gig economy to encompass knowledge-based work. Prior research in HCI has primarily focused on forms of gig work such as ride-sharing and microwork but has not addressed how freelancing platforms are disrupting high-skilled knowledge work. To understand freelancers’ perspectives on how these platforms are disrupting their work we have collected and thematically analysed 528 posts with 7499 comments from four relevant subforums on Reddit. The qualitative findings reveal tensions between wanting autonomy and control and the necessity of opportunities and convenience. Freelancing platforms are perceived as systems that present advantages to find clients, gain experience and mitigate precarity. However, these platforms constrain the control over their work that freelancers value. The paper contributes an improved understanding of freelance work, the role and potential for freelancing platforms in the knowledge-based gig economy, and directions for worker-centred design.

Manual schedule creation often involves satisfying numerous unique and conflicting constraints, which becomes more cognitively demanding when creating a common academic schedule with other individuals. Poor decision making caused by cognitive overload can result in unsuitable schedules. This study proposes the use of Boolean satisfiability (SAT) solvers in an academic scheduling system to help students balance scheduling preferences and satisfy necessary constraints. Based on the availability of courses and the scheduling preferences of users, the system automatically resolves conflicts and presents possible schedules. In a controlled experiment with 42 undergraduate students, cognitive demand was reduced by eliminating menial decisions, which significantly optimized the creation of a common schedule among peers. We found that human errors and emotional stress were diminished, and schedules created using the system were more satisfactory to participants. Finally, we present recommendations and design implications for future academic scheduling systems.

With growing interest regarding cooperation support using Augmented Reality (AR), social presence has become a popular measure of its quality. While this concept is established throughout cooperation research, its role in AR is still unclear: Some work uses social presence as an indicator for support quality, while others found no impact at all. To clarify this role, we conducted a literature review of recent publications that empirically investigated social presence in cooperative AR. After a thorough selection procedure, we analyzed 19 publications according to factors influencing social presence and the impact of social presence on cooperation support. We found that certain interventions support social presence better than others, that social presence has an influence on user's preferences and that the relation between social presence and cooperation quality may depend on the symmetry of the cooperation task. This contributes to existing research by clarifying the role of social presence for cooperative AR and deriving corresponding design recommendations.

Performers acutely need but lack tools to remotely rehearse and create live theatre, particularly due to global restrictions on social interactions during the Covid-19 pandemic. No studies, however, have heretofore examined how remote video-collaboration affects performance. This paper presents the findings of a field study with 16 domain experts over six weeks investigating how tele-immersion affects the rehearsal and performance of improvisational theatre. To conduct the study, an original media server was developed for co-locating remote performers into shared virtual 3D environments which were accessed through popular video conferencing software. The results of this qualitative study indicate that tele-immersive environments uniquely provide performers with a strong sense of co- presence, feelings of physical connection, and an increased ability to enter the social-flow states required for improvisational theatre. Based on our observations, we put forward design recommendations for video collaboration tools tailored to the unique demands of live performance.

Immersive Analytics is a quickly evolving field that unites several areas such as visualisation, immersive environments, and human-computer interaction to support human data analysis with emerging technologies. This research has thrived over the past years with multiple workshops, seminars, and a growing body of publications, spanning several conferences. Given the rapid advancement of interaction technologies and novel application domains, this paper aims toward a broader research agenda to enable widespread adoption. We present 17 key research challenges developed over multiple sessions by a diverse group of 24 international experts, initiated from a virtual scientific workshop at ACM CHI 2020. These challenges aim to coordinate future work by providing a systematic roadmap of current directions and impending hurdles to facilitate productive and effective applications for Immersive Analytics.

Geographic data visualisation on virtual globes is intuitive and widespread, but has not been thoroughly investigated. We explore two main design factors for quantitative data visualisation on virtual globes: i) commonly used primitives (2D bar, 3D bar, circle) and ii) the orientation of these primitives (tangential, normal, billboarded). We evaluate five distinctive visualisation idioms in a user study with 50 participants. The results show that aligning primitives tangentially on the globe’s surface decreases the accuracy of area-proportional circle visualisations, while the orientation does not have a significant effect on the accuracy of length-proportional bar visualisations. We also find that tangential primitives induce higher perceived mental load than other orientations. Guided by these results we design a novel globe visualisation idiom, Geoburst, that combines a virtual globe and a radial bar chart. A preliminary evaluation reports potential benefits and drawbacks of the Geoburst visualisation.

We present an observational study to compare co-located and situated real-time visualizations in basketball free-throw training. Our goal is to understand the advantages and concerns of applying immersive visualization to real-world skill-based sports training and to provide insights for designing AR sports training systems. We design both a situated 3D visualization on a head-mounted display and a 2D visualization on a co-located display to provide immediate visual feedback on a player’s shot performance. Using a within-subject study design with experienced basketball shooters, we characterize user goals, report on qualitative training experiences, and compare the quantitative training results. Our results show that real-time visual feedback helps athletes refine subsequent shots. Shooters in our study achieve greater angle consistency with our visual feedback. Furthermore, AR visualization promotes an increased focus on body form in athletes. Finally, we present suggestions for the design of future sports AR studies.

Most mobile health apps employ data visualization to help people view their health and activity data, but these apps provide limited support for visual data exploration. Furthermore, despite its huge potential benefits, mobile visualization research in the personal data context is sparse. This work aims to empower people to easily navigate and compare their personal health data on smartphones by enabling flexible time manipulation with speech. We designed and developed Data@Hand, a mobile app that leverages the synergy of two complementary modalities: speech and touch. Through an exploratory study with 13 long-term Fitbit users, we examined how multimodal interaction helps participants explore their own health data. Participants successfully adopted multimodal interaction (i.e., speech and touch) for convenient and fluid data exploration. Based on the quantitative and qualitative findings, we discuss design implications and opportunities with multimodal interaction for better supporting visual data exploration on mobile devices.

Data physicalisations afford people the ability to directly interact with data using their hands, potentially achieving a more comprehensive understanding of a dataset. Due to their complex nature, the representation of graphs and networks could benefit from physicalisation, bringing the dataset from the digital world into the physical one. However, no empirical work exists investigating the effects physicalisations have upon comprehension as they relate to graph representations. In this work, we present initial design considerations for graph physicalisations, as well as an empirical study investigating differences in comprehension between virtual and physical representations. We found that participants perceived themselves as being more accurate via touch and sight (visual-haptic) than the graphical-only modality, and perceived a triangle count task as less difficult in visual-haptic than in the graphical-only modality. Additionally, we found that participants significantly preferred interacting with visual-haptic over other conditions, despite no significant effect on task time or error.

Natural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language. We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and (2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization.

We explore network visualisation on a two-dimensional torus topology that continuously wraps when the viewport is panned. That is, links may be “wrapped” across the boundary, allowing additional spreading of node positions to reduce visual clutter. Recent work has investigated such pannable wrapped visualisations, finding them not worse than unwrapped drawings for small networks for path-following tasks. However, they did not evaluate larger networks nor did they consider whether torus-based layout might also better display high-level network structure like clusters. We offer two algorithms for improving toroidal layout that is completely autonomous and automatic panning of the viewport to minimiswe wrapping links. The resulting layouts afford fewer crossings, less stress, and greater cluster separation. In a study of 32 participants comparing performance in cluster understanding tasks, we find that toroidal visualisation offers significant benefits over standard unwrapped visualisation in terms of improvement in error by 62.7% and time by 32.3%.

Sketchnoting is a form of visual note taking where people listen to, synthesize, and visualize ideas from a talk or other event using a combination of pictures, diagrams, and text. Little is known about the design space of this kind of visual note taking. With an eye towards informing the implementation of digital equivalents of sketchnoting, inking, and note taking, we introduce a classification of sketchnote styles and techniques, with a qualitative analysis of 103 sketchnotes, and situated in context with six semi-structured follow up interviews. Our findings distill core sketchnote components (content, layout, structuring elements, and visual styling) and dimensions of the sketchnote design space, classifying levels of conciseness, illustration, structure, personification, cohesion, and craftsmanship. We unpack strategies to address particular note taking challenges, for example dealing with constraints of live drawings, and discuss relevance for future digital inking tools, such as recomposition, styling, and design suggestions.

Plots and tables are commonplace in today’s data-driven world, and much research has been done on how to make these figures easy to read and understand. Often times, however, the information they contain conveys only the end result of a complex and subtle data analysis pipeline. This can leave the reader struggling to understand what steps were taken to arrive at a figure, and what implications this has for the underlying results. In this paper, we introduce datamations, which are animations designed to explain the steps that led to a given plot or table. We present the motivation and concept behind datamations, discuss how to programmatically generate them, and provide the results of two large-scale randomized experiments investigating how datamations affect people’s abilities to understand potentially puzzling results compared to seeing only final plots and tables containing those results.

We present Marvis, a conceptual framework that combines mobile devices and head-mounted Augmented Reality (AR) for visual data analysis. We propose novel concepts and techniques addressing visualization-specific challenges. By showing additional 2D and 3D information around and above displays, we extend their limited screen space. AR views between displays as well as linking and brushing are also supported, making relationships between separated visualizations plausible. We introduce the design process and rationale for our techniques. To validate Marvis’ concepts and show their versatility and widespread applicability, we describe six implemented example use cases. Finally, we discuss insights from expert hands-on reviews. As a result, we contribute to a better understanding of how the combination of one or more mobile devices with AR can benefit visual data analysis. By exploring this new type of visualization environment, we hope to provide a foundation and inspiration for future mobile data visualizations.

Recent research in the area of immersive analytics demonstrated the utility of head-mounted augmented reality devices for visual data analysis. However, it can be challenging to use the by default supported mid-air gestures to interact with visualizations in augmented reality (e.g. due to limited precision). Touch-based interaction (e.g. via mobile devices) can compensate for these drawbacks, but is limited to two-dimensional input. In this work we present STREAM: Spatially-aware Tablets combined with Augmented Reality Head-Mounted Displays for the multimodal interaction with 3D visualizations. We developed a novel eyes-free interaction concept for the seamless transition between the tablet and the augmented reality environment. A user study reveals that participants appreciated the novel interaction concept, indicating the potential for spatially-aware tablets in augmented reality. Based on our findings, we provide design insights to foster the application of spatially-aware touch devices in augmented reality and research implications indicating areas that need further investigation.

In this paper, we present MIRIA, a Mixed Reality Interaction Analysis toolkit designed to support the in-situ visual analysis of user interaction in mixed reality and multi-display environments. So far, there are few options to effectively explore and analyze interaction patterns in such novel computing systems. With MIRIA, we address this gap by supporting the analysis of user movement, spatial interaction, and event data by multiple, co-located users directly in the original environment. Based on our own experiences and an analysis of the typical data, tasks, and visualizations used in existing approaches, we identify requirements for our system. We report on the design and prototypical implementation of MIRIA, which is informed by these requirements and offers various visualizations such as 3D movement trajectories, position heatmaps, and scatterplots. To demonstrate the value of MIRIA for real-world analysis tasks, we conducted expert feedback sessions using several use cases with authentic study data.

Composite data physicalizations allow for the physical reconfiguration of data points, creating new opportunities for interaction and engagement. However, there is a lack of understanding of people’s strategies and behaviors when directly manipulating physical data objects. In this paper, we systematically characterize different reconfiguration strategies using six exemplar physicalizations. We asked 20 participants to reorganize these exemplars with two levels of restriction: changing a single data object versus changing multiple data objects. Our findings show that there were two main reconfiguration strategies used: changes in proximity and changes in atomic orientation. We further characterize these using concrete examples of participant actions in relation to the structure of the physicalizations. We contribute an overview of reconfiguration strategies, which informs the design of future manually reconfigurable and dynamic composite physicalizations.

Virtual pets are an alternative to real pets, providing a substitute for people with allergies or preparing people for adopting a real pet. Recent advancements in mixed reality pave the way for virtual pets to provide a more natural and seamless experience for users. However, one key challenge is embedding environmental awareness into the virtual pet (e.g., identifying the food bowl’s location) so that they can behave naturally in the real world.

We propose a novel approach to synthesize virtual pet behaviors by considering scene semantics, enabling a virtual pet to behave naturally in mixed reality. Given a scene captured from the real world, our approach synthesizes a sequence of pet behaviors (e.g., resting after eating). Then, we assign each behavior in the sequence to a location in the real scene. We conducted user studies to evaluate our approach, which showed the efficacy of our approach in synthesizing natural virtual pet behaviors.

A common task for scatterplots is communicating trends in bivariate data. However, the ability of people to visually estimate these trends is under-explored, especially when the data violate assumptions required for common statistical models, or visual trend estimates are in conflict with statistical ones. In such cases, designers may need to intervene and de-bias these estimations, or otherwise inform viewers about differences between statistical and visual trend estimations. We propose data-driven mark orientation as a solution in such cases, where the directionality of marks in the scatterplot guide participants when visual estimation is otherwise unclear or ambiguous. Through a set of laboratory studies, we investigate trend estimation across a variety of data distributions and mark directionalities, and find that data-driven mark orientation can help resolve ambiguities in visual trend estimates.

We investigated the experiences of 15 parents and their tween children (ages 8-12, n=23) during nature explorations using the NatureCollections app, a mobile application that connects children with nature. Drawing on parent interviews and in-app audio recordings from a 2-week deployment study, we found that tweens’ experiences with the NatureCollections app were influenced by tensions surrounding how parents and tweens negotiate technology use more broadly. Despite these tensions, the app succeeded in engaging tweens in outdoor nature explorations, and parents valued the shared family experiences around nature. Parents desired the app to support family bonding and inform them about how their tween used the app. This work shows how applications intended to support enriching youth experiences are experienced in the context of screen time tensions between parents and tween during a transitional period of child development. We offer recommendations for designing digital experiences to support family needs and reduce screen time tensions.

Parental mediation literature is mostly situated in the contexts of television, Internet use, video games, and mobile devices, while there is less understanding of how parents mediate their children’s engagement with educational-focused media. We examine parental involvement in young children’s use of a creation-oriented educational media, i.e., coding kits, from a mediation perspective through an interview study. We frame parents’ mediation practices along three dimensions: (1) creative mediation, where parents mediate to support children’s creating and learning with media; (2) preparative mediation, where parents explore and prepare media for children’s engagement; and (3) administrative mediation, where parents administer and regulate their children’s media use. Compared to the restrictive, active, and co-using mediation theory, our proposed framework highlights various supportive practices parents take to help their children learn and create with media. We further connect our findings to Joint Media Engagement and reflect on implications for parent involvement in children’s creation-oriented media design.

Novice tangible interaction design students often find it challenging to generate input action ideas for tangible interfaces. To identify opportunities to aid input action idea generation, we built and evaluated a tool consisting of interactive physical artifacts coupled with digital examples of tangible systems and technical implementation guidance. Through video recorded design sessions and interviews with twelve students, we investigated how they used the tool to generate input action ideas, how it supported them, and what challenges they faced. We found that the tool helped in generating input action ideas by enabling to experience input actions, supporting hands-on explorations, and introducing possibilities. However, introducing examples at times caused design fixation. The tool fell short in supporting the planning of technical implementation of the generated ideas. This research is useful for tangible interaction design students, instructors, and researchers to apply in education, design similar tools, or conduct further research.

Distance learning is facing a critical moment finding a balance between high quality education for remote students and engaging them in hands-on learning. This is particularly relevant for project-based classrooms and makerspaces, which typically require extensive trouble-shooting and example demonstrations from instructors. We present RobotAR, a teleconsulting robotics toolkit for creating Augmented Reality (AR) makerspaces. We present the hardware and software for an AR-compatible robot, which behaves as a student’s voice assistant and can be embodied by the instructor for teleconsultation. As a desktop-based teleconsulting agent, the instructor has control of the robot’s joints and position to better focus on areas of interest inside the workspace. Similarly, the instructor has access to the student’s virtual environment and the capability to create AR content to aid the student with problem-solving. We also performed a user study which compares current techniques for distance hands-on learning and an implementation of our toolkit.

Grandparents and grandchildren, who cannot meet face-to-face (e.g., due to dislocation or physical distancing induced by a pandemic), often use audio-visual communication tools in order to maintain their relationship online. In a qualitative online survey (n = 85), we inquired into the various ways that grandparents and grandchildren came up with when being physically distant; many of them are tangible in nature as they include “things” or incorporate “spaces”. In this paper, we illustrate related temporal and spatial trajectories and unpack how online meetings are characterized by constant negotiations of agency. We discuss how online meetings could complement face-to-face meetings, instead of mimicking or replacing them. We finally articulate a collection of design sensitivities with the aim to both inspire and question designing for intergenerational online meetings.

Aging often comes with declining social interaction, a known adversarial factor impacting the life satisfaction of senior population. Such decline appears even in family–a permanent social circle, as their adult children eventually go independent. We present MomentMeld, an AI-powered, cloud-backed mobile application that blends with everyday routine and naturally encourages rich and frequent inter-generational interactions in a family, especially those between the senior generation and their adult children. Firstly, we design a photographic interaction aid called mutually stimulatory memento, which is a cross-generational juxtaposition of semantically related photos to bring natural arousal of context-specific inter-generational empathy and reminiscence. Secondly, we build comprehensive ensemble AI models consisting of various deep neural networks and a runtime system that automates the creation of mutually stimulatory memento on top of the user’s usual photo-taking routines. We deploy MomentMeld in-the-wild with six families for an eight-week period, and discuss the key findings and further implications.

Children are increasingly using wearables with physical activity tracking features. Although research has designed and evaluated novel features for supporting parent-child collaboration with these wearables, less is known about how families naturally adopt and use these technologies in their everyday life. We conducted interviews with 17 families who have naturally adopted child-owned wearables to understand how they use wearables individually and collaboratively. Parents are primarily motivated to use child-owned wearables for children's long-term health and wellbeing, whereas children mostly seek out entertainment and feeling accomplished through reaching goals. Children are often unable to interpret or contextualize the measures that wearables record, while parents do not regularly track these measures and focus on deviations from their children's routines. We discuss opportunities for making naturally-occurring family moments educational to positively contribute to children's conceptual understanding of health, such as developing age-appropriate trackable metrics for shared goal-setting and data reflection.

Physical space and materials we work with, as well as people we interact with affect the design and making processes. These aspects in relation to 3D designing in alignment with 3D printing need considerable exploration within Child-Computer interaction (CCI) research community. We conducted our case study by collecting 9-full-day-observation and interview data and examining 3D modeling and 3D printing activities, as work duties of 15-17-years-old summer trainees organized at the university. We identified, inspired by nexus analysis, different discourses circulating around these activities of novice young people and how the discourses are intermingled with the space, the materials, and the task at hand in complex ways, constructing and shaping the experience of the young participants. In our research and design implications, by signifying the impact of the people, challenges, tasks, spaces and tools, we provide recommendations for maintaining children's engagement in digital fabrication, significantly 3D designing and 3D printing, activities.

Pedagogical agents are theorized to increase humans’ effort to understand computerized instructions. Despite the pedagogical promises of VR, the usefulness of pedagogical agents in VR remains uncertain. Based on this gap, and inspired by global efforts to advance remote learning during the COVID-19 pandemic, we conducted an educational VR study in-the-wild (N = 161). With a 2 × 2 + 1 between subjects design, we manipulated the appearance and behavior of a virtual museum guide in an exhibition about viruses. Factual and conceptual learning outcomes as well as subjective learning experience measures were collected. In general, participants reported high enjoyment and had significant knowledge acquisition. We found that the agent’s appearance and behavior impacted factual knowledge gain. We also report an interaction effect between behavioral and visual realism for conceptual knowledge gain. Our findings nuance classical multimedia learning theories and provide directions for employing agents in immersive learning environments.

With rapid developments in consumer-level head-mounted displays and computer graphics, immersive VR has the potential to take online and remote learning closer to real-world settings. However, the effects of such digital transformations on learners, particularly for VR, have not been evaluated in depth. This work investigates the interaction-related effects of sitting positions of learners, visualization styles of peer-learners and teachers, and hand-raising behaviors of virtual peer-learners on learners in an immersive VR classroom, using eye tracking data. Our results indicate that learners sitting in the back of the virtual classroom may have difficulties extracting information. Additionally, we find indications that learners engage with lectures more efficiently if virtual avatars are visualized with realistic styles. Lastly, we find different eye movement behaviors towards different performance levels of virtual peer-learners, which should be investigated further. Our findings present an important baseline for design decisions for VR classrooms.

Classroom sensing is an important and active area of research with great potential to improve instruction. Complementing professional observers – the current best practice – automated pedagogical professional development systems can attend every class and capture fine-grained details of all occupants. One particularly valuable facet to capture is class gaze behavior. For students, certain gaze patterns have been shown to correlate with interest in the material, while for instructors, student-centered gaze patterns have been shown to increase approachability and immediacy. Unfortunately, prior classroom gaze-sensing systems have limited accuracy and often require specialized external or worn sensors. In this work, we developed a new computer-vision-driven system that powers a 3D “digital twin” of the classroom and enables whole-class, 6DOF head gaze vector estimation without instrumenting any of the occupants. We describe our open source implementation, and results from both controlled studies and real-world classroom deployments.

This project aims to foster shared positive experiences between people living with moderate to advanced dementia and their visitors as they may struggle to find topics to talk about and engaging things to do together. To promote a better visit, we trialed a previously developed app that includes eight games with twenty-one residents and their partners or carers across four care centers for three months each. Through interviews and data logging, we found that residents preferred games that were closer to their interests and skills, and that gameplay and cooperation fostered meaningful and shared interactions between residents and their visitors. The contribution of this work is twofold: (1) insights and opportunities into dyadic interactions when using an app and into promoting positive social experiences through technology design, and (2) reflections on the challenges of evaluating the benefits of technology for people living with dementia.

In recent work, design researchers have sought to ensure that people with disabilities are engaged as competent and valued contributors to co-design. Yet, little is known about how to achieve this with adults with severe intellectual disabilities. Navigating design in the context of complex care practices is challenging, charged with uncertainty, and requires sustained effort of methodological and affective adjustments. To establish a respectful co-design relationship and enrich participation, we turn to Active Support (AS), an evidence-based strategy for engaging adults with severe intellectual disabilities. We present a reflective account of long-term field work that utilized the four aspects of AS, a) every moment has potential; b) graded assistance; c) little and often; d) maximizing choice and control. We discuss how these principles contribute to deepening HCI methods by ensuring interactional turns for adults with severe disabilities, revealing their unique competences, thereby shaping design direction and providing design insight.

Many efforts to increase accessibility in coding for developers with visual impairments (DWVI) focus on supporting interactions with development tools. But, to understand how to appropriately modify and write source code, developers must seek information from a variety of disparate and highly technical sources. DWVI might benefit from technological support in this process. But, it is unclear what accessibility issues arise in technical information sources, whether accessibility impacts strategies for seeking technical information, or how best to support DWVI in information seeking. We conducted observations and interviews with twelve DWVI, to explore their information behaviors. We found that DWVI seek information in many of the same sources as their sighted peers, and accessibility issues in technical information sources were similar to those in nontechnical sources. But, despite these similarities, examining development as an information seeking process highlighted the role of contextual and social factors in determining accessibility for DWVI.

Knitting is a popular craft that can be used to create customized fabric objects such as household items, clothing and toys. Additionally, many knitters find knitting to be a relaxing and calming exercise. Little is known about how disabled knitters use and benefit from knitting, and what accessibility solutions and challenges they create and encounter. We conducted interviews with 16 experienced, disabled knitters and analyzed 20 threads from six forums that discussed accessible knitting to identify how and why disabled knitters knit, and what accessibility concerns remain. We additionally conducted an iterative design case study developing knitting tools for a knitter who found existing solutions insufficient. Our innovations improved the range of stitches she could produce. We conclude by arguing for the importance of improving tools for both pattern generation and modification as well as adaptations or modifications to existing tools such as looms to make it easier to track progress

Command-line interfaces (CLIs) remain a popular tool among developers and system administrators. Since CLIs are text-based interfaces, they are sometimes considered accessible alternatives to predominantly visual developer tools like IDEs. However, there is no systematic evaluation of the accessibility of CLIs in the literature. In this paper, we describe two studies with 12 developers on their experience of using CLIs with screen readers. Our findings show that CLIs have their own set of accessibility issues - the most important being CLIs are unstructured text interfaces. Based on our results, we provide a set of recommendations for improving the accessibility of command-line interfaces.

Technology adoption among older adults has increased significantly in recent years. Yet, as new technologies proliferate and the demographics of aging shift, continued attention to older adults’ adoption priorities and learning preferences is required. Through semi-structured interviews, we examine the factors adults 65+ prioritize in choosing new technologies, the challenges they encounter in learning to use them, and the human and material resources they employ to support these efforts. Using a video prototype as a design probe, we present scenarios to explore older adults’ perceptions of adoption and learning new technologies within the lens of health management support, a relevant and beneficial context for older adults. Our results reveal that participants appreciated self-paced learning, remote support, and flexible learning methods, and were less reliant on instruction manuals than in the past. This work provides insight into older adults’ evolving challenges, learning needs, and design opportunities for next generation learning support.

While HCI researchers have begun designing personalised VR experiences for older adults, there has been limited research examining the use of social VR - where users interact via avatars in a virtual environment. Avatar-mediated communication (AMC) is a crucial component of the social VR experience, but older users’ experience with AMC is poorly understood. We conducted a five-month study with 16 older adults evaluating a co-designed social VR prototype. Results show that AMC in social VR was seen as medium that supported introverted users to express themselves and was viewed as offering advantages when discussing sensitive topics. Our study provides new insights into how older adults view AMC in social VR as a communication medium and we contribute six design reflections, based on our results, that highlight the steps that can be taken to ensure that AMC in social VR can meet the communication needs of older users.

There is an open call for technology to be more playful [5, 79] and for tech design to be more inclusive of people with disabilities [80]. In the era of COVID19, it is often unsafe for the public in general and people with disabilities, in particular, to engage in in-person design exercises using traditional methods. This presents a missed opportunity as these populations are already sharing playful content rich with tacit design knowledge that can be used to inspire the design of playful everyday technology. We present our process of scraping play potentials [4] from TikTok from content creators with disabilities to generate design concepts that may inspire future technology design. We share 7 emerging themes from the scraped content, a catalog of design concepts that may inspire designers, and discuss the relevance of the emerging themes and possible implications for the design concepts.

High-fidelity prototyping tools are used by software designers and developers to iron