UIST '19- The Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology

While eye tracking has been typically used for achieving intuitive user interfaces, it is not sufficient when it comes to dealing with multiple-display environments. In such environments, which have become popular recently, the point-of-gaze should be estimated on multiple screens. Therefore, we propose a cross-ratio based gaze estimation using a polarization camera for multiple screens. The point-of-gaze can be estimated on each monitor by identifying the screen reflected on the corneal surface at a polarization angle. Near-infrared light emitting diodes (NIR-LEDs) attached to the display are not required. This means that standard displays can be used with high general versatility as an advantage.

Reach redirection in virtual reality uses spatial distortion to augment interaction with passive props as well as active haptic devices. For such dynamic physical systems, motion modeling is needed to update the interface based on users' predicted targets & trajectories. However, it remains unclear how well standard predictive models hold under redirection. In this work we evaluate one such commonly used model, the Minimum-Jerk (MJ) model, during redirected reach at various lateral offsets up to 16cm. Results show that larger redirection significantly worsens MJ model fit, suggesting that models should be adjusted for reaches with considerable redirection. Predicted arrival times, based on fitting an MJ model on the first half of reach data, led to an average error of -0.29s for redirected reach, compared to -0.03s for normal reach.

In an effort to find a way to more efficiently and accurately display localized high definition tactile information to the skin we studied the vibrotactile constructive wave interference properties of silicone gel. By placing two actuators at a given distance from each other and controlling the delay when each actuator is activated in relation to the other, we can achieve a point of constructive wave interference. The time when the interference occurs along with any loses due to attenuation depends on the material the vibration travels through. The goal is to find a compliant material that can be used as a reference to human tissue when designing feedback for assistive robots.

In this paper, we present a novel methodology for simulating the physical and cognitive demands that individuals experience when waking from sleep. Better understanding this scenario has significant implications for research in Autonomous Vehicles (AV), where prior research has shown that many drivers would like to sleep while the vehicle is in operation. Our experiment setup replicates the waking situation in two ways: (1) Subjects wear a sleep shade (physical demand) for 3 sessions (5min, 8min, and 11min) in randomly assigned order, after which (2) they view a screen (cognitive demand) that fades from blurry to clear over a 10s-timeframe. We compared subjects' experiences in-study to the physical and cognitive conditions they experience when waking in real life. Our experiment setup was highly rated in effectiveness and appropriateness for alternating sleeping situation. Findings will be utilized as scenario design in future AV studies and can be adopted in other fields, as well.

The use of the minimalistic design in the document editing UI based on novel handwriting recognition technologies for mobile devices with a touch screen is very limited in real-world applications, mainly due to the propagation of errors made by recognition engines for different types of input. It is caused by the iterative nature of recognition for diverse input. Currently, one of the common ways is to pre-configure the input type manually for different types of documents' content that leads to an increase in UI complexity. Bearing this in mind, we revisit UI design issues regarding the unification of recognition flow by introducing the document layout analyzer, to avoid switching between modes during input of handwritten documents. The system for the iterative input of diverse document by using handwritten strokes recognition techniques was presented. Using an evaluation study, we have shown that the proposed approach provides a more interactive experience for the user and increases the input speed by 1.5--2 times compared to the classical approaches.

Autonomous systems (e.g., long-distance driverless trucks) aim to reduce the need for people to complete tedious tasks. In many domains, automation is challenging because systems may fail to recognize or comprehend all relevant aspects of its current state. When an unknown or uncertain state is encountered in a mission-critical setting, recovery often requires human intervention or hand-off. However, human intervention is associated with decision (and communication, if remote) delays that prevent recovery in low-latency settings. Instantaneous crowdsourcing approaches that leverage predictive techniques reduce this latency by preparing human responses for possible near future states before they occur. Unfortunately, the number of possible future states can be vast and considering all of them is intractable in all but the simplest of settings. Instead, to reduce the number of states that must later be explored, we propose the approach that uses the crowd to first predict the most relevant or likely future states. We examine the latency and accuracy of crowd workers in a simple future state prediction task, and find that more than half of crowd workers were able to provide accurate answers within one second. Our results show that crowd predictions can filter out critical future states in tasks where decisions are required in less than three seconds.

We introduce an interactive system to design 2D data sets on a scatter plot. 2D data sets are often used to obtain an intuition about the behavior of machine learning techniques or statistical modeling techniques . A traditional method for creating a data set is to use textural commands, such as generating a normal distribution with specific parameters. However, it is difficult to create an arbitrary data distribution from scratch or to edit an existing one using these methods. We therefore developed an interactive system for creating and editing a 2D data distribution on a scatter plot by using a graphics user interface. It also allows the user to observe the response of a data analysis method while the user is editing a distribution. We show that the proposed interface is effective for understanding characteristics of data analysis methods.

User calibration is a significant problem in eye-based interaction. To overcome this, several solutions, such as the calibration-free method and implicit user calibration, have been proposed. Pursuits-based interaction is another such solution that has been studied for public screens and virtual reality. It has been applied to select graphical user interfaces (GUIs) because the movements in a GUI can be designed in advance. Smooth pursuit eye movements (smooth pursuits) occur when a user looks at objects in the physical space as well and thus, we propose a method to identify the focused object by using smooth pursuits in the real world. We attempted to determine the focused objects without prior information under several conditions by using the pursuits-based approach and confirmed the feasibility and limitations of the proposed method through experimental evaluations.

We identify different types of failures in a voice user interface application, from both developer and user perspectives. Our preliminary experiment suggests that user-perceived Pattern Match Failure may have a strong negative effect on user frustration; based on this result, we conduct power analysis to obtain more conclusive results in a future experiment.

Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.

This paper introduces LiftTiles, a modular and reconfigurable room-scale shape display. LiftTiles consist of an array of retractable and inflatable actuator that is compact (e.g., 15cm tall) and light (e.g., 1.8kg), while extending up to 1.5m to allow for large-scale shape transformation. Inflatable actuation also provides a robust structure that can support heavy objects (e.g., 10 kg weight). This paper describes the design and implementation of LiftTiles and explores the application space for reconfigurable room-scale shape displays.

Despite promising initial studies, a speaker's original voice can cause problems when it comes to the application of real-time voice conversion (data-driven speaker conversion) technology in our daily lives, specifically in our near-field communication, because the overlapping speech degrades the sense of immersion to the converted speech. We present TransVoice, a real-time voice conversion system that physically confines original speech with a mask-shaped device. Our preliminary study shows the proposed device can reduce the volume of original speech significantly, while it ameliorates the deteriorated conversion quality of the deep neural network (DNN) thanks to an integrated filter that weakens the low frequency range. We discuss novel applications using TransVoice that can augment our communication.

Pneu-Multi-Tools is a novel passive haptic feedback (PHF) wearable device that let users grip different shapes of virtual props in virtual reality(VR) with sensing the shape changes of foldable airbags driven by pneumatics. The solution to the limitation of primitive shapes in haptic interfaces for VR in the past is proposed in this research. TPU films can be manufactured into 4 kinds of folding shapes(clip, rectangle, cylinder and cone shape) owing to the numbers and orientation of folding hinges on a single airbag. Therefore, Pneu-Multi-Tools, which is stacked with different folding shapes of airbags and capable of automatically folding, enables users to use multi-props intuitively in VR games. There are 3 interaction scenarios are provided by this interface: "Pick-Up", "Order" and "Hot Key" in multi-props games to make it possible for users to switch the props more efficiently.

While mobile technologies are moving closer to our body and novel wearable gadgets and smart textile interfaces emerge, current approaches are often expensive individual solutions for specific applications and lack reconfiguration possibilities. With this work, we introduce BodyHub, a modular wearable approach that allows users to realize their own smart garment applications by arranging and configuring exchangeable functional modules. To address individual user requirements and preferences, we developed a comprehensive repertoire of input and output modules that can be placed freely onto slide-in sockets which are imprinted in the textile by using 3D printing. Further, we developed a smartphone companion app that facilitates the creation of user-defined system functions without any programming skills. BodyHub thereby allows the creation of personalized wearable solutions by the users themselves and also supports ad-hoc assemblies for interface design explorations in research labs. To demonstrate the range of possible applications, we describe real-world use-cases from the areas of work life, shopping, mobility, and gaming.

The rapid advancement of augmented reality technology brings museum visitors enhanced interaction and immersion experience. However, most existing augmented reality museums adopt individual-based user interfaces that hinder joint interaction across multiple users. We present the distance-driven user interface (DUI) to enable collaborative exhibit viewing in augmented reality museum. We classify the users in four groups according to the user-exhibit distance and assign each group specific interaction privileges - put simply, the DUI elicits the users standing far from an exhibit to explore more of the desired exhibit by approaching closer to the exhibit. We describe the DUI architecture and preliminarily evaluate users' acceptance and effectiveness of the DUI and find that, the DUI is interpretable, improves users' awareness of collaboration, and increases user interests to the exhibit with considerably improved willingness of approaching the exhibit.

Reconstructing a previous working context (files, websites and software) when resuming a task is inevitable for computer users after switching software or being interrupted. The process of retrieving relevant applications, files, and web pages can be time-consuming; while users may be able to use a "recent documents" feature to resume work in some applications, it is challenging to find relevant information from a text-based history --- file names, web page titles, or URLs may not be informative enough. To address this problem, we tested the idea of using a visual history of a computer screen through the development of ScreenTrack. ScreenTrack is software that captures an image of a computer screen at regular intervals. It lets a user watch and navigate a time-lapse video made of computer screenshots to retrieve a previous working context from a screenshot. Through a controlled user study, it was found that participants were able to retrieve requested information more quickly with ScreenTrack than the control condition; this difference was statistically significant. Additionally, participants gave positive feedback on possible future uses for this software. We present the software, our preliminary user study result, and a plan for further validation.

Recently, adopting a hands-on approach to conventional 3D fabrication has been attracting attention due to its advantages in design activity. In this context, we aim to support hands-on design activity in digital fabrication by designing internal structures for alleviating issues of external heating for shape deformation. As a first step, we simulate four simple structures with Computational Fluid Dynamic (CFD) simulation to investigate effective structural parameters such as cavity's ratio, its geometry, exposure to the heat source for influencing thermal properties, and deformation in a malleable state. Through the pilot experiment, we figured out that the simulation results of the basic structures are valid, the structure is stable in a malleable state, and the parameters are effective. In the future, we will design functional structures based on the explored parameters and embed them on various topologies.

Video editing is a difficult task for both professionals and amateur editors. One of the biggest reasons is that screening useful clips from raw material in the early stage of editing is too much time-consuming and laborious. To better understand current difficulties faced by users in editing task, we first conduct a pilot study involving a survey and an interview among 20 participants. Based on the results, we then design Pre-screen, a novel tool to provide users with both global-view and detailed-view video analysis as well as material screening features based on intelligent video processing, analysis and visualization methods. User study shows that Pre-screen can not only effectively help users screen and arrange raw video material to save much more time than a widely used video editing tool in video editing tasks, but also inspire and satisfy users.

In Massive Open Online Courses (MOOCs), learners face a lot of distractions which will cause divided attention (DA). However, it is not easy for learners to realize that they are distracted and to find out which part of the course they have missed. In this paper, we present Reminder, a system for detecting divided attention and reminding learners what they just missed on both PC and mobile devices with a camera capturing their status. To get learners' attention level, we build a regression model to predict attention score from an integrated feature vector. Meanwhile, we design an interactively updating method to make the model adaptive to a specific user. We also propose a visualization method to help learners review missed content easily. User study shows that Reminder detects learners' divided attention and assists them to review missed course contents effectively.

The traditional storytelling experiences are often one dimensional, wherein they only contain a single channel of communication with the audience through narration. With the advancements in technology, storytelling experiences have been augmented with the help of digital media to be more engaging and immersive. Authoring these scenarios, however, is complicated as it requires technical knowledge to interface the means of engagement. In this work, we talk about Aesop, a system which assists the narrator to author engaging storytelling experiences. Aesop provides a block-based interface like Scratch and manifests words of a story, Cues, and Visualization (Outputs) as blocks that enable the user to create captivating stories. Our system also leverages physical actions performed by the user as Cues. These cues can trigger visualizations like robot actions, animations, environments simulation using sound and lighting effects.

This paper describes a novel device called the "Stick User Interface", comprised of a robotic projector-camera system, an onboard digital computer connected to the Internet, sensors, and a hardware interface capable of sticking to any surface such as walls, ceilings, etc. The computer further consists of other subsystems, devices, and sensors such as an accelerometer, compass, gyroscope, flashlight, etc. The device unfolds (expands) on activation, detects a surface, and attaches itself to it. After a successful attachment, the device then projects or augments images, information, and user interfaces on nearby surfaces and walls. The user interface may contain applications, information about the object being augmented, and information from the Internet. Users can interact with the user-interface using commands and gestures such as hand, body, feet, voice, etc.

Augmented Reality (AR) is rapidly gaining popularity, enhancing human perception of the real world by augmenting digital experiences. Existing tools for authoring AR scenes are either template based or require domain knowledge from experts, and are therefore restrictive. ARComposer is a novel interface that enables easy authoring of AR experiences from free-form text describing the scene. Our proposed interface allows creators to compose varied scenes comprising of multiple objects with diverse relationships to each other as well as human models with animations, starting merely with a textual description. A qualitative evaluation shows that ARComposer provides a good flow experience to its users. Furthermore, a crowd-sourced experiment evaluating various aspects of the rendered AR scenes indicates the viability of the proposed approach.

A simple way to prototype touch interaction is to extend electrodes from a capacitive touch screen to off-screen areas. With that we aim to develop a toolkit that transforms a user-designed layout into a layout of screen-extension electrodes that realizes touch for rapid prototyping. Nevertheless, this kind of extension cannot detect touch if the physical properties of the electrodes become large. In this work, we decompose the physical design properties of the extension electrode into two factors, the target area, and line bridge, and investigate the limitation of each separately. While revealing some factors, such as area, is extremely limited in term of designing freely, we look into the causes by measuring the capacitive charge on-screen and on-extensions.

How firmly a haptic device, such as a smartwatch, is coupled to the body can change how its haptic effects are perceived. However, hapticians often rely on vague subjective coupling characteristics such as "strapped snugly" or "tight yet comfortable". Achieving consistent strap tightness across body sites and between participants can be challenging, since even if strap tension is consistent, differences in limb circumference alter the resulting normal force under the haptic actuator in potentially unintuitive ways. Furthermore, when participants must attach the devices on their own, e.g., during a longitudinal in-the-wild study, they may not use the same tightness each day without guidance. We present HaptiStrap, a low-cost, easily fabricated tool, as a contribution towards a standard method for ensuring that wearable haptic studies do better than vague and subjective "tight yet comfortable" guidelines.

In this work, a solution for handwriting text extraction from images with visual user assistance is proposed. Use of end-to-end systems that pipe together text detection and recognition is often awkward because the user cannot influence the detection stage. On the other hand, glossing over the word's regions to help system with text localization requires a manual job and can be unacceptable. This paper proposes a solution that gives visual cues to the user during a detection stage. These hints differ from traditional bounding boxes in two ways. Firstly, the found text is surrounded with polygonal bounding reflecting a possible complex nature of text blocks. Secondly, TextRadar scanning effect provides a non-overloaded camera view, helping the user to capture the most relevant part of the text on image on-the-fly. CalliScan works on-device and keeps the user's privacy. The evaluation study has shown that users need such a solution, but it is necessary to carefully handle the text layout complexity.

User interfaces in self-order terminals aim to satisfy the need for information of a broad audience and thus get easily clut-tered. Online shops present personalized product recommen-dations based on previously gathered user data to channel the user's attention. In contrast, stateless point-of-sales machines generally have no access to the user's personal information nor previous purchase behavior. User preferences must therefore be determined during the interaction. We thus propose using gaze data to determine preferences in real-time. In this paper we present a system for dynamic gaze-based fltering.

Fabricating and actuating inflatables for shape-changing interfaces and soft robotics is challenging and time-consuming, requiring knowledge in diverse domains such as pneumatics, manufacturing processes for elastomers, and embedded systems. We propose in this poster a scheme for rapid prototyping and pneumatically actuating piecewise multi-chambered inflatables, using balloons as our building blocks. We provide a construction kit containing pneumatic control boards, pneumatic components, and balloons for constructing simple actuated balloon models. We also provide various primitives of actuation and locomotion to help the user put together their desired actuator, along with an Android app and software API for controlling it via Bluetooth. Finally, we demonstrate the construction and actuation of these inflatable structures using three sample applications.

To guide the design of interactive image annotation systems that generalize to new domains and applications, we need ways to evaluate the capabilities of new annotation tools across a range of different types of image, content, and task domains. In this work, we introduce Corsica, a test harness for image an- notation tools that uses calibration images to evaluate a tool's capabilities on general image properties and task requirements. Corsica is comprised of sets of three key components: 1) synthesized images with visual elements that are not domain- specific, 2) target microtasks that connects the visual elements and tools for evaluation, and 3) ground truth data for each mi- crotask and visual element pair. By introducing a specification for calibration images and microtasks, we aim to create an evolving repository that allows the community to propose new evaluation challenges. Our work aims to help facilitate the robust verification of image annotation tools and techniques.

In this paper, we propose the novel concept of manifesting the Choice Blindness paradigm in real-life scenarios through immersive Virtual Reality (VR). We designed a VR application wherein the participants encountered two-alternative choices regarding implicit racial bias. Our observations show that 92% of subjects failed to notice a mismatch in their choices, while 75% exhibited choice blindness, hence indicating a healthy scope for exploring choice blindness through virtual reality.

A person with hearing impairments is difficult to notice greets and calling attention from their back due to low ability of listening, and it leads the decline of the ability of sensing danger and the wrong guess at beginning communication. Therefore, we propose the system conveying the information of sound from backwards through haptic feedback. Through the experiment verifying recognition rate of DoA estimation, it is confirmed that the system can substitute the ability of rear sound image localization. Through the experiment verifying recognition rate of type of sounds, it is also confirmed that the system can aware the sound from backwards.

We propose an auditory card game system towards playing equally with others, regardless of whether they have a visual impairment or not. To develop an audible card game system, we convert a Japanese traditional card game: Karuta into tablet devices which emit identificated sound signals, respectively. The proposed method makes it possible to boost the sense of victory and rivalry even without visually information.

This paper describes Ardestan, a visual programming language (VPL) for Arduino currently under development. The language is designed with art and design students in mind as novice programmers. Although multitasking and event scheduling are essential for interactive prototypes, novices often face difficulty in implementing these features. By borrowing the language design from Pure Data, a VPL for interactive music, Ardestan facilitates the implementation of multitasking and event scheduling, while generating C++ code for a standalone Arduino system. Such features would be beneficial to support prototyping activity by art and design students in undergraduate interaction design courses.

ReMap is a multimodal interface that enables searching for learning videos using speech and in-task pointing. ReMap extends multimodal interaction to help-seeking for complex tasks. Users can speak search queries, adding app-specific terms deictically. Users can navigate ReMap's search results via speech or mouse. These features allow people to stay focused on their task while simultaneously searching for and using help resources. Future work should explore how to implement more robust deictic resolution and more modalities.

We introduce a novel prototyping method using a metal leaf and laser printer. Proposed metal leaf circuit is capable of producing circuits on normal paper that can be printed by a laser printer. The metal leaf is adhered to paper using a toner used in a laser printer. When the metal leaf is placed on the printed circuit pattern and pressed with an iron, the metal leaf adheres to the circuit pattern. Removal of the excess metal leaf completes the metal leaf circuit diagram on the paper. In addition, by using the change of temperature of toner, we can easily cut and repair the circuit. We made a metal leaf circuit on the cloth with the masu sake cup, and evaluated it.

Optical See-Through Head-Mounted Displays (OST-HMDs) have attracted much attention as augmented reality (AR) devices. Depth of field (DoF) is a key parameter for OST-HMDs because a shallow DoF often yields focal point gaps between computer-generated graphics (CGs) and real scenes forcing users to re-adjust their focus. However, it is difficult to achieve a deep DoF for CGs over real scenes in compact and low-cost devices because laser projectors or complicated optics are needed. In this paper, we propose an OST-HMD that uses polarizing plates with a pinhole to achieve a deep DoF. The optics of the proposed device comprises two polarizing plates and a beam splitter. The polarization planes of the plates are orthogonal and one of the plates has a pinhole. The key idea of the proposal is the pinhole effect using polarizing plates to achieve a deep DoF without any influence to the field of view (FoV) for real scenes. This method can be implemented at significantly low cost and ease in compact thanks to its simple optics and is applicable to OST-HMDs that employ smartphones. In addition, we confirm that CGs are clearly seen whatever the focal point by constructing a prototype.

We propose a system that automatically generates a projection mapping onto natural objects such as leaves and flowers, for which it is currently difficult to apply tracking markers. Our PM allows user interactions such as contact and covering. We describe our results with some animated effects on various moving leaves and flowers.

Prior work on choreographic learning support systems lacks supports of group dancers. In particular, existing systems cannot quantify the degree of synchronization, which is a critical factor for successful group dancing. In this paper, we create a system to support multi-person choreographic learning. The system visualizes body parts of dancers which are out of synchronization. Our interface aims to enable dancers to quickly identify moments where they need additional practice. This paper presents our current prototype interface and demonstration.

We demonstrate a system to deliver dynamic guidance in drawing, sketching and handwriting tasks via an electromagnet moving underneath a high refresh rate pressure sensitive tablet presented in \citelangerak2019dynamic. The system allows the user to move the pen at their own pace and style and does not take away control. Using a closed-loop time-free approach allows for error-correcting behavior. The user will experience to be smoothly and natural pulled back to the desired trajectory rather than pushing or pulling the pen to a continuously advancing setpoint. The optimization of the setpoint with regard to the user is unique in our approach.

Conventional camera-based hand interaction technique suffered from self-occlusion among fingers, which lowers the detection accuracy of fingertip positions, leading to uncomfortable UI controls. Based on observations, self-occlusion depends on hand postures. We design an interaction framework in which interaction is decided in response to a recognized hand posture. Using a tabletop projection system that has a projector and a depth sensor, we implement the framework by integrating five touch and in-air interactions that balance its stability and usability.

Embedding conductive material into 3D printed objects enables non-interactive objects to become tangible without the need to attach additional components. We present a novel use for such touch-sensitive objects in an augmented reality (AR) setting and explore the use of gestures for enabling different types of interaction with digital and physical content. In our demonstration, the setting is an urban planning scenario. The multi-material 3D printed buildings consist of thin layers of white plastic filament and a conductive wireframe to enable touch gestures. Attendees can either interact with the physical model or with the mobile AR interface for selecting, adding or deleting buildings.

The multi-touch interface is essential for smartphone user experience; however many issues such as a lack of physical feedback, the occlusion problem, and the reachability problem still remain. The back-of-device (BoD) approach is a way to address these issues; nevertheless, there are problems such as that the user finger or external landscape affects the view of the camera and that simultaneous input is difficult in the technique using the acceleration sensor. We propose a thin and low-cost method using internal reflection to actualize an additional physical UI on the back of the smartphone. No special devices such as mirrors are needed only acrylic panels and plastic parts. This method extends the smartphone input UI to include a physical button, dial, pointing stick, and also a slider. In addition, the method supports simultaneous input and overcome the effect of the user finger and external landscape.

We present AuraRing, a wearable electromagnetic tracking system for fine-grained finger movement. The hardware consists of a ring with an embedded electromagnetic transmitter coil and a wristband with multiple sensor coils. By measuring the magnetic fields at different points around the wrist, AuraRing estimates the five degree-of-freedom pose of the finger. AuraRing is trained only on simulated data and requires no runtime supervised training, ensuring user and session independence. AuraRing has a resolution of 0.1 mm and a dynamic accuracy of 4.4 mm, as measured through a user evaluation with optical ground truth. The ring is completely self-contained and consumes just 2.3 mW of power.

We proposed a lens forming method that does not require post-output processing using an ultraviolet (UV) printer. The transparent ink is first laminated to form a shape. A UV printer then smooths its surface by filling the layer roughness with a gloss. We implemented a tool to design the lens shape by inputting the lens diameter, focal length, etc.; with this tool, multiple lenses can be arranged and output as data to be printed by a UV printer.

Making modern electric circuits involves a lot of processes, skills, and effort. One challenge in fabricating prototypes is the difficulty of making electric circuits. We propose a fabrication technique that allows users to easily build electric circuits. It is based on a hot stamping technique but uses a single extruder 3D printer instead of dies. The circuits are made of metal foil and can be deposited on various surfaces including paper and acrylic board. Tests confirm that the technique can fabricate circuits on which electronic devices can be implemented by soldering.

Virtual reality (VR) and augmented reality (AR) with head-mounted displays (HMDs) are rapidly becoming popular. Unfortunately, optical see-through HMDs for AR are not compatible with VR and users must prepare the other type of HMD to experience VR. We present a new method for optical see-through HMDs that enables switching between VR and AR by changing the degree of transparency. The proposed method controls liquid crystals to change the transparency in an enclosure constructed of polarizing plates to achieve the switch. The proposed method does not require the use of expensive modules and is applicable to various HMDs including those that are smartphone based and retinal projection based. We construct a prototype implementing the proposed method and evaluate its performance and unique characteristics.

We propose a programming environment for prototyping workflows that consists of custom digital fabrication machines and user-defined interactions. At its core, Machine-o-Matic comprises a domain-specific programming language for defining custom CNC machines as aconfiguration of tools and moving stages connected together. Given a software defined machine configuration, the language compiles to firmware code that allows a user to control and test a physical machine immediately. The language includes constructs for users to define custom actions with the tool and to interface with input from sensors or a camera feed. To aid users in writing Machine-o-Matic programs, we include a drag and drop GUI for assembling, simulating, and experimenting with potential machine configurations before physically fabricating them. We present three proofs of concept to showcase the potential of our programming environment.

Recent hand-held controllers have explored a variety of haptic feedback sensations for users in virtual reality by producing both kinesthetic and cutaneous feedback from virtual objects. These controllers are grounded to the user's hand and can only manipulate objects through arm and wrist motions, not using the dexterity of their fingers as they would in real life. In this paper, we present TORC, a rigid haptic controller that renders virtual object characteristics and behaviors such as texture and compliance. Users hold and squeeze TORC using their thumb and two fingers and interact with virtual objects by sliding their thumb on TORC's trackpad. During the interaction, vibrotactile motors produce sensations to each finger that represent the haptic feel of squeezing, shearing or turning an object. We demonstrate the TORC interaction scenarios for a virtual object in hand.

We are demonstrating a novel design of a live lightfield camera capturing a scene and showing the captured lightfield in realtime in a lightfield display. The simple, distributed design of the camera allows for low-cost construction of an array of 2D cameras that captures high quality, artifact-free imagery of the most challenging of subjects. This camera takes advantage of the natural duality of outside-in lightfield cameras with inside-out lightfield displays, letting us render complex lightfield imagery with a minimum of processing power.

The diversity and the scale of available online instructions introduce opportunities but also user challenges in currently used software interfaces; Users have limited computational resources, and thus often make strategic decisions when browsing, navigating, and understanding instructions to accomplish a task. These strategic user interactions possess nuanced semantics such as users' interpretations, intents, and contexts in which the task is carried out. My dissertation research introduces techniques in constructing data structures that capture the diverse strategies users employ in which the collective nuanced semantics across multiple strategies are preserved. These computational representations are then used as building blocks for designing novel interactions that allow users to effectively browse and navigate instructions, and provide contextual task guidance. Specifically, I investigate 1) structure of instructions for task analysis at scale, 2) structure of collective user task demonstrations, and 3) structure of object uses in how-to videos to support tracking, guiding and searching task states. My research demonstrates that the user-centered organization of information extracted from interaction traces enables novel interfaces with contextual task support.

Affordable rapid 3D printing technologies have become key enablers of the Maker Movement by giving individuals the ability to create physical finished products. However, existing computer-aided design (CAD) tools that allow authoring and editing of 3D models are mostly visually reliant and limit access to people with blindness and visual impairment (BVI). In this paper I address three areas of research that I will conduct as part of my PhD dissertation towards bridging a gap between blind and sighted makers. My dissertation aims to create an accessible 3D design and printing workflow for BVI people through the use of 2.5D tactile displays, and to rigorously understand how BVI people use the workflow in the context of perception, interaction, and learning.

Effective circuit design and debugging require developing an intimate understanding of the behaviors of a complex system. In my work, I've distilled barriers to such understanding to three fundamental challenges of circuit design: transparency, malleability, and modelability. In turn, my research contributes tools that address these challenges through novel changes to the circuit design workflow: Pinpoint improves transparency and malleability in the debugging of printed circuit boards (PCBs) by augmenting board connections with automatic instrumentation and reconfigurable connectivity. Scanalog similarly improves transparency and malleability in prototyping by providing an interactively reprogrammable platform on which to design and tune fully instrumented mixed-signal circuits. My ongoing work addresses issues in modelability through tools that generate empirically-derived fault models and highlight causal relationships between components in a circuit. By evaluating these interactions, my research examines the role of exploratory understanding in circuit design, asking, "How can tools promote understanding of a circuit by facilitating exploration and reflection?"

In this paper, I introduce collective shape-changing interfaces, a class of shape-changing interfaces that consist of a set of discrete collective elements. Through massively parallel transformation, locomotion, and connection of individual building blocks, the overall physical structure can be dynamically changed. Given this parallel change of individual elements, I propose three approaches for user interaction: dynamic, improvised, and actuated collective shape transformation. I exemplify each approach through my own work and possible future work.

The focus of intelligent systems is on "making things easy'' through automation. However, for many cognitive tasks---such as learning, creativity, or sensemaking---there is such a thing as too easy or too automated. Current human-AI design principles, as well as general usability guidelines, prioritize automation, and efficient task execution over human effort. However, this type of advice may not be suitable for designing systems that need to balance automation with other cognitive goals. In these cases, designers lack the necessary tools that will allow them to consider the trade-offs between automation, AI assistance, and human-effort. My dissertation looks at using models from cognitive psychology to inform the design of intelligent systems. The first system, Florum, looks at automation after human-effort as a strategy to facilitate learning from science text. The second system, TakeToons, explores automation as a complementary strategy to human-effort to support creative animation tasks. A third set, SmartCues and Affinity Lens use AI as a last-mile optimization strategy for human sensemaking tasks. Based on these systems, I am looking to develop a design framework that (1) classifies threats across different levels of design including automation, user interface, expectations from AI, and cognition and (2) offers ways to validate design decisions.

Live streaming has recently gained worldwide popularity, due to the affordable digital video devices, high-speed Internet access, and social media. While video games and other entertainment content attract a broad audience to live streaming, it has become an important channel for sharing a variety of non-entertainment content, such as civil content, knowledge sharing, and even promoting traditional cultural practices. However, little research has explored the practices and challenges of the vibrant communities of these streamers who share knowledge or showcase cultural practices through live streams, and few tools have been designed and developed to support their needs for engaging viewers and communicate with viewers more efficiently. The goal of my research is to better understand the practices of these streamers and their communities, and to design tools to better support knowledge sharing and cultural heritage preservation through live streaming.

For decades, animation has been a popular storytelling technique. Traditional tools for creating animations are labor-intensive requiring animators to painstakingly draw frames and motion curves by hand. An alternative workflow is to equip animators with direct real-time control over digital characters via performance, which offers a more immediate and efficient way to create animation. Even when using these existing expression transfer and lip sync methods, producing convincing facial animation in real-time is a challenging task. In this position paper, I describe my past and proposed future research in developing interactive systems for perceptually-valid expression retargeting from humans to stylized characters, real-time lip sync for 2D animation, and building an expressive style aligned embodied conversational agent.

Advances in digital fabrication have created new capabilities and simultaneously reinforced outdated workflows. In my thesis work, I primarily explore alternative workflows for digital fabrication that introduce new capabilities and interactions. Methodologically, I build fabrication systems spanning mechanical design, electronics, and software in order to examine these ideas in specific detail. In this paper, I introduce related work and frame it within the historical context of digital fabrication, and discuss my previous and ongoing work.