ISS '19- Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces

The 2007 launch of the Apple iPhone marked a turning point in the evolution of the experience of humans interacting with and/or through digital technology. Despite leaving ample room for improvement, the iPhone made clear that the field had matured to the point where the primary challenge no longer lay in individual interactions with individual technologies; rather, it had shifted to battling the every-growing, largely technology induced complexity in the social interactions within and amongst the societies of technologies and people. By analogy, the better we get at the architecture and construction of houses, the more the need for urban planning escalates.

From this perspective, the holy grail of "next big thing" turns out not to be a "thing" at all - not a device, application nor service. Rather, it will be a change in the social relationships amongst the things which already exist, will exist, and with and amongst the society of people with whom the technologies are intermingled.

Many of the decacorn or centicorn companies in this startup world are linked to the shared economy in somehow especially for the ones in the space or mobility industry. Many of the home appliance companies like LG Electronics are trying to integrate their products into furniture, while furniture company is trying to create their own space, and some of the architects are now saying they are creating the part of Nature rather than the artificial construction. As a space itself is being recognized as the business opportunity, the inside of the surfable(interactive) surface could be the canvas of the human imagination where limitless supernatural dreams come true. I will cover the future possibilities of interactive surface and space in today's 5G, AI and connected context.

The feeling of presence of virtual entities is an important objective in virtual reality, teleconferencing, augmented reality, exposure therapy and video games. Presence creates emotional involvement and supports intuitive and efficient interactions. As a feeling, presence is mostly measured via subjective questionnaire, but its validity is disputed. We introduce a new method to measure the contribution of several technical parameters toward presence. Its robustness stems from asking participant to rank contrasts rather than asking absolute values, and from the statistical analysis of repeated answers. We implemented this method in a user study where virtual entities were created with a handheld perspective corrected display. We evaluated the impact on two virtual entities' presence of four important parameters of digital visual stimuli: resolution, latency, frame rate and jitter. Results suggest that jitter and frame rate are critical for presence but not latency, and resolution depends on the explored entity.

Interactive tactile graphics have shown a true potential for people with visual impairments, for instance for acquiring spatial knowledge. Until today, however, they are not well adopted in real-life settings (e.g. special education schools). One obstacle consists in the creation of these media, which requires specific skills, such as the use of vector-graphic software for drawing and inserting interactive zones, which is challenging for stakeholders (social workers, teachers, families of people with visual impairments, etc.). We explored how a Spatial Augmented Reality approach can enhance the creation of interactive tactile graphics by sighted users. We developed the system using a participatory design method. A user study showed that the augmented reality device allowed stakeholders (N=28) to create interactive tactile graphics more efficiently than with a regular vector-drawing software (baseline), independently of their technical background.

We present DesignAR, an augmented design workstation for creating 3D models. Our approach seamlessly integrates an interactive surface displaying 2D views with head-mounted, stereoscopic Augmented Reality (AR). This creates a combined output space that expands the screen estate and enables placing 3D objects beyond display borders. For the effective combination of 2D and 3D views, we define different levels of proximity and alignment. Regarding input, multi-touch and pen mitigate issues of precision and ergonomics commonly found in mid-air VR/AR interaction. For creating and refining 3D models, we propose a set of pen and touch techniques with immediate AR feedback, including sketching of rotational solids or tracing physical objects on the surface. To further support a designer's modeling process, we additionally propose orthographic model views and UI offloading in AR as well as freely placeable model instances with real-world reference. Based on our DesignAR prototype, we report on challenges and insights regarding this novel type of display augmentation. The combination of high-resolution, high-precision interactive surfaces with carefully aligned AR views opens up exciting possibilities for future work and design environments, a vision we call Augmented Displays.

Navigating Augmented Reality (AR) environments with a handheld device often requires users to access digital contents (i.e. Points of Interests - POIs) associated with physical objects outside the field of view of the device's camera. Halo3D is a technique that displays the location of off-screen POIs as halos (arcs) along the edges of the screen. Halo3D reduces clutter by aggregating POIs but has not been evaluated. The results of a first experiment show that an enhanced version of Halo3D was 18% faster than the focus+context technique AroundPlot* for pointing at a POI, and perceived as 34% less intrusive than the arrow-based technique Arrow2D. The results of a second experiment in more realistic settings reveal that two variants of Halo3D that show the spatial distribution of POIs in clusters (1) enable an effective understanding of the off-screen environment and (2) require less effort than AroundPlot* to find POIs in the environment.

Traditional computer systems based on the WIMP paradigm (Window, Icon, Menu, Pointer) have shown potential benefits at school (e.g. for web browsing). On the other hand, they are not well suited as soon as hands-on and collaborative activities are targeted. We present CARDS, a Mixed-Reality system that combines together physical and digital objects in a seamless workspace to foster active and collaborative learning. We describe the design process based on a participatory approach with researchers, teachers, and pupils. We then present and discuss the results of a user study that tends to show that CARDS has a good educational potential for the targeted activities.

For pen-steering tasks, the steering law can predict the movement time for both straight and circular paths. It has been shown, however, that those path shapes require different regression expression coefficients. Because it has also been shown that the path curvature (i.e., the inverse of the curvature radius) linearly decreases the movement speed, we refined the steering law to predict the movement time under various curvature radii; our pen-steering experiment included linear to circular shapes. The result showed that the proposed model had good fit for the empirical data: adjusted r2 > 0.95, with the smallest Akaike information criterion (AIC) value among various candidate models. We also discuss how this refined model can contribute to other fields such as predicting driving difficulties.

We propose ShearSheet, a low-cost, feasible, and simple DIY method that enables tangential (shear) force input on a touchscreen using a rubber-mounted slim transparent sheet. The sheet has tiny conductive material(s) attached to specific positions on the underside so that displacement of the sheet is recognized as touch input(s). This allows the system to distinguish between normal touch input and shear force input depending on whether the sheet moves or not, without requiring any external sensors or power. We first introduce ShearSheet's simple implementation for a smartphone and then discuss several promising interaction techniques that augment shear interaction with one- and two-finger operations. We demonstrate the effectiveness of ShearSheet's smooth transition between position- and rate-based control through a controlled user study using simple scrolling tasks. The results confirm that ShearSheet offers improved performance such as fewer operations and higher subjective preference, suggesting its substantial benefits and potential applications.

We conduct a two-part study to better understand pen grip postures for general input like mode switching and com-mand invocation. The first part of the study asks participants what variations of their normal pen grip posture they might use, without any specific consideration for sensing capabilities. The second part evaluates three of their sug-gested postures with an additional set of six postures designed for the sensing capabilities of a consumer EMG armband. Results show that grips considered normal and mature, such as the dynamic tripod and the dynamic quadrupod, are the best candidates for pen-grip based interaction, followed by finger-on-pen postures and grips using pen tilt. A convolutional neural network trained on EMG data gathered during the study yields above 70% within-participant recognition accuracy for common sets of five postures and above 80% for three-posture subsets. Based on the results, we propose design guidelines for pen interaction using variations of grip postures.

Many images on the Web, including photographs and artistic images, feature spatial relationships between objects that are inaccessible to someone who is blind or visually impaired even when a text description is provided. While some tools exist to manually create accessible image descriptions, this work is time consuming and requires specialized tools. We introduce an approach that automatically creates spatially registered image labels based on how a sighted person naturally interacts with the image. Our system collects behavioral data from sighted viewers of an image, specifically eye gaze data and spoken descriptions, and uses them to generate a spatially indexed accessible image that can then be explored using an audio-based touch screen application. We describe our approach to assigning text labels to locations in an image based on eye gaze. We then report on two formative studies with blind users testing EyeDescribe. Our approach resulted in correct labels for all objects in our image set. Participants were able to better recall the location of objects when given both object labels and spatial locations. This approach provides a new method for creating accessible images with minimum required effort.

Permeable, mid-air displays, such as those using fog or water mist are limited by our ability to shape and control the aerosol and deal with two major issues: (1) the size and complexity of the system, and (2) the creation of laminar flow, to retain display quality. Here we present SonicSpray, a technique using ultrasonic Bessel beams to create reconfigurable mid-air displays. We build a prototype from low-cost, off-the-shelf parts. We explore the potential and limitations of SonicSpray to create and redirect laminar flows of fog. We demonstrate a working prototype that precisely controls laminar aerosols through only 6x6 ultrasound transducers array. We describe the implementation steps to build the device, verify the control and projection algorithm for the display, and evaluate its performance. We finally report our exploration of several useful applications, in learning, entertainment and arts.

Physically moving real objects via a computational force connects computers and the real world and has been applied to tangible interfaces and mid-air display. Many researchers have controlled only a stationary real object by computational force. On the other hand, controlling a moving object can expand the real space that is controllable by the computer. In this paper, we explore the potential of computational force from the viewpoint of changing the trajectory of a moving object. Changing the trajectory is the primitive model to control a moving object, and it is the technological challenge requiring high-speed measurement and non-contact force with high-spatial resolution. As a proof-of-concept, we introduce Hopping-Pong changing the trajectory of a flying Ping-Pong Ball (PPB) using ultrasound force. The result shows that Hopping-Pong changes the trajectory of a PPB 344 mm. We conclude that a computational force is capable of controlling a moving object in the real world. This research contributes to expanding the computationally controlled space with applications for augmented sports, HCI and factory automation.

Current gesture interaction paradigm mainly involves a one-to-one gesture-command mapping. This leads to memorability issues regarding (1) the mapping - as each new command requires a new gesture, and (2) the gestures specifics (e.g., motion paths) - that can be complex to leverage the recognition of several gestures. We explore the concept of combining 3D gestures when interacting in smart environments. We first propose a design space to characterize the temporal and spatial combination aspects, and the gesture types used by the combination. We then report results from three user studies in the context of smart TV interaction. The first study reveals that end-users can create gesture sets with combinations fully optimized to reuse gestures. The second study shows that combining gestures can lead to improved memorability compared to single gestures. The third study reveals that preferences for gestures combination appear when single gestures have an abstract gesture-command mapping.

While pressing can enable a wide variety of interesting applications, most press sensing techniques operate only at close distances and rely on fragile electronics. We present EchoTube, a robust, modular, simple, and inexpensive system for sensing low-resolution press events at a distance. EchoTube works by emitting ultrasonic pulses inside a flexible tube which acts as a waveguide and detecting reflections caused by deformations in the tube. EchoTube is deployable in a wide variety of situations: the flexibility of the tubes allows them to be wrapped around and affixed to irregular objects. Because the electronic elements are located at one end of the tube, EchoTube is robust, able to withstand crushing, impacts, water, and other adverse conditions. In this paper, we detail the design, implementation, and theory behind EchoTube; characterize its performance under different configurations; and present a variety of exemplar applications that illustrate its potential.

To achieve better touch GUI designs, researchers have studied optimal target margins, but state-of-the-art results on this topic have been observed for only middle-size touchscreens. In this study, to better determine more practical target arrangements for smartphone GUIs, we conducted four experiments to test the effects of gaps among targets on touch pointing performance. Two lab-based and two crowd-based experiments showed that wider gaps tend to help users reduce the task completion time and the rate of accidental taps on unintended items. For 1D and 2D tasks, the results of the lab-based study with a one-handed thumb operation style showed that 4-mm gaps were the smallest sizes to remove negative effects of surrounding items. The results of the crowd-based study, however, indicated no specific gaps to completely remove the negative effects.

With smartphones being a prime example, touchscreens became one of the most widely used interface to interact with computing systems. Compared to other touchscreen devices, smartphones pose additional challenges as the hand that interacts with the device is commonly used to also hold the device. Consequently, determining how fingers of the hand holding the device can interact with the screen is a non-trivial challenge. A body of recent work investigated the comfortable area in controlled lab studies. This poses limitations as it is based on the assumption that the grips used in the studies are representative for normal smartphone use. In this paper, we extend previous work by providing insights from in-the-wild studies using two different apps that were deployed in the Android App Store. Comparing our results with previous work we confirm that our data fits previously proposed models. Further analyzing the data, we highlight the sweet spot, the position that is touched if the input can be performed on the whole screen.

In graphical user interfaces, users can select multiple objects simultaneously via lasso selection. This can be implemented, for example, by having users select objects by looping around their centers or entire areas. Based on differences in lassoing criteria, we presume that the performance of the criteria also differs. In this study, we compare three lassoing criteria and model lassoing motions while considering these criteria. We conducted two experiments; participants steered through straight-line and one-loop paths by using three criteria. The participants handled the lassoing criteria correctly and performed lassoing at appropriate speeds for each path shape. Although the drawn trajectories varied depending on the lassoing criteria, the criteria in the performance and subjective evaluations did not differ significantly. Additionally, from our results, we build a baseline model to predict the movement time by considering the lassoing criteria. We also discuss further experiments to predict movement time under more complex conditions.

Revealing a hidden widget with a dedicated sliding gesture is a common interaction design in today's handheld devices. Such "Swhidgets" (for swipe-revealed hidden widgets) provide a fast (and sometime unique) access to some commands. Interestingly, swhidgets do not follow conventional design guidelines in that they have no explicit signifiers, and users have to discover their existence before being able to use them. In this paper, we discuss the benefits of this signifierless design and investigate how iOS users deal with this type of widgets. We report on the results of a laboratory study and an online survey, investigating iOS users' experience with swhidgets. Our results suggest that swhidgets~are moderately but unevenly known by participants, yet the awareness and the discovery issues of this design is worthy of further discussion.

Considering the potential physical limitations of older adults, the naturalness of touch-based gestures as an interaction method is questionable. Assuming touch-based gestures are natural, they should be highly learnable and amenable to enhancement through social interactions. To investigate whether social interactions can enhance the learnability of touch gestures for older adults with low digital literacy, we conducted a study with 42 technology- naïve participants aged 64 to 82. They were paired and encouraged to play two games on an interactive tabletop with the expectation to use the drag gesture to complete the games socially. We then compared these results with a previous study of technology-naïve older adults playing the same games individually. The results of the comparisons show that dyadic interactions had some benefits for the participants in helping them to become comfortable with the drag gesture by negotiation and imitation. Further qualitative analysis suggested that playing pairs generally helped learners to comfortably explore the digital environment using the newly acquired skill.

Situated sketching and enactment aim at grounding designs in the spatial, social and cultural practices of a particular place. This is particularly relevant when designing for public places in which human activities are open-ended, multi-faceted, and difficult to anticipate, such as libraries, train stations, or commercial areas. In order to investigate situated sketching and enactment, we developed Ébauche. It enables designers to collaboratively sketch interfaces, distribute them across multiple displays and enact use cases. We present the lessons learned from six situated sketching and enactment workshops on public displays with Ébauche. And we present the results of a controlled study with 8 pairs of designers who used paper and Ébauche. We present the various ways in which participants leveraged the place, and how paper or Ébauche influenced the integration of their designs in the place. Looking at the design outcomes, our results suggest that paper leads to broader exploration of ideas and deeper physical integration in the environment. Whereas Ébauche leads to more refined sketches and more animated enactments.

This paper describes an explorative design study conducted in the scope of a collaborative research project in the district heating domain. In the scope of the project, we have arranged extensive field studies at two power plants to understand the workflows, problems, and needs of industrial operators. We relied on the gained knowledge to design and develop novel visual interfaces that would communicate the overall status of the district heating system at-a-glance. We aimed at exploring potential directions and alternatives beyond conventional industrial interfaces. One particular aspect of our research was related to how the physicality of the underlying industrial processes can be expressed by purely visual means. The paper introduces three high-fidelity prototypes demonstrating the novel visualizations developed. The paper explains the design choices made, namely the relation between the selected visual encodings to the requirements of the industrial operators' tasks. Preliminary evaluation indicates industrial operators' interest in the designed solutions. Future work will incorporate an extensive qualitative evaluation on site.

In immersive display environments, such as virtual or augmented reality, we can make explicit the connections between data points in visualisations and their context in the world, or in other visualisations. This paper considers the requirements and design space for drawing such links in order to minimise occlusion and clutter. A novel possibility in immersive environments is to optimise the link layout with respect to a particular point of view. In collaborative scenarios there is the need to do this for multiple points of view. We present an algorithm to achieve such link layouts and demonstrate its applicability in a variety of practical use cases.

Externalizing one's thoughts can be helpful during data analysis, such as which one marks interesting data, notes hypotheses, and draws diagrams. In this paper, we present two exploratory studies conducted to investigate types and use of externalizations during the analysis process. We first studied how people take notes during different stages of data analysis using VoyagerNote, a visualization recommendation system augmented to support text annotations, and coupled with participants' favorite external note-taking tools (e.g., word processor, pen & paper). Externalizations manifested mostly as notes written on paper or in a word processor, with annotations atop views used almost exclusively in the initial phase of analysis. In the second study, we investigated two specific opportunities: (1) integrating digital pen input to facilitate the use of free-form externalizations and (2) providing a more explicit linking between visualizations and externalizations. We conducted the study with VoyagerInk, a visualization system that enabled free-form externalization with a digital pen as well as touch interactions to link externalizations to data. Participants created more graphical externalizations with VoyagerInk and revisited over half of their externalizations via the linking mechanism. Reflecting on the findings from these two studies, we discuss implications for the design of data analysis tools.

In this paper, we present BEXHI, a new mechanical structure for prototyping expandable and bendable handheld devices. Many research projects have pushed bendable surfaces from prototypes to commercially viable devices. In the meantime, expandable devices have become a topic of interest letting one foresee that such devices are on the horizon. With BEXHI, we provide a structure to explore the combined capabilities of these devices. The structure consists of multiple interweaved units allowing non-porous expandable surfaces to bend. Through an instanciation, we illustrate and discuss that the BEXHI structure allows for the exploration of the combination of both bend and expansion interaction spaces.

We present a digital assistance approach for applied metrology on near-symmetrical objects. In manufacturing, systematically measuring products for quality assurance is often a manual task, where the primary challenge for the workers lies in accurately identifying positions to measure and correctly documenting these measurements. This paper focuses on a use-case, which involves metrology of small near-symmetrical objects, such as LEGO bricks. We aim to support this task through situated visual measurement guides. Aligning these guides poses a major challenge, since fine grained details, such as embossed logos, serve as the only feature by which to retrieve an object's unique orientation. We present a two-step approach, which consists of (1) locating and orienting the object based on its shape, and then (2) disambiguating the object's rotational symmetry based on small visual features. We apply and compare different deep learning approaches and discuss our guidance system in the context of our use case.

User interfaces rarely adapt to the specific user preferences or the task at hand. We present a method that allows to quickly and inexpensively create personalized interfaces from plain paper. Users can cut out shapes and assign control functions to these paper snippets via a simple configuration interface. After configuration, control takes place entirely through the manipulation of the paper shapes, providing the experience of a tailored tangible user interface. The shapes and assignments can be dynamically changed during use. Our system is based on markerless tracking of the user's fingers and the paper shapes on a surface using an RGBD camera mounted above the interaction space, which is the only hardware sensor required. Our approach and system are backed up by two studies where we determined what shapes and interaction abstractions users prefer, and verified that users can indeed employ our system to build real applications with paper snippet interfaces.

We present SpaceState, a system for designing spatial user interfaces that react to changes of the physical layout of a room. SpaceState uses depth cameras to measure the physical environment and allows designers to interactively define global and local states of the room. After designers defined states, SpaceState can identify the current state of the physical environment in real-time. This allows applications to adapt the content to room states and to react to transitions between states. Other scenarios include analysis and optimizations of work flows in physical environments. We demonstrate SpaceState by showcasing various example states and interactions. Lastly, we implemented an example application: A projection mapping based tele-presence application, which projects a remote user in the local physical space according to the current layout of the space.

Annotating chart images for training machine learning models is tedious and repetitive especially in that chart images often have a large number of visual elements to annotate. We present Autotator, a semi-automatic chart annotation system that automatically provides suggestions for three annotation tasks such as labeling a chart type, annotating bounding boxes, and associating a quantity. We also present a web-based interface that allows users to interact with the suggestions provided by the system. Finally, we demonstrate a use case of our system where an annotator builds a training corpus of bar charts.

We present a remote assistance system that enables a remotely located expert to provide guidance using hand gestures to a customer who performs a physical task in a different location. The system is built on top of a web-based real-time media communication framework which allows the customer to use a commodity smartphone to send a live video feed to the expert, from which the expert can see the view of the customer's workspace and can show his/her hand gestures over the video in real-time. The expert's hand gesture is captured with a hand tracking device and visualized with a rigged 3D hand model on the live video feed. The system can be accessed via a web browser, and it does not require any app software to be installed on the customer's device. Our system supports various types of devices including smartphone, tablet, desktop PC, and smart glass. To improve the collaboration experience, the system provides a novel gravity-aware hand visualization technique.

We propose a data-driven method for realistic texture rendering on an electrovibration display. To compensate the nonlinear dynamics of an electrovibration display, we use nonlinear autoregressive with external input (NARX) neural networks as an inverse dynamics model of an electrovibration display. The neural networks are trained with lateral forces resulting from actuating the display with a pseudo-random binary signal (PRBS). The lateral forces collected from the textured surface with various scanning velocities and normal forces are fed into the neural network to generate the actuation signal for the display. For arbitrary scanning velocity and normal force, we apply the two-step interpolation scheme between the closest neighbors in the velocity-force grid.

Indirect writing interface is currently widely used for handwriting text entry with a pen, however, it may cause additional eye and hand movement. Gu and Lee suggested a revival of the concept of direct writing and they showed the potential of the direct writing interface in editing text. In this demo, we demonstrate an In-Place Ink text editor prototype which is a notepad-like simple text editor with direct writing feature.

Planning for epilepsy surgery requires precise localization of the seizure onset zone. This permits physicians to make accurate estimates about the postoperative chances of seizure freedom and the attendant risk. Patients with complex epilepsies may require intracranial electroencephalography (iEEG) to best lateralize and localize the seizure onset zone. Magnetic resonance imaging (MRI) data of implanted intracranial electrodes in a brain is used to confirm correct placement of the intracranial electrodes and to accurately map seizure onset and spread through the brain. However, the relative lack of tools co-registering iEEG with MRI data renders this a time-consuming investigation since for epilepsy specialists who have to manually map this information in 2-dimensional space. Our immersive analytics tool, EPES (Epilepsy Pre-Evaluation Space), provides an application to analyze iEEG data and its fusion with the corresponding intracranial electrodes' recordings of the brain activity. EPES highlights where a seizure is occurring and how it propagates through the virtual brain generated from the patient MRI data.

We present FIESTA, a prototype system for collaborative immersive analytics (CIA). In contrast to many existing CIA prototypes, FIESTA allows users to collaboratively work together wherever and however they wish---untethered from mandatory physical display devices. Users can freely move around in a shared room-sized environment, author and generate immersive data visualisations, position them in the space around them, and share and communicate their insights to one another. Certain visualisation tasks are also supported to facilitate this process, such as details on demand and brushing and linking.

Deep reinforcement learning (DRL) has had many successes on complex tasks, but is typically considered a black box. Opening this black box would enable better understanding and trust of the model which can be helpful for researchers and end users to better interact with the learner. In this paper, we propose a new visualization to better analyze DRL agents and present a case study using the Pommerman benchmark domain. This visualization combines two previously proven methods for improving human understanding of systems: saliency mapping and immersive visualization.

One-handed mobile use, which is predominantly thumb-driven, presents interaction challenges like screen occlusion, reachability of far and inside corners, and an increased chance of dropping the device. We adopt a Research through Design approach around single-hand mobile interaction by exploring a variety of back-of-device tangibles (including a touchpad, scroller, magnetic button, push button, slider, stretchable spiral and a ring joystick). The latter 'joy'-stick was inspired from the recent popular but passive ring phone 'holders', which we combined into ?JoyHolder' - a joystick-based interactive phone holder for tangible back-of-device input interactions. We demonstrate our low-fidelity and medium-fidelity prototypes (using crafting and digital fabrication methods) and our interactive JoyHolder to encourage discussion on tangible back-of-device interactions. Preliminary insights from a pilot-study we ran reflects the hesitation for adopting some of these tangibles, the potential of others and the importance of physical feedback while using back-of-device input modalities.

Complex data comprehension is a hard task for visually impaired people, for the lack of viable supporting tools. We designed a web-based interactive navigation system to enable visually impaired people to effectively explore a simple data table on common touch devices. Due to ecological factor, there are still many blind people who are not used to complex structured dataset. Thus we made user interactions consistent with major mobile screen readers to minimize the users' burden. Users can easily overview and query detailed information while optimizing the cognitive workload.

There are needs for pen interaction on a laptop, and the market sees many pen-enabled laptop products. Many of them can be transformed into tablets when pen interaction is needed. In a real situation, however, users use multiple applications simultaneously, and such a convertible feature may not be useful. We introduce MirrorPad, a novel interface device contained in a laptop for direct pen interaction. It is both a normal touchpad and a viewport for pen interaction with a mirrored region on the screen. MirrorPad allows users to perform pen annotation work and keyboard work interspersed in the same workflow while the device remains in the laptop-like configuration.

Despite more than three decades of research in augmented reality and shown positive effect of AR in educational settings, we still don't witness spread of this technology in the schools. Complex technology and limited educational content are among the reasons for this absence. Authoring systems can play a positive role in introduction of AR into the school settings. In this paper we introduce ISAR, a domain-independent authoring system for a camera-projector based interactive tabletop. ISAR allows teachers to create learning content and define interactions for the tabletop themselves, without the need for programming, and hence reduces the entrance barrier of educational practitioners for experimenting with augmented reality and tangible interactions.

A system control technique (changing the interaction mode) is needed to provide various spatial interaction techniques for virtual, augmented and mixed reality (VR/AR/MR) applications. We propose HandPoseMenu, a hand posture-based virtual menus system in 3D space. HandPoseMenu recognizes the non-dominant hand's posture using a depth sensor that is attached to a wearable display and presents the virtual menu corresponding to the hand posture beside the non-dominant hand. When the user selects a desired function among the presented menu as a dominant hand, the interaction mode changes. The combination of the two system controls (graphical menu and hand posture) makes it possible to display the menu that the user wants to use in 3D space. This paper describes the implementation of HandPoseMenu and its application.

We present an interactive exhibition concept built on existing physical wall panels in a museum. By combining the unique exhibition contents with various rendering and interface techniques, we create immersive contents that have two-way communication and multi-modal feedback as key features to provide engaging experiences. We apply our interactive panel concept to a mummy exhibit in a natural history museum in Korea and demonstrate its effectiveness. The final goal of this project is for it to serve as a platform for virtual museums to provide an opportunity to expand offline museums to an online experience.

The present paper describes research results and lessons learned from a multisensory exhibit held in Gwacheon National Science Museum. We introduced an all-in-one multisensory device that can produce visual, auditory, and vibrotactile stimuli to create a novel sensory experience. Three sensory conditions were considered, and the ways sensory feedback differed under the different conditions were examined. In total, 113 people visited our exhibition. From among these visitors, we carried out self-report questionnaires, interviews, and EDA activities measurement with 60 participants. The results showed that the vibrotactile and auditory stimuli conditions were pleasing and present compared to the condition of only auditory stimulus. When applying vibrotactile stimulus, understanding and developing situational vibrotactile stimulus in response to the auditory sensory stimulus might be effective compared to deploying ambient vibrotactile sensory stimulus. Additional experiments and data analysis are needed. However, we hope that this case study will inspire researchers and curators who are willing to introduce the multisensory experience into the actual museum setting.

The proliferation of mobile payment applications in recent years has further decoupled the physical act of paying from the consumption experience. Prior research suggests that this decreases the psychological 'pain' that consumers feel when making a purchase, and leads them to spend more money than they otherwise would. The present research proposes a system that restores the physical sensation of paying to mobile payment apps. Drawing on a preliminary user study, as well as prior theories on prehensile grip types and haptic vibration feedback thresholds, we develop a device consisting of an array of vibration motors that can be attached to the back of a mobile phone and can be controlled through a custom application to provide vibration feedback of varying frequencies and durations at the time of mobile payment. Our preliminary findings suggest that a configuration of vibration motors located around the right and left edges of the phone, and which give high frequency, short duration vibrations upon payment is the most effective way inducing the pain of payment.

Though Miles Apart is an interactive installation that encourages people to join together through a soft tangible interface, to reminisce on their personal experiences. Based on a traditional Chinese melody, Shuidiao Getou, the installation embodies the idea of separation but, though apart, "we are still able to share the beauty of the moon together''. Initially, as people enter the area, they are invited to sit around the pond-like surface, near glowing points. When a viewer grasps a glowing, soft conductive unit in their hands, it will trigger videos in response to their touch. Viewers will listen and watch stories from people of different cultures and backgrounds, as they recollect on their memories. It creates a meditative, immersive environment, where the collective is allowed to sit and experience nostalgia.

In a telepresence scenario with remote users discussing a document, it can be difficult to follow which parts are being discussed. One way to address this is by showing the user's hand position on the document, which also enables expressive gestural communication. An important practical problem is how to capture and transmit the hand movements efficiently with high resolution document images. We propose a tabletop system with two channels that integrates document capture with a 4K video camera and hand tracking with a webcam, in which the document image and hand skeleton data are transmitted at different rates and handled by a lightweight Web browser client at remote sites. To enhance the rendering, we employ velocity based smoothing and ephemeral motion traces. We tested our prototype over long distances from USA to Japan and to Italy, and report on latency and jitter performance. Our system achieves relatively low latency over a long distance in comparison with a tele-immersive system that transmits mesh data over much shorter distances.

Gesture recognition devices provide a new means for natural human-computer interaction. However, when selecting these devices for games, designers might find it challenging to decide which gesture recognition device will work best. In the present research, we compare three vision-based, hand gesture devices: Leap Motion, Microsoft's Kinect, and Intel's RealSense. We developed a simple hand-gesture based game to evaluate performance, cognitive demand, comfort, and player experience of using these gesture devices. We found that participants' preferred and performed much better using Leap Motion and Kinect compared to using RealSense. Leap Motion also outperformed or was equivalent to Kinect. These findings suggest that not all gesture recognition devices can be suitable for games and that designers need to make better decisions when selecting gesture recognition devices and designing gesture based games to insure the usability, accuracy, and comfort of such games.

Composite wearable computers consist of multiple wearable devices connected together and working as a cohesive whole. These composite wearable computers are promising for augmenting our interaction with the physical, virtual, and mixed play spaces (e.g., mixed reality games). Yet little research has directly addressed how mixed reality system designers can select wearable input devices and how these devices can be assembled together to form a cohesive wearable computer. We present an initial taxonomy of wearable input devices to aid designers in deciding which devices to select and assemble together to support different mixed reality systems. We undertook a grounded theory analysis of 84 different wearable input devices resulting in a design taxonomy for composite wearable computers. The taxonomy consists of two axes: TYPE OF INTERACTIVITY and BODY LOCATION. These axes enable designers to identify which devices fill particular needs in the system development process and how these devices can be assembled together to form a cohesive wearable computer.

We experimentally compare the performance and usability of tablet-based and see-through head-mounted display (HMD)-based interaction techniques for selecting 3D virtual objects projected on a table. This study is a first step toward a better understanding of the advantages and limitations of these interaction techniques, with the perspective of improving interaction with augmented maps. To this end, we evaluate the performance of 3 interaction techniques in selecting 3D virtual objects in sparse and dense environments: (1) the direct touch interaction with a HMD; (2) the ray-casting interaction with a HMD; and (3) the touch interaction on a tablet. Our results show that the two techniques using a HMD are faster, less physically tiring and preferred by the participants over the tablet. The HMD-based interaction techniques perform equally well but the direct touch technique seems to be less impacted by small targets and occlusion.

Augmented reality (AR) technology has been used to integrate online product information to the physical shopping experience by relying on hardware outputs such as head-mounted displays or mobile devices, which interferes with the customers natural in-store shopping experience. Different from traditional AR, spatial augmented reality (SAR) could potentially offer an unmediated augmented physical experience. In this paper, we present a proof-of-concept prototype that augments existing physical shops with online personalized product information without disrupting the customer's natural shopping behavior. We envision a real and augmented book shop experience, using SAR to provide immersive digital and online information overlaid onto physical products, breaking down the boundary between physical space and digital information at retail shops. Moreover, we design unmediated interaction that reacts to the relative spatial and temporal context of the users, based on user's natural interactions with the physical products, without the need for learning special interactions/gestures. We also discuss the design challenges and implications for this in-store shopping experience.

Interaction on the surface usually limited in mechanical movement and does not have surface texture, which results in a lack of haptic feedback. Various techniques have been introduced to supplement this limitation. The main objective of this tutorial is to let the ISS community experience those technologies in hand. By its nature, the haptic interface is hardly explainable with a document, video, and audio. We provide a unique opportunity for the ISS community to feel and experience the wide range of haptic techniques on the surface. In specific, the tutorial will cover the following topics. 1) haptic feedback on a small mobile device 2) Texture rendering on a larger surface 3) An illusion of compliance on a rigid device 4) Pin-array display for information transfer.

Navigation in the virtual environment is one of the interactions which happens very frequently. Physical locomotion method is popular due to its naturalness in VR but it requires wide area. To use minimum space for the navigation, walk-in-place method is suggested as an alternative way for the virtual environment navigation. In this tutorial, we will cover the following topics. 1) Treadmill based walk-in-place methods 2) Non-treadmill based walk-in-place methods

Immersive Analytics is an emerging interdisciplinary research area that investigates the use of nontraditional display and input technology to immerse users in their data. A prominent aspect this research investigates is "immersive data visualization", which uses augmented and virtual reality technology to visually immerse users in their data to facilitate collaboration, exploration and understanding. Currently, there is a lack of simple tools to build interactive data visualizations in immersive environments. The de facto approach is to use off-the-shelf game engines because they allow easy prototyping of 3D user interfaces in these immersive environments. However, game engines do not consider the specialized requirements of data visualization, such as the type and structure of datasets, the breadth of data, especially in the age of "big data", and typical information visualization tasks. IATK: Immersive Analytics Toolkit is an open source visualization toolkit for the Unity game engine that fills this gap. Specifically, IATK: supports an infovis pipeline for virtual and augmented reality environments; visualizes large (up to 1 million) data points at an optimal framerate for immersive applications; and, provides a technology-agnostic model for user interactions with immersive visualizations. This tutorial will introduce participants to the Unity game engine and teach practical skills for implementing immersive data visualisations using IATK.