Product visualization in AR/VR applications requires a largely manual process of data preparation. Previous publications focus on error-free triangulation or transformation of product structure data and display attributes for AR/VR applications. This paper focuses on the preparation of the required geometry data. In this context, a significant reduction in effort can be achieved through automation. The steps of geometry preparation are identified and examined concerning their automation potential. In addition, possible couplings of sub-steps are discussed. Based on these explanations, a structure for the geometry preparation process is proposed. With this structured preparation process, it becomes possible to consider the available computing power of the target platform during the geometry preparation. The number of objects to be rendered, the tessellation quality, and the level of detail (LOD) can be controlled by the automated choice of transformation parameters. Through this approach, tedious preparation tasks and iterative performance optimization can be avoided in the future, which also simplifies the integration of AR/VR applications into product development and use. A software tool is presented in which partial steps of the automatic preparation are already implemented. After an analysis of the product structure of a CAD file, the transformation is executed for each component. Functions implemented so far allow, for example, the selection of assemblies and parts based on filter options, the transformation of geometries in batch mode, the removal of certain details, and the creation of UV maps. Flexibility, transformation quality, and timesavings are described and discussed.

This article presents a software platform to design serious games for the rehabilitation of severe memory loss by means of virtual reality (VR). In particular, the focus is on retrograde amnesia, a condition affecting patient's quality of life usually after brain stroke. Currently, the standard rehabilitation process includes showing pictures of familiar environments to help memory recover, while the proposed rehabilitation solution aims at developing patient-specific serious games for memory loss starting from 3D scanning of familiar environments. The Occipital Structure sensor and the Skanect application have been used for the virtualization of the real objects and the environment. Instead of following the traditional approach to design a video game during which the game logic is specifically developed for a virtual scene and the software code is not meant to be recombined, a modular procedure has been designed using Unity to interface the virtual objects of each acquired environment without modifying the game logic. In addition, the developed solution makes available a set of software modules for patient's monitoring and data management to automatically generate medical reports, which can be easily connected to each new serious game. A test has been performed to assess the main features of the VR platform and its usability recruiting ten testers. Medical personnel evaluated positively the platform, and they highlighted the importance of objective data to improve the ecological validity of the cognitive rehabilitation for retrograde amnesia.

This work presents a virtual reality (VR) Exergame application designed to prevent work-related musculoskeletal disorders (WMSDs). Moreover, to help adapt the tasks of the exergame, a machine learning model that predicts users’ exercise intensity level is presented. WMSDs are an important issue that can have a direct economic impact to an organization. Exercise and stretching is one method that can benefit workers and help prevent WMSDs. While several applications have been developed to prevent WMSDs, most of them suffer from a lack of immersivity or they just focus on education and not necessarily on helping workers warm-up or stretch. In light of this, an Exergame application that leverages VR and Depth-sensor technology to help provide users with an immersive first-person experience that engages them in physical activities is introduced in this work. The objective of the Exergame is to motivate users to perform full-body movements in order to pass through a series of obstacles. While in the game, users can visualize their motions by controlling the virtual avatar with their body movements. It is expected that this immersivity will motivate and encourage the users. Initial findings show the positive effects that the base exergame has on individuals’ motivation and physical activity. The results indicate that the application was able to engage individuals in low-intensity exercises that produced significant and consistent increases in their heart rate. Lastly, the results show that the machine learning model predicted users’ exercise activity level with an accuracy of 76.67%.

The issue of training operators in the use of machinery is topical in the industrial field and in many other contexts, such as university laboratories. Training is about learning how to use machinery properly and safely. Beyond the possibility of studying manuals to learn how to use a machine, operators typically learn through on-the-job training. Indeed, learning by doing is in general more effective, tasks done practically are remembered more easily, and the training is more motivating and less tiresome. On the other hand, this training method has several negative factors. In particular, safety may be a major issue in some training situations. An approach that may contribute overcoming negative factors is using Virtual Reality and digital simulation techniques for operators training. The research work presented in this paper concerns the development of a multisensory virtual reality application for training operators to properly use machinery and personal protective equipment (PPE). The context selected for the study is a university laboratory hosting manufacturing machinery. The application allows user to navigate the laboratory, to approach a machine and learn about how to operate it, and also to use proper PPE while operating a machine. Specifically, the paper describes the design and implementation of the application and presents the results of preliminary testing sessions.

This paper investigates the hypothesis that men and women perform differently in virtual environments in terms of balance and coordination during surprise events. It reports on an experiment in human subject balance and coordination using an HTC Vive head-mounted display (HMD) to create a virtual environment. For the experiment, 30 male and 30 female subjects of college age were asked to navigate along a clear path in a virtual world using a controller with their dominant hand and asked to balance a virtual ball on a virtual plate using the other controller in the nondominant hand. The test subjects moved along a clearly marked path, with three surprise obstacles occurring: a large rock landing near the path, an explosion near the path, and a flock of birds coming across the path. Data included six degrees-of-freedom (DOF) trajectories for the head and both hands, as well as data gathered by the computer system on ball location and velocity, plate location and velocity, and ball status. Likert scale questionnaires were answered by the test subjects relative to video game experience, sense of presence, and ease of managing the ball movement. Statistics showed that the male students dropped the ball less frequently at p = 0.0254 and p = 0.0036. In contrast, female students were aware of their performance with correlation levels of 0.632 and 0.588.

This work presents a deep reinforcement learning (DRL) approach for procedural content generation (PCG) to automatically generate three-dimensional (3D) virtual environments that users can interact with. The primary objective of PCG methods is to algorithmically generate new content in order to improve user experience. Researchers have started exploring the use of machine learning (ML) methods to generate content. However, these approaches frequently implement supervised ML algorithms that require initial datasets to train their generative models. In contrast, RL algorithms do not require training data to be collected a priori since they take advantage of simulation to train their models. Considering the advantages of RL algorithms, this work presents a method that generates new 3D virtual environments by training an RL agent using a 3D simulation platform. This work extends the authors’ previous work and presents the results of a case study that supports the capability of the proposed method to generate new 3D virtual environments. The ability to automatically generate new content has the potential to maintain users’ engagement in a wide variety of applications such as virtual reality applications for education and training, and engineering conceptual design.

The paper describes the design of an innovative virtual reality (VR) system, based on a combination of an olfactory display and a visual display, to be used for investigating the directionality of the sense of olfaction. In particular, the design of an experimental setup to understand and determine to what extent the sense of olfaction is directional and whether there is prevalence of the sense of vision over the one of smell when determining the direction of an odor, is described. The experimental setup is based on low-cost VR technologies. In particular, the system is based on a custom directional olfactory display (OD), a head mounted display (HMD) to deliver both visual and olfactory cues, and an input device to register subjects' answers. The paper reports the design of the olfactory interface as well as its integration with the overall system.

Augmented reality (AR) has experienced a breakthrough in many areas of application thanks to cheaper hardware and a strong industry commitment. In the field of management of urban facilities, this technology allows virtual access and interaction with hidden underground elements. This paper presents a new approach to enable AR in mobile devices such as Google Tango, which has specific capabilities to be used outdoors. The first objective is to provide full functionality in the life-cycle management of subsoil infrastructures through this technology. This implies not only visualization, interaction, and free navigation, but also editing, deleting, and inserting elements ubiquitously. For this, a topological data model for three-dimensional (3D) data has been designed. Another important contribution of the paper is getting exact location and orientation performed in only a few minutes, using no additional markers or hardware. This accuracy in the initial positioning, together with the device sensing, avoids the usual errors during the navigation process in AR. Similar functionality has also been implemented in a nonubiquitous way to be supported by any other device through virtual reality (VR). The tests have been performed using real data of the city of Jaén (Spain).

Modern color and depth (RGB-D) sensing systems are capable of reconstructing convincing virtual representations of real world environments. These virtual reconstructions can be used as the foundation for virtual reality (VR) and augmented reality environments due to their high-quality visualizations. However, a main limitation of modern virtual reconstruction methods is the time it takes to incorporate new data and update the virtual reconstruction. This delay prevents the reconstruction from accurately rendering dynamic objects or portions of the environment (like an engineer performing an inspection of a machinery or laboratory space). The authors propose a multisensor method to dynamically capture objects in an indoor environment. The method automatically aligns the sensors using modern image homography techniques, leverages graphics processing units (GPUs) to process the large number of independent RGB-D data points, and renders them in real time. Incorporating and aligning multiple sensors allows a larger area to be captured from multiple angles, providing a more complete virtual representation of the physical space. Performing processing on GPU's leverages the large number of processing cores available to minimize the delay between data capture and rendering. A case study using commodity RGB-D sensors, computing hardware, and standard transmission control protocol internet connections is presented to demonstrate the viability of the proposed method.

In the past few years, there have been some significant advances in consumer virtual reality (VR) devices. Devices such as the Oculus Rift, HTC Vive, Leap Motion™ Controller, and Microsoft Kinect ® are bringing immersive VR experiences into the homes of consumers with much lower cost and space requirements than previous generations of VR hardware. These new devices are also lowering the barrier to entry for VR engineering applications. Past research has suggested that there are significant opportunities for using VR during design tasks to improve results and reduce development time. This work reviews the latest generation of VR hardware and reviews research studying VR in the design process. Additionally, this work extracts the major themes from the reviews and discusses how the latest technology and research may affect the engineering design process. We conclude that these new devices have the potential to significantly improve portions of the design process.

Existing techniques for motion imitation often suffer a certain level of latency due to their computational overhead or a large set of correspondence samples to search. To achieve real-time imitation with small latency, we present a framework in this paper to reconstruct motion on humanoids based on sparsely sampled correspondence. The imitation problem is formulated as finding the projection of a point from the configuration space of a human's poses into the configuration space of a humanoid. An optimal projection is defined as the one that minimizes a back-projected deviation among a group of candidates, which can be determined in a very efficient way. Benefited from this formulation, effective projections can be obtained by using sparsely sampled correspondence, whose generation scheme is also introduced in this paper. Our method is evaluated by applying the human's motion captured by an RGB-depth (RGB-D) sensor to a humanoid in real time. Continuous motion can be realized and used in the example application of teleoperation.

With design teams becoming more distributed, the sharing and interpreting of complex data about design concepts/prototypes and environments have become increasingly challenging. The size and quality of data that can be captured and shared directly affects the ability of receivers of that data to collaborate and provide meaningful feedback. To mitigate these challenges, the authors of this work propose the real-time translation of physical objects into an immersive virtual reality environment using readily available red, green, blue, and depth (RGB-D) sensing systems and standard networking connections. The emergence of commercial, off-the-shelf RGB-D sensing systems, such as the Microsoft Kinect, has enabled the rapid three-dimensional (3D) reconstruction of physical environments. The authors present a method that employs 3D mesh reconstruction algorithms and real-time rendering techniques to capture physical objects in the real world and represent their 3D reconstruction in an immersive virtual reality environment with which the user can then interact. Providing these features allows distributed design teams to share and interpret complex 3D data in a natural manner. The method reduces the processing requirements of the data capture system while enabling it to be portable. The method also provides an immersive environment in which designers can view and interpret the data remotely. A case study involving a commodity RGB-D sensor and multiple computers connected through standard TCP internet connections is presented to demonstrate the viability of the proposed method.

The research presented here describes an industry case study of the use of immersive virtual reality (VR) as a general design tool with a focus on the decision making process. A group of design and manufacturing engineers, who were involved in an active new product development project, were invited to participate in three design reviews in an immersive environment. Observations, interviews, and focus groups were conducted to evaluate the effect of using this interface on decision making in early product design. Because the team members were actively engaged in a current product design task, they were motivated to use the immersive technology to address specific challenges they needed to solve to move forward with detailed product design. This case study takes the approach of asking not only what can users do from a technology standpoint but also how their actions in the virtual environment influence decision making. The results clearly show that the team identified design issues and potential solutions that were not identified or verified using traditional computer tools. The design changes that were the outcome of the experience were implemented in the final product design. Another result was that software familiarity played a significant role in the comfort level and subsequent effectiveness of the team discussions. Finally, participants commented on how the immersive VR environment encouraged an increased sense of team engagement that led to better discussions and fuller participation of the team members in the decision process.

The sense of smell has a great importance in our daily life. Recently, smells have been used for marketing purposes for improving the people's mood and for communicating information about products as household cleaners and food. However, the scent design discipline can be used for creating a “scent identity” of these products not traditionally associated to a specific smell, in order to communicate their features to customers. In the area of virtual reality (VR), several researches concerned the integration of smells in virtual environments. The research questions addressed in this paper concern if virtual prototypes (VP), including smell simulation, can be used for evaluating products as effectively as studies performed in real environments, and also if smells can enhance the users' sense of presence in virtual environments. For this purpose, a VR experimental framework including a prototype of a wearable olfactory display (wOD) has been set up, and experimental tests have been carried out.

Haptic force-feedback can provide useful cues to users of virtual environments. Body-based haptic devices are portable but the more commonly used ground-based devices have workspaces that are limited by their physical grounding to a single base position and their operation as purely position-control devices. The “bubble technique” has recently been presented as one method of expanding a user's haptic workspace. The bubble technique is a hybrid position-rate control system in which a volume, or “bubble,” is defined entirely within the physical workspace of the haptic device. When the device's end effector is within this bubble, interaction is through position control. When the end effector moves outside this volume, an elastic restoring force is rendered, and a rate is applied that moves the virtual accessible workspace. Publications have described the use of the bubble technique for point-based touching tasks. However, when this technique is applied to simulations where the user is grasping virtual objects with part-to-part collision detection, unforeseen interaction problems surface. Methods of addressing these challenges are introduced, along with discussion of their implementation and an informal investigation.

A competitive usability study was employed to measure user performance and user preference for immersive virtual environments (VEs) with multimodal gestural interfaces when compared directly with nonstereoscopic traditional CAD interfaces that use keyboard and mouse. The immersive interfaces included a wand and a data glove with voice interface with an 86 in. stereoscopic rendering to screen; whereas, the traditional CAD interfaces included a 19 in. workstation with keyboard and mouse and an 86 in. nonstereoscopic display with keyboard and mouse. The context for this study was a set of “real world” engineering design scenarios. These design scenarios include benchmark 1—navigation, benchmark 2—error finding and repair, and benchmark 3—spatial awareness. For this study, two populations of users were employed, novice (n = 15) and experienced (n = 15). All users experienced three successive trials to quantify the effects of limited learning. Statistically based comparisons were made using both parametric and nonparametric methods. Conclusions included improved capability and user preference for immersive VEs and their interfaces were statistically significant for navigation and error finding/repair for immersive interfaces.

This paper focuses on the use of virtual reality (VR) systems for teaching industrial assembly tasks and studies the influence of the interaction technology on the learning process. The experiment conducted follows a between-subjects design with 60 participants distributed in five groups. Four groups were trained on the target assembly task with a VR system, but each group used a different interaction technology: mouse-based, Phantom Omni® haptic, and two configurations of the Markerless Motion Capture (Mmocap) system (with 2D or 3D tracking of hands). The fifth group was trained with a video tutorial. A post-training test carried out the day after evaluated performance in the real task. The experiment studies the efficiency and effectiveness of each interaction technology for learning the task, taking in consideration both quantitative measures (such as training time, real task performance, evolution from the virtual task to real one), and qualitative data (user feedback from a questionnaire). Results show that there were no significant differences in the final performance among the five groups. However, users trained under mouse and 2D-tracking Mmocap systems took significantly less training time than the rest of the virtual modalities. This brings out two main outcomes: (1) the perception of collisions using haptics does not increase the learning transfer of procedural tasks demanding low motor skills and (2) Mmocap-based interactions can be valid for training this kind of tasks.

When digitally realized, virtual environments (VEs) do not perfectly match the physical environments they are supposed to emulate. This paper deals with energy aspects of such a mismatch, i.e., artificial energy leaks. A methodology is developed that employs smooth correction (SC) and leak dissipation (LD) to achieve a stable interconnection of the VE with the haptic device. The SC-LD naturally blends with the original laws for rendering the VE and gives rise to modified force feedback laws. These laws can be regarded as energy-consistent discretizations of their continuous-time counterparts. For some fundamental examples including virtual springs and masses, these laws are analytically reduced to simple closed-form equations. The methodology is then generalized to the multivariable case. Several experiments are conducted including a 2-DOF coupled nonlinear VE example, and a scenario leading to a sequence of contacts with a virtual object. Besides the conceptual advantage, simulation and experimental results demonstrate some other advantages of the SC-LD over well-known time-domain passivity methods. These advantages include improved fidelity, simpler implementation, and less susceptibility to produce impulsive/chattering response.

In the field of minimally invasive surgery (MIS), trainers based on virtual reality provide a very useful, nondegradable, realistic training environment. The project of building this new type of trainers requires the development of new tools. In this paper, we describe a set of new measures that allow calculating the optimal position and orientation of haptic devices versus the virtual workspace of the application. We illustrate the use of these new tools applying them to a practical application.