2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) | 978-1-6654-1298-8/21/$31.00 ©2021 IEEE | DOI: 10.1109/ISMAR-Adjunct54149.2021.00098

Multi-scale Mixed Reality Collaboration for Digital Twin

Hyung-il Kim*
KAIST UVR Lab

Taehei Kim†
Motion Computing Lab,
KAIST

Eunhwa Song‡
KAIST UVR Lab
Woontack Woo||
KAIST UVR Lab

Seo Young Oh§
KAIST UVR Lab

Dooyoung Kim¶
KAIST UVR Lab

ABSTRACT
In this poster, we present a digital twin-based mixed reality system for remote collaboration with the size-scaling of the user and the space. The proposed system supports collaboration between an AR host user and a VR remote user by sharing a 3D digital twin of the AR host user. To enhance the coarse authoring of a shared digital twin environment, we provide a size scaling of the digital twin environment with the world-in-miniature view. Also, we enable scaling the size of the VR user’s avatar to enhance both coarse (size-up) and fine-grained (size-down) authoring of the digital twin environment. We describe the system setup, input methods, and interaction methods for scaling space and user.
Index Terms: Human-centered computing—Human computer interaction (HCI)—Interaction paradigms—Mixed / augmented reality; Human-centered computing—Human computer interaction (HCI)—Interaction paradigms—Collaborative interaction
1 INTRODUCTION
Combining the advantage of both the Augmented Reality (AR) to interact with real world and Virtual Reality (VR) to provide immersive experience, Mixed Reality (MR) creates a more realistic virtual experience. While MR can be applied and utilized in various areas, it can be effective when assisting collaboration between two users at a distance. In this case, a local host user with an AR device will summon a remote VR user in his space, and the VR user has full access to the AR user’s space in virtual reality format. While collaborative system engaging either one of AR or VR users has been in high attention, collaboration in MR environment engaging both has been studied relatively little. With technical improvement of digital twin (DT) technology, MR collaboration can be made for a more immersive collaborative experience.
We propose a digital twin-based novel MR system for remote collaboration bridging the gap between AR and VR. Our work introduces size-scaling of two elements to enhance collaboration in a mixed reality environment; space and user. In real environment such as floor planning, interior design, and city planning, we face constraints; Moving and measuring massive objects in person (in our specific scenario eg. furniture) and seeing the space in a larger context. Approach using our multi-scale space solves this physical issue. Both the AR and VR user can observe and manipulate the AR user’s space, operating virtual objects in a miniature size. This reduces the burden of directly moving objects in a physical space.
Scaling the size of the space in our study was acquired from the concept of the World In Miniature (WIM) to see the whole space at
*e-mail: hyungil@kaist.ac.kr †e-mail: hayleyy321@kaist.ac.kr ‡e-mail: eunhwa99@kaist.ac.kr §e-mail: seoyoung.oh@kaist.ac.kr ¶e-mail: dooyoung.kim@kaist.ac.kr ||e-mail: wwoo@kaist.ac.kr

Figure 1: Our proposed collaborative interaction between an AR host user and a VR user. a) Normal-scale collaboration b) Multi-scale space with an miniature space c) Giant user d) Miniature user
one glance in VR, which was first introduced by Stoakey et al. [3]. More recent works replicated the real world for minute manipulation of objects in AR by creating a photo-realistic composite image to allow the user to see the space with a bird’s eye view in MR. [4, 5] Applied from these previous studies, our work uses the concept of WIM to manipulate the object in small scale and generates a digital twin scanned from the real environment for remote collaboration in MR.
The concept of scaling the size of the user in MR collaboration is explored in recent studies, by utilizing the miniaturized avatar [2], the giant avatar [1]. Our multi-scale user function allows the AR user to change the size of the VR user which enables the VR user to see the space from a different viewpoint. For the detailed manipulation in a small scale, the AR user changes the VR user’s size into miniature so that the VR user can author the AR object carefully. On the other hand, by enlarging the size of the VR user into giant, the VR user can view the whole environment at glance and visually comprehend the space, solving the difficulties of perceiving the relative locations and distances of surrounding objects in a life scale space.
2 SYSTEM DESIGN AND IMPLEMENTATION
2.1 Overview
Our proposed system is a multi-user mixed-reality system with a shared digital twin environment. A host user wears an AR headset and summons remote VR user to his own host space. A remote VR user wears a VR headset, and is summoned to the digital twin of the host space. Both the AR user and the VR user can see each other as a virtual avatar in their shared space. The AR user sees the remote VR user’s avatar in his own physical space, and the VR user sees the

978-1-6654-1298-8/21/$31.00 ©2021 IEEE

435

DOI 10.1109/ISMAR-Adjunct54149.2021.00098

Authorized licensed use limited to: Technische Informationsbibliothek (TIB). Downloaded on February 12,2025 at 09:41:19 UTC from IEEE Xplore. Restrictions apply.

Figure 2: System diagram of proposed digital twin system
AR host user’s avatar in the virtual digital twin space.
2.2 Digital Twin Management
To enable multi-platform mixed reality collaboration using digital twin, we designed following digital twin system. (Figure 2) Our proposed digital twin system consists of host user data, remote user data, and shared space and objects data. For the AR host user data, we manage the 6DOF head pose and the 6DOF hand pose of both hands. For the remote VR user data, we manage the 6DOF head pose, 6DOF hand pose, and additional 1DOF scale data. Space data consists of the scanned space geometry, the scale of the miniature space, and the 6DOF poses of the objects. Every data in the digital twin is synchronized with both collaborators in real-time.
For real-time pose synchronization between real and virtual environment, finding local coordinate in relative to the real host environment is crucial. To find relative pose in sync with digital twin, the system need to detect and to track the host environment. Camerabased image marker detection, finding spatial anchor, or outside-in tracking can solve this problem.
2.3 Interaction Methods
Multi-scale Space Both the AR host user and the remote VR user can summon the miniature of the digital twin by tapping a miniature summon button. After summoning the miniature digital twin, they can place the miniature, or can manipulate the scale of the miniature in their space using their hands or controllers. (Figure 1b) Miniature space object has two modes - Placement mode and Viewing Mode. In the Placement mode, both users can manipulate the miniature in their shared virtual space and the users can’t manipulate the objects inside the miniature. In the Viewing mode, the pose of the miniature is fixed in the environment, and the users can manipulate the virtual objects inside the miniature. The objects in the original digital twin and miniature are synchronized.
Multi-scale User Since the AR user can’t manipulate his own size, we enable manipulation of the VR user’s size in the digital twin. (Figure 1c, 1d) To manipulate the scale of the VR user, we propose two modes - Collaboration mode and User Manipulation mode. In the Collaboration mode, the size of the VR user can’t be adjusted and both users can manipulate their shared virtual objects. In the User Manipulation mode, both users can manipulate the size of the VR user. AR user can manipulate the remote VR user’s scale, and can place remote VR user to any location. The VR user can manipulate his own scale, and can navigate the digital twin space by walking or teleporting. When the position and the size of the VR user is manipulated, the VR user teleports to the designated position with specified size.

Collaboration Cues We share both user’s head and hand pose for the collaboration cues. Shared head and hand poses are reflected as an avatar in their real/virtual space. Users can know where the collaborator is looking at, which objects the collaborator is interacting with, or which gesture the collaborator is doing by sharing those cues. Also, our system shares both users’ audio to enable basic verbal communication.
2.4 Setup and Implementation
For the host user (AR), we used Microsoft HoloLens 2 for the AR headset. To scan the 3D model of the host environment, we used Apple iPhone 12 Pro. For the remote VR side, we used Oculus Quest 2 for the VR headset. Both devices are connected to the 5GHz wireless network.
For the software, we used Unity 2019.4.25f1 to develop a prototype interior design application for both AR and VR side. To enable basic manipulation and system input in MR, we used MRTK 2.6.1. To align the virtual digital twin and real host environment, we used Azure Spatial Anchor to find and locate the spatial anchor, then synchronized the coordinate system of the real and virtual environment. For networking, we used Photon Unity Networking to share both users’ 6DOF head pose, hand pose for the both hands, and the pose of shared virtual objects.
3 CONCLUSION
In this poster, we presented a digital twin-based mixed-reality remote collaboration system with multi-scale space and user. For the collaborative virtual object manipulation scenario like collaborative interior design, we proposed multi-scale space and multi-scale user. Using multi-scale space, we provide world-in-miniature of the host digital twin for quick and coarse manipulation of the shared virtual objects, and bird-eye view of their own shared space. By manipulating the size of the remote VR user, we can enhance both coarse and fine-grained authoring of the digital twin environment.
ACKNOWLEDGMENTS
This work was supported by Korea Institute for Advancement of Technology(KIAT) grant funded by the Korea Government(MOTIE) (P0012746, The Competency Development Program for Industry Specialist)
REFERENCES
[1] T. Piumsomboon, G. A. Lee, B. Ens, B. H. Thomas, and M. Billinghurst. Superman vs giant: A study on spatial perception for a multi-scale mixed reality flying telepresence interface. IEEE Transactions on Visualization and Computer Graphics, 24(11):2974–2982, 2018. doi: 10.1109/TVCG. 2018.2868594
[2] T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H. Thomas, and M. Billinghurst. Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, p. 1–13. Association for Computing Machinery, New York, NY, USA, 2018. doi: 10.1145/3173574.3173620
[3] R. Stoakley, M. J. Conway, and R. Pausch. Virtual reality on a wim: Interactive worlds in miniature. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’95, p. 265–272. ACM Press/Addison-Wesley Publishing Co., USA, 1995. doi: 10.1145/223904 .223938
[4] B. Thoravi Kumaravel, F. Anderson, G. Fitzmaurice, B. Hartmann, and T. Grossman. Loki: Facilitating remote instruction of physical tasks using bi-directional mixed-reality telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, UIST ’19, p. 161–174. Association for Computing Machinery, New York, NY, USA, 2019. doi: 10.1145/3332165.3347872
[5] Z. Wang, C. Nguyen, P. Asente, and J. Dorsey. DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments. Association for Computing Machinery, New York, NY, USA, 2021.

436 Authorized licensed use limited to: Technische Informationsbibliothek (TIB). Downloaded on February 12,2025 at 09:41:19 UTC from IEEE Xplore. Restrictions apply.