Projection Mapping combined with Mixed Reality Experience

For this project I worked on setting up the projectors, aligning the digital world with the real world and connecting the MR headsets with the projectors.

I worked in Unity and found a package which let us easily connect cameras from the Unity scene to the projectors, so they could show the parts of the scene we wanted to be visible in real life. Then it was running some tests locally checking if it was possible by creating the stage in the scene and playing it in the headset.

After teh test I started working on connecting multiple headset to the same computer. The setup of the stage is multiple projectors controlled by a PC, while MR headsets connect wireless. I added some networking packages to Unity and made it so that the PC acted as a server host and controlled the entire experience. All the users headsets are synced which means they will all see the same thing and can interact with objects in the scene that updates for them all.

Important: Assets seen in the experience are not made by us and are assets bought from the Unity Assets Store.

This is a school project where we are supposed to explore unique ways of using technologies. My group wanted to study the possibilities of combining a virtual stage or a projection mapped room with MR headset, so that we could bring objects out of the wall and into the room with the observers. With a full visuals of the real world the users can see the pictures shown on the walls and as well whatever objects are going through.

We were working on this following the school schedule, until our teacher approached us and wanted us to work full-time for two weeks on this project to showcase it to various stakeholders. The team agreed to the work and we started producing a scene for a Proof of Concept to show them. We went through multiple ideas of what we could show and landed on an underwater experience, with a turtle floating back and forth with fish swimming around.

We wanted to add some interaction that would happen with the turtle when it was in the space with the users. So we added a feature where whenever a user touched the turtle it would stop and look around for a few seconds before continuing on its journey.

We received a lot of positive feedback from teachers, stakeholders and fellow students who have tested the experience we created.

Important: Assets seen in the experience are not made by us and are assets bought from the assets store.

This project was a collaborative effort with valuable contributions from the following team members:

• Tobias Hol Fehn - Role: Developer
• Jacques Parraga-Michitsch - Role: Game & Sound Designer