Tire Explorer, developed for our client Bridgestone, is targeted for retail and also for training scenarios. The app provides information about tire components and how those are built. It does so by recognizing and tracking a real-world tire and augmenting it with virtual components (watch a video of the app experience).

Valorem's aquarium booth at Build 2017

Building the aquarium booth

We were honored to be invited by Microsoft to have a dedicated HoloLens demo booth called aquarium at Build. Each aquarium consisted of a demo room with a spectator view camera installed and a tech room where the spectator view PC was monitored and HoloLens devices were cleaned and charged. The demo room also featured a big window and a screen mounted outside the booth that showed the spectator view camera stream.

Spectator view camera

A look behind the scenes in the tech room

Valorem’s Immersive Experiences team worked very hard, and long hours went into the development of Tire Explorer, but it was worth it and a great experience to show our work to the world. Almost 500 Build attendees had the chance to try the app and share their overall very positive feedback.

NEW, NEVER-BEFORE-SEEN TECHNOLOGY

Tire Explorer features new, key technical aspects: high-end rendering, non-fiducial marker tracking, and high-resolution HoloBeam.

High-end Rendering

Achieving a high-end product visualization while still maintaining real-time framerate is very challenging with a mobile device like the HoloLens and its limited computing power. However, we reached this goal with clever modelling techniques that keep enough detail without having to use millions of polygons.

Custom shaders allowed for high-end product visualization

What allowed us to achieve extraordinary results were custom shaders, which define how a material is rendered. Complex shaders look amazing but do not result in high-end HoloLens rendering; only custom, specific shaders can create quality results with real-time framerate.

We developed those shaders with the Unity Asset Shader Forge, a node-based shader editing tool. Although Shader Forge generates good enough shader code, it is it not optimized to its maximum capacity, so we tuned the shader code manually by hand to ensure maximum performance while displaying amazing visuals.

Non-fiducial Marker Tracking

The Tire Explorer demo uses a real tire that is recognized by advanced computer vision based on a 3D model of the tire, so non-fiducial markers like AR Tags are needed and only the real tire is tracked. We achieved this by using the new Vuforia Model Targets support. We are proud to say that we were the first to showcase this new tracking technology for UWP HoloLens in public for the first time ever. The new Model Targets are available on an invite-only basis for now. You can reach out to Vuforia to get access if accepted.

Tire Explorer tracking results are amazing

We are very pleased with the tracking results, which are nothing less than amazing when experienced firsthand. To perfect it, some tweaking of the 3D model was needed to get it below 100k polygons. We also had to tweak the lighting setup to improve the contrast on the tire treads and make the recognition and tracking flawless.

High-resolution HoloBeam

Once the tire is tracked, the app allows users to gaze and air-tap at components to see and hear explanations about each tire component. For this, we don’t just use text or Spatial Sound, we actually use our HoloBeam technology developed by Valorem. HoloBeam uses a 3D camera like the Kinect and its depth and color frames, which are streamed in real-time stream into a Mixed Reality device like the HoloLens. We have been working on this technology since last year, and you can see in this video from November 2016.

Recently, we were able to get full HD color mapping working and running smoothly while maintaining real-time framerates. In order to achieve this we had to develop our custom-depth encoding and packaging algorithm. If one would use normal image or video encoding and apply that to depth data, lots of artifacts would appear, which is not acceptable. Additionally, lots of C++ native code was written to make the encoding and decoding into graphic memory textures as fast as possible. Another key item of HoloBeam is the custom shader that renders the depth and color data. The Geometry shader emits thousands of quads, which use a special color-depth registration and smart math to get it right.

A hologram explains tire components and how they're built in Tire Explorer, now in full HD

HoloBeam allows the real-time streaming and rendering of the RGB-D camera, but we can also record sequences/clips like we did for Tire Explorer where Valorem’s own Rod Shelton was recorded and can now be seen as HoloRod, the tire expert explaining tire components.

Jesse Havens, Valorem's UX Lead, designed the interiors of the aquarium booth

TURNING CONTENT INTO AMAZING EXPERIENCES

Tire Explorer allows expert teams to take mundane content (a tire!) and turn it into an exciting, interactive experience. By allowing the user to explore the hidden features of the object, we can create a memorable purchase experience. Currently, we are polishing the app even further and have plans to deploy it and bring it to market soon.