1 of 43
DownloadGo Live

Mobile VR

Published on Nov 18, 2015

VR TEASER

PRESENTATION OUTLINE

Mobile VR ON ANDrOID

droidcon montreal 2015
Photo by uBookworm

hello Droidcon!

  • Francois Legare
  • Laurence de Villers
Hi there!

We are two Bell Canada mobile gurus that do mobile applications for Bell Digital Solution a group of professional services that deliver mobile and web solutions for our clients.

For more information:
https://business.bell.ca/shop/enterprise/mobile-application-development-ser...

We are also passionate about new trends and today we will discuss about mobile VR!
Photo by Michiel2005

Definition Of VR

"simulate physical presence in places in the real world or imagined worlds"
What is VR? Well this sentence sumarize quite well the actual concensus.

Since it's the morning and we don't want to flood you with code just yet, let's start slow and
review quickly review the evolution of VR.

This is base on a interesting article found here:

http://design.osu.edu/carlson/history/lesson17.html

Also wikipedia summarize it well : http://en.wikipedia.org/wiki/Virtual_reality
Photo by i k o

1860

First glimpse of virtual presence at least according to wikipedia

1860's Baldassare Peruzzi painted walls of in the Villa Farnesina, Rome, Itally to create the Sala delle Prospettive (Room of perspective)

Since LED screen was not yet available he used the next best thing "Paint"


Source: http://en.wikipedia.org/wiki/Baldassare_Peruzzi

1930

1930 Autor Stanley G. Weinbaum wrote a short story called Pygmalion's Spectacles the story speak about a way of capturing scene with super cool sci-fi liquids that capture glimpse of scenes and replay them using spectacles. These are still more advance than what we have today since it was also capturing the 5 senses...

You can read the short story for free here:
http://gutenberg.net.au/ebooks06/0607251.txt

Source: http://fr.wikipedia.org/wiki/Stanley_G._Weinbaum
Photo by *CQ*

1962

Sensorama was a real immersive experience to emulate a bike ride in brooklin. It was able to display stereoscopic 3-D images in a wide-angle view, provide body tilting, supply stereo sound, and also had tracks for wind and aromas to be triggered during the film.

Oddly enough in hindsight, Heilig was unable to obtain financial backing for his visions and patents, and so the Sensorama work was halted. Today, it remains primarily a curiosity in the history of Virtual Reality.

Source:

http://en.wikipedia.org/wiki/Sensorama

1968

1968 Ivan Sutherland, Create the first Augmented Reality devices for the USAF. It was displaying augmented reality, putting digital wireframes over actual room. Yet this prototype was laying fondation of actual F35 helmet and modern HUD.


Source:
http://en.wikipedia.org/wiki/The_Sword_of_Damocles_(virtual_reality)

1991

It seem SEGA was the first to propose a VR prototype and not nintendo! The perception is probably due to the fact that it was never released to the public unlike the Nintendo Virtual Boy released in 1995

Source: http://en.wikipedia.org/wiki/Sega_VR

1991

The same year, a arcade rig called Virtuality hit the market, with a price tag of 73k$US this was clearly not aimed to be mass produced.

Source: http://en.wikipedia.org/wiki/Virtuality_(gaming)

1995

First consumer grade VR-ish device, Nintendo sadly wasn't able to seduce the masses. In order to reduce price to the maximum monochrome LED was used and yet it was still pretty expensive for the experience it offered.

We needed to wait near 20 years to see practical new solution appear on the market.

today dedicated vr headset

  • Oculus Rift
  • Sony Morpheus
  • Valve Vive
  • Microsoft Hololens
  • VRVANA - Totem (Montreal)
Lets see quickly the dedicated VR headsets:

- Oculus Rift, probably the most well know device, use accelerometers and IR Camera to track devices in space. Very clever IR LED flashing allow the camera to estimate the exact position of the helmet in space.

- Valve Vive with fraking lasers seems very promising !

- Sony Morpheus for PS4, not much info available

- Holo lense, more augmented reality than VR but seems promising. Air mouse gesture seems a bit ackward, hope they will think a better way of tackling with the untangible

- Montreal startup - Totem from VRVANA https://www.vrvana.com/
for montrealer in the room you can meet them at Montreal VR Meetup monthly


Photo by wuestenigel

... mobile?

2014 was clearly a new wave of new devices, lets review what is the current market state in term of these devices.

Photo by szczym

Cheap, hand held, not good for long experience

Google card board clones
To be used hand held, more or less unusable with head straps without a lot of modifications and padding. Try not to turn it into an HMD. There are some advantages of using lenses with larger diameter and better build quality, but the differences between the different Cardboard versions are minor, so the cheapest one will usually do unless you tend to spend a lot of time with it. Group includes Cardboard clones made of other materials, some featuring head straps without padding.

Source for upcoming slide:

http://www.reddit.com/r/GoogleCardboard/comments/2vabmr/lets_have_a_review_...

plastic, head strap, adjustable lens, good for watching 3d movies

Improved plastic cardboard
Mostly unusable for VR unless you alter the lenses and the case. Several (ColorCross[1] , Ritech 3D[2] , Storm I[3] , Storm II[4] , Xiaozhai[5] , UnicornVR[6] ) can be bought cheaply, most of them are reasonable comfortable and they have features like adjustable lens-lens and lens-screen distance, especially useful with very small or very large screens. Unfortunately they are intended for 3D movie watching first and all have a horrible FoV even with larger screens, which breaks immersion. I've been tinkering a lot with Cardboard and plastic 3D goggle modifications with single and stacked lens configurations, and while I can get a rather impressive FoV out of these for under USD 25 total, it is a lot of work and you won't get near the image of high quality lenses matched to a specific screen.

Some of them come bundle with a bluetooth controller

higher quality optics, better FOV, under 100$

VR mobile goggles
Technically very similar to group B, but actually intended for VR.

This primarily means the lenses have magnification similar to Cardboard, are of (slightly) better quality and the phone is held closer to the lenses. These (Durovis Dive[7] , Zeiss VR ONE[8] , Homido[9] , XG HMD[10] , VRizzmo[11] ) are not mass produced in China, some of them come from Kickstarter campaigns with small production runs, so they cost significantly more.
If you need an HMD with headstraps, proper lenses and large FoV without having to mostly build it yourself, these are what you can currently get, but you still get pretty much the same VR experience as with a cheap Cardboard. This may be worth it if you are streaming a lot of games from a PC, but there aren't many sufficiently long VR experiences that would require something like this, most apps now target the handheld Cardboard. The best may be the Zeiss VR ONE due to optics, trays to align the phone and their SDK that optimizes for this, but it only works with a few phones and I don't know any apps that actually use the SDK.

optimized for vr, advance touch interface, be$t experience on mobile so far

samsung gear vr
Best VR Experience so far for mobile, its the mobile alternative of the oculus.


But at 200$ for the case, and 799$ for the Note 4 (299 with a 2 yrs contract at Bell wink wink) ... not the cheapest VR option but clearly the best VR exp. on mobile so far.

Multiple features:

- OS optimized for maximum FPS
- Touchpad intregrated with the helmet
- High quality overall and well designed software experience

Still technically a prototype ("innovator edition")

Power of immersion

  • Depth of perception
  • 360 scenes + 3d sound
  • Vertigo
  • Scare factor x1000
  • Lots of possible application!
The better the quality of the device the better will be the illusion of immersion.

There is a lot of power associated with this immersion, just think of the possibility for a second. VR bring a new subset of experience to the software piece.

Depth of perception, 360 view with head tracking, possibility of creating virtual Vertigo! I'm not even sure I would be game to play horror game with VR headset on...

Of course games is one of the first obvious application for VR but think for a second all the opportunity that VR bring to the table.

Medecine, try to fix phobia
Architecture, feel a room prior constructing it. Retail, walk in a virtual store. Even Bjork is about to release a music clip VR friendly, I presume the CD will be based on a cardboard !

Challenge of Ergonomic

  • Comfort
  • Can't walk (yet)
  • User inputs, Controllers
  • Interface design (UX)
We see lots of possible application but there is still lots of challenges. One of the biggest is making VR accessible to humans!


- Comfort: can be limited especially for lower quality rig (ie. use your hand, lens not adapted to your sight, greasy cardboard)

- Can't walk: VR when well done can be very convincing, first user reflex often be to start walking to move in the scene. This is dangerous and will soon be fixed with tech. like Valve Vive and most probably using phone similar to the one found in the Tango project

- User inputs: when we have the helmet on the eye, we can hardly interact with the real environment. Most practical way so far is bluetooth controllers or the clever magnet on the cardboard. Samsung Gear come with it's own touchpad.
A more fancy approach is the Leap Motion:
https://vimeo.com/104854975


- UX for VR need to be rethink with these new challenges. Current trend is to put a visor at center focal point that will act as a selector if left over a actionable item long enough.

INDUCE Presence

  • >60hz, 90+ Hz
  • > 80 degree F.O.V
  • > 1080p - avoid screen door effect
  • Optical calibration
  • Fast and accurate tracking
  • Reduce spacial jitter
To get a really good presence feeling we need to design our software to meet or be close of these specification.

Photo by ecstaticist

reduce Motion sickness on MOBILE VR

  • Design exp. for mobile
  • Selective Stereoscopy
  • Virtual nose
  • Lowest vertice for wow
How we reach this ? Be designing more specifically for Mobile VR and not parsing regular VR app to mobile. A bit like web experience, we dont take a regular website and try to fit this in mobile phone.

Here are few optimisation trick stolen from VR Philly :
https://www.youtube.com/watch?v=5WkRAcjjGk8

For an simple example, if you allow the user to use head motion but don't allow the visual to follow the real motion movement, it can get confusing for the mind and can create motion sickness.

If you want to see what is the feeling try VR scene application.
Photo by parkydoodles

Capturing the real thing

quick overview of methods to catpure content
These tricks was mainly for 3D rendered scene, now let's see how you can capture our world to put in your own experience withtout making a 3D scene from scratch.

Usually this is mostly done using a 3D scene with a skybox or spherebox and the magic happend. This is the theory in practice making this possible with high FPS on most of the phone can be a bit challenging.



our humble attemp

This 3D cardboard camera was done using 2x gopro, some scotch tape and a lot of naivety.

Synch the two footage. Let's say this was also a time waster, pro tip make a lound bang when you start your footage and try to synch with this.

We are not cinematographer, so here is at least to error you can avoid :)



2.5 inch or ~ 63.5 mm

First challenge, space between eyes. We didn't bother too much about the Interpupillary distance. The space between the two camera.

The result was a major see sickness when we played back the video. So make sure if you put the camera distance around 60 mm ishhh distance.

To know more:

http://www.dashwood3d.com/blog/beginners-guide-to-shooting-stereoscopic-3d/

images & Videos

  • 360 Camera : ricoh theta
  • 3D Camera 180+ : Hero360
  • Full-3d-360-6k : HypeVR
If you are really serious about capturing content buy the real stuff you will save a lot of time.

For picture, the Ricoh Theta was not bad, compared to Photosphere it take picture in one snap and give a similar result than streetview

For 3D footage, Hero360 and GoPro have special case that allow capturing 3D and even 360 3D footage. It require the advance gopro for synchronisation.

For unlimited budget, the HypeVR is pretty much the top of the line. Price tag must be around like a Villa near a beach.

Free Space Omni-Binaural Microphone

For sound this seems to do the trick around 3k$...


Display 360 content

Once you have captured the content there is multiple way to display it in VR

For none 3D 360 pictures, the easiest way to simply map the picture to a cube or a sphere and render the scene in the middle of it. This off course require to adapt the picture to fit the mapping selected.

There is a couple of panorama formats existing, in the next slides we will see project that attempt to reproduce the full cube and sphere using the SDKs.

Approach 1

webgl

Ok now you that you have your content how do we actually build the app???


Well the first approach is probably the easiest one if you are new to the 3D programming world.

It use WEBGL for rendering the scene. Plain html can do VR.


- Ressources
----------------
mozvr.com
vr.chromexperiments.com
http://threejs.org/examples/ (#vr)

where to start

Head to this URL, you will find a couple of sample experience. At Wear hack montreal we where able to make a photosphere viewer web based in under couple hours.

- Sample code easy to understand

- You can reverse engineers the other sample provided on the chrome experiment


main components

  • Three.js (framework)
  • StereoEffect.js
  • DeviceOrientationControls.js
  • The rest is basic scene 3D rendering
- StereoEffect.js

Off-axis stereoscopic, Simply render the scene for two eyes

- DeviceOrientationControls.js

Manage to get the orientation of the devices

PRO: It's web!

- Nothing to install just go the url and enjoy the experience. No need for app store, compiling stuff. Simple URL... Can't be easier assuming the visitor have a decent mobile web browser that support Web GL

- Three.js is easy to use vs OpenGL

- Free and open source :)

- Multiplatform but I wont mention iOS at Droidcon wouldn't I?


Photo by striatic

CON: It's web!

- Long download time prior anything move, need to build custom loader if you want a good user experience.

- Performance is limited (at the moment) - FPS hit vs dedicated approach in openGL. This should be kept for simple stuff.

- No lens distortion, when tried, probably just a question of time

Approach 2

android google cardboard sdk
Second approach the Android SDK

where to start

- A bit more complex code, will require some OpenGL base to be fully able to enjoy the sample

- The project create a scene and show you the basics, handle click, render basic form in openGL, couple of shaders, these guy worked hard to get a very good starting point




main components

  • CardboardActivity
  • CardboardView
  • onDrawEye
  • Distortion
  • The rest is basic surfaceView rending with OpenGL
- Cardboard acitvity:

add basic tools, especially the
onCardboardTrigger () that detect magnet "click"

- CardboardView

Convenience extension of GLSurfaceView that can be used for VR rendering.

Designed to work in full screen mode with a landscape or reverse landscape orientation.

This view can be used as a normal GLSurfaceView by implementing one of its rendering interfaces:

CardboardView.StereoRenderer: abstracts all stereoscopic rendering details from the renderer.


The view allows switching from VR mode to normal rendering mode in stereo renderers at any time by calling the setVRModeEnabled method.

- onDrawEye

Like the regular onDraw but this time one for each eye

- Distortion

If for some reasons you have a other type of lens than the one the original cardboard (45mm focal) you can update the distortion optics using this


PRO: It's openGL!

OpenGL stack, plus these goodies:

- Lens distortion correction.
- Head tracking.
- 3D calibration.
- Side-by-side rendering.
- Stereo geometry configuration.
- User input event handling.

Overall performance are very good, still require a recent phone to plenty enjoy the experience

- Free and open source :)
Photo by striatic

CON: It's openGL!

- Unless you have good experience in OpenGL or plan to invest sometime in learning openGL it's a bit tedious to grasp all the concept initially.

- From scratch, not a game engine already built for you

- Debuging is painful - compile see if it does the job

Again we are not game developers maybe we missed some super cool framework that does all this for you ;)

Approach 3

android unity cardboard sdk
Unity 3D SDK

Requirement :
Unity version 4.5+
Unity Pro :
Enable the distortion-correction
Cardboard GUI features.

where to start

Same stating point, but click unity3D sdk

Easy step to Adapt or Create a VR Application: ( Demontration with unity )
Put the camera under Head under CardboardMain
Add a Head that contain the CardboardHead
Add StereoController script to Main Camera
Update Stereo Cameras
Add GazeInputScript to Event system
Add a script on the object to be trigger

main components

  • Scripts
  • Prefab
CardboardMain

This prefab contains a full stereo rig, with a camera tagged MainCamera, and an instance of the Cardboard script for controlling VR Mode.

CardboardHead
Contain the cardboardhead script

CardboardAdapter
For an existing project with a Camera that is already fully populated with control scripts, it may be best to instead add aCardboardAdapter prefab as a child. This places the
CardboardHead under the main Camera, so that only the stereo cameras are affected by user head motion. Then, finish the rig by executing the Update Stereo Cameras command, which adds the StereoController to the Camera itself.

PRO: It's unity!

No need to tackle on OpenGL matrix, the tool make this for you. You only have to build your scene and apply the required game logic.

Performances between the pure OpenGL handcrafted with love and the WebGL

Multi OS in theory but the Android SDK is obviously not aimed at iOS :P

Already an game engine
Retroaction with the play mode
Scene where you can place manually your element
Simulate a user's head movement in Unity play mode ( with alt/control)
Simulating magnet "clicks"
Simulating distortion correction( with Unity Pro Only)


Photo by striatic

CON: It's unity!

Not open source and free, but the community edition is free and pretty much technically vendor lock in, but hey at least you didn't had to undust your matrix maths to draw a simple cube in face of a camera or figure out how shader work :)

A little more slow for loading the application
Unity splash screen loading on your app ( if you do not have Unity Pro)
Price :
Unity Pro = 1500$ or 75$ per month
Android Pro = 1500$ or 75$ per month
3000$

Di$tortion is also a paid feature

Q&A

- Cardboard acitvity:

add basic tools, especially the
onCardboardTrigger () that detect magnet "click"

- CardboardView

Convenience extension of GLSurfaceView that can be used for VR rendering.

Designed to work in full screen mode with a landscape or reverse landscape orientation.

This view can be used as a normal GLSurfaceView by implementing one of its rendering interfaces:

CardboardView.StereoRenderer: abstracts all stereoscopic rendering details from the renderer.
CardboardView.Renderer: for complex engines that need to handle all stereo rendering details by themselves.
The CardboardView.StereoRenderer interface is recommended for all applications that can make use of it, while the CardboardView.Renderer interface is discouraged and should only be used if really needed.

The view allows switching from VR mode to normal rendering mode in stereo renderers at any time by calling the setVRModeEnabled method.

- onDrawEye

Like the regular onDraw but this time one for each eye

- Distortion

If for some reasons you have a other type of lens than the one the original cardboard (45mm focal) you can update the distortion optics using this


THE END
Francois Legare : flegare@gmail.com
Laurence de Villers : laurence.de.villers.prog@gmail.com
(cc 2015)
http://goo.gl/U8TirJ

Deck available here: http://goo.gl/U8TirJ