Extended reality (XR)
Mixed reality passenger experience
Designing new interactions for future mixed reality devices that utilise hand tracking.
Extended reality (XR)
Mixed reality passenger experience
Designing new interactions for future mixed reality devices that utilise hand tracking.
Extended reality (XR)
Mixed reality passenger experience
Designing new interactions for future mixed reality devices that utilise hand tracking.
the situation
Ultraleap wanted to demonstrate the value its hand tracking and haptic technology could bring to mixed reality.
This was before other XR companies were close to having the same level of accuracy as Ultraleap's hand tracking.
the situation
Ultraleap wanted to demonstrate the value its hand tracking and haptic technology could bring to mixed reality.
This was before other XR companies were close to having the same level of accuracy as Ultraleap's hand tracking.
the situation
Ultraleap wanted to demonstrate the value its hand tracking and haptic technology could bring to mixed reality.
This was before other XR companies were close to having the same level of accuracy as Ultraleap's hand tracking.
old demos
Existing Leap Motion demos showcased the tech’s capabilities, but were mostly novel and didn't convey its usefulness for real-world scenarios effectively.
old demos
Existing Leap Motion demos showcased the tech’s capabilities, but were mostly novel and didn't convey its usefulness for real-world scenarios effectively.
old demos
Existing Leap Motion demos showcased the tech’s capabilities, but were mostly novel and didn't convey its usefulness for real-world scenarios effectively.
the mission
Create an XR app that demonstrates the benefits of using hand tracking tech for everyday tasks in mixed reality.
the team
I had a tremendously talented team to work with on this project. My contributions were primarily focused on the mid-air interactions themselves, and how the UI behaved in general. I also helped decide the narrative of the experience.
Leadership
Creative Lead / Product Owner
Senior Product Manager
Creative & UX
Creative / Technical Artist
2D / UI Artist
Senior UX Designer (me)
UX Researcher
Engineering
Senior Unity Engineer
the team
I had a tremendously talented team to work with on this project. My contributions were primarily focused on the mid-air interactions themselves, and how the UI behaved in general. I also helped decide the narrative of the experience.
Leadership
Creative Lead / Product Owner
Senior Product Manager
Creative & UX
Creative / Technical Artist
2D / UI Artist
Senior UX Designer (me)
UX Researcher
Engineering
Senior Unity Engineer
the team
I had a tremendously talented team to work with on this project. My contributions were primarily focused on the mid-air interactions themselves, and how the UI behaved in general. I also helped decide the narrative of the experience.
Leadership
Creative Lead / Product Owner
Senior Product Manager
Creative & UX
Creative / Technical Artist
2D / UI Artist
Senior UX Designer (me)
UX Researcher
Engineering
Senior Unity Engineer
writing the narrative
Having an effective narrative is essential for VR experiences with a beginning and an end. We got together and brainstormed what should be in the demo, ensuring we met the requirements of the brief which included it being used in a vehicle.
We decided to include elements of:
Using media apps
E-commerce
Mindfulness / relaxation
writing the narrative
Having an effective narrative is essential for VR experiences with a beginning and an end. We got together and brainstormed what should be in the demo, ensuring we met the requirements of the brief which included it being used in a vehicle.
We decided to include elements of:
Using media apps
E-commerce
Mindfulness / relaxation
writing the narrative
Having an effective narrative is essential for VR experiences with a beginning and an end. We got together and brainstormed what should be in the demo, ensuring we met the requirements of the brief which included it being used in a vehicle.
We decided to include elements of:
Using media apps
E-commerce
Mindfulness / relaxation
exploring existing interactions
The primary source of existing interactions to refer to were previous Leap Motion demos.
They allowed us to get a head start on what worked well.
exploring existing interactions
The primary source of existing interactions to refer to were previous Leap Motion demos.
They allowed us to get a head start on what worked well.
exploring existing interactions
The primary source of existing interactions to refer to were previous Leap Motion demos.
They allowed us to get a head start on what worked well.
designing the interactions
I took the best parts of existing hand interactions we had used before in isolation, and figured out how to use them together without them conflicting with each other when interacting with different types of virtual objects.
Providing effective visual, audio, and haptic feedback was equally important to ensure the user could interact with virtual objects properly.
designing the interactions
I took the best parts of existing hand interactions we had used before in isolation, and figured out how to use them together without them conflicting with each other when interacting with different types of virtual objects.
Providing effective visual, audio, and haptic feedback was equally important to ensure the user could interact with virtual objects properly.
designing the interactions
I took the best parts of existing hand interactions we had used before in isolation, and figured out how to use them together without them conflicting with each other when interacting with different types of virtual objects.
Providing effective visual, audio, and haptic feedback was equally important to ensure the user could interact with virtual objects properly.
prototypes
The technical artist and unity engineer created prototypes of interactions so we could all try them before deciding what to continue to with.
prototypes
The technical artist and unity engineer created prototypes of interactions so we could all try them before deciding what to continue to with.
prototypes
The technical artist and unity engineer created prototypes of interactions so we could all try them before deciding what to continue to with.
testing it myself
I regularly tested the demo to make sure all of the interactions, UI and feedback matched the design and worked well together.
The speaker-looking things emit ultrasound which provides the mid-air haptic feedback.
testing it myself
I regularly tested the demo to make sure all of the interactions, UI and feedback matched the design and worked well together.
The speaker-looking things emit ultrasound which provides the mid-air haptic feedback.
testing it myself
I regularly tested the demo to make sure all of the interactions, UI and feedback matched the design and worked well together.
The speaker-looking things emit ultrasound which provides the mid-air haptic feedback.
tutorials
At the start of the demo, we gave users interactive tutorials to complete so they could learn the interaction before starting.
The one shown here teaches users that "orbs" can be grabbed and dragged. An interaction pattern that's used throughout the demo.
tutorials
At the start of the demo, we gave users interactive tutorials to complete so they could learn the interaction before starting.
The one shown here teaches users that "orbs" can be grabbed and dragged. An interaction pattern that's used throughout the demo.
tutorials
At the start of the demo, we gave users interactive tutorials to complete so they could learn the interaction before starting.
The one shown here teaches users that "orbs" can be grabbed and dragged. An interaction pattern that's used throughout the demo.
internal user testing
I ran internal user testing with 6 non-technical employees that had no exposure to the project (most with little to no VR experience).
This was a very cheap way to find out about common usability issues before testing with external participants.
internal user testing
I ran internal user testing with 6 non-technical employees that had no exposure to the project (most with little to no VR experience).
This was a very cheap way to find out about common usability issues before testing with external participants.
internal user testing
I ran internal user testing with 6 non-technical employees that had no exposure to the project (most with little to no VR experience).
This was a very cheap way to find out about common usability issues before testing with external participants.
external user testing
Once we had made some refinements to the demo, we tested with 6 external participants with zero previous experience using VR.
We refined the demo and retested with an 6 additional external participants before finalising the design of the demo.
external user testing
Once we had made some refinements to the demo, we tested with 6 external participants with zero previous experience using VR.
We refined the demo and retested with an 6 additional external participants before finalising the design of the demo.
external user testing
Once we had made some refinements to the demo, we tested with 6 external participants with zero previous experience using VR.
We refined the demo and retested with an 6 additional external participants before finalising the design of the demo.
testing results methodology
The UX researcher devised a fantastic task completion recording system.
Each task within the experience was given its own score, and as participants completed a task, their ability to complete it was rated as:
Pass
Pass with vague prompt
Pass with detailed instruction
Fail
This made understanding the weaker areas of the demo very clear.
testing results methodology
The UX researcher devised a fantastic task completion recording system.
Each task within the experience was given its own score, and as participants completed a task, their ability to complete it was rated as:
Pass
Pass with vague prompt
Pass with detailed instruction
Fail
This made understanding the weaker areas of the demo very clear.
testing results methodology
The UX researcher devised a fantastic task completion recording system.
Each task within the experience was given its own score, and as participants completed a task, their ability to complete it was rated as:
Pass
Pass with vague prompt
Pass with detailed instruction
Fail
This made understanding the weaker areas of the demo very clear.
building a physical car for CES
Ultraleap wanted to construct a physical car for CES for event goers to sit in to make the mixed reality passenger experience more immersive.
Myself and others supervised its construction to ensure the ergonomics were OK.
building a physical car for CES
Ultraleap wanted to construct a physical car for CES for event goers to sit in to make the mixed reality passenger experience more immersive.
Myself and others supervised its construction to ensure the ergonomics were OK.
building a physical car for CES
Ultraleap wanted to construct a physical car for CES for event goers to sit in to make the mixed reality passenger experience more immersive.
Myself and others supervised its construction to ensure the ergonomics were OK.
Me checking ergonomics
Demo in use at CES
Me checking ergonomics
Demo in use at CES
Me checking ergonomics
Demo in use at CES
video of entire experience
The final version of what was referred to as the "Backseat Driver Demo" was complete. Watch the full end-to-end experience below.
You'll notice a voice actor was also hired to give users hints on how to progress.
video of entire experience
The final version of what was referred to as the "Backseat Driver Demo" was complete. Watch the full end-to-end experience below.
You'll notice a voice actor was also hired to give users hints on how to progress.
video of entire experience
The final version of what was referred to as the "Backseat Driver Demo" was complete. Watch the full end-to-end experience below.
You'll notice a voice actor was also hired to give users hints on how to progress.
project outcomes
The project was a huge success. And it was all thanks to how we worked as a team to create something truly unique and groundbreaking.
Delievered in time for CES 2020.
Continued use in events and client meetings all around the world.
The learnings from the demo's research and design contributed significantly to Ultraleap's XR design guidelines.
project outcomes
The project was a huge success. And it was all thanks to how we worked as a team to create something truly unique and groundbreaking.
Delievered in time for CES 2020.
Continued use in events and client meetings all around the world.
The learnings from the demo's research and design contributed significantly to Ultraleap's XR design guidelines.
project outcomes
The project was a huge success. And it was all thanks to how we worked as a team to create something truly unique and groundbreaking.
Delievered in time for CES 2020.
Continued use in events and client meetings all around the world.
The learnings from the demo's research and design contributed significantly to Ultraleap's XR design guidelines.
plus...
Widely shared on social media by XR enthusiasts.
Learnings from the demo helped us design headset manufacturer Lynx's main menu (released winter 2023).
plus...
Widely shared on social media by XR enthusiasts.
Learnings from the demo helped us design headset manufacturer Lynx's main menu (released winter 2023).
plus...
Widely shared on social media by XR enthusiasts.
Learnings from the demo helped us design headset manufacturer Lynx's main menu (released winter 2023).