Reality Labs Research
Designing interaction languages for emerging hardware.
For two years at Meta Reality Labs Research, I built interactive proofs-of-concept to unblock researchers, empower designers and inform organizational decision-making. Much of this work remains confidential; I’ve described what I can below.
With hardware ranging from microcontroller benchtop rigs to experimental prototype headsets, my job was to build enough scaffolding to make the system designable, then use it to answer interaction questions. I built reusable Unity packages so other designers could bootstrap from my progress, demoed frequently to testers and stakeholders, and iterated quickly based on feedback.
Taking an experimental display from test patterns to wearable prototype
I joined a team of research scientists, engineers and prototypers evaluating an experimental display. In the beginning, the display ran an Arduino driver written for research simplicity rather than performance; it could show test patterns and static images, but nothing else.
Device bringup to unblock animation and interactivity
I rewrote the driver in Python, changing how frame memory was allocated to achieve 15× faster updates on the same hardware, which unblocked animation and real-time interaction. I wrote a Unity SDK bridge to bring the display into my team’s familiar prototyping environment, unlocking 6DoF world-locking, hand-tracking and camera access. I built simple, quick demos highlighting use cases for each of these capabilities. Within a few weeks the display went from a benchtop rig to a wearable prototype the team could use for design exploration.
Finding the interaction language
The point of the technical work was to unlock the features we’d need to answer key design questions, like:
- How should navigation be represented?
- How could the display communicate notifications or alerts?
- How should a physical object or location be indicated?
- Which patterns were intuitively intelligible, and which introduced visual discomfort?
We worked backwards from product use cases to the capabilities each would need, then tested various approaches and gathered feedback to downselect the most intuitive.
A particular tradeoff was comfort vs legibility - we exploited a fact about human peripheral vision to achieve a highly visible signal, but the same effect was nauseating for some people. I targeted this feedback by producing a simple demo that allowed me to tweak parameters and switch through several approaches. After testing with roughly a dozen volunteers, we found a combination of settings that wassufficient to reduce nausea while preserving legibility.
Visualizing product fit through a product-vision video
As a frequent user and advocate for the technology, I wanted to share a vision for how it might be productized and marketed. In addition to an immersive interactive symposium demo, I produced a product-vision video that portrayed the display as something sleek, useful and desirable. A redacted version of the video is included below (the original showcased the display’s capabilities, which remain confidential).
Prototyping platform & product decision
The Unity SDK I built was used by other VR developers and designers to build further symposium demos after I moved on. I also produced a product-vision video connecting the research to a coherent future experience.
The prototype was demoed to senior leadership at multiple internal symposiums and demo days; leadership decided not to productize the display. That decision was our mission accomplished - the team took an unproven display from test patterns to a wearable prototype with an intuitive interaction language, making an informed “no” possible.
Precise navigation with spatial audio alone
With an audio research scientist, I tested whether a person could navigate a real space safely and precisely using nothing but a spatially positioned audio beacon. The technique is described in U.S. Patent Application US-20260089459-A1, on which I’m a named co-inventor.
I started by building a Unity sandbox that exposed the scientist’s parameters (like ‘decision point falloff radius’) as interactive gizmos. This let us sit together and tune the effect in real time.
Once the guiding principles were locked in, I scanned a large section of the office and used spatial anchors to pin the virtual map to it, which let us borrow Unity’s navmesh pathing and audio occlusion to simulate future device capabilities.
Instilling confidence with clear UX
Technically, the cue pointed the right direction. But without visual confirmation, people became overly cautious and walked slowly. Confidence became a design goal. I made turns sharper and more perceptible - in practice, less lead time meant that signal movement was clearer and easier to follow. I also removed the separate wrong-way cue - it was disruptive when someone was already disoriented. The beacon became simple, clear, and a reliable target to move towards.
Demo and handoff
We invited a blind participant to try the prototype on two indoor routes: one with simple directional changes, one a tight winding course around furniture. Using only the beacon, they deftly navigated both. It was an exploratory session rather than a formal accessibility study, but it was strong evidence that the interaction was legible and precise enough to keep developing.
I handed off my project code to another prototyper, who continued to build on it with the audio scientist. Whether the feature will eventually appear in future devices is, as far as I know, up in the air.
Summary
My work touched research initiatives rather than public launches, so I evaluate it by what it made possible:
- Hard-to-access research systems became experiences teams could evaluate, and both were picked up and extended by other people after I left.
- Interaction decisions came from observed behavior, not technical feasibility alone.
- The work supported multiple symposium and conference demonstrations, led to ongoing and expanded lines of research, and contributed to a patent filing (US-20260089459-A1).