SpaceSense
Agentic intelligence applied to multimodal spatial data.
I worked with Astra, contracted by Boeing, to explore uses for spatial intelligence applied to the domain of aircraft interior design. I conceived, designed and built a functional prototype dashboard, called SpaceSense, that fused multiple captured data streams together and allowed the user to query them with natural language to derive insights.
Querying multimodal data with agentic intelligence
Behind the scenes, the dashboard leveraged an agentic loop with a variety of tools for querying data in different ways, including VLM for visual understanding and Pandas dataframe queries for structured data analysis. The tool was designed to permit any type of data that could be loaded in a .rrd (Rerun.io replay) file - to answer a question, the agent determined what types of data it had available, evaluated the tools with which it could query that data, and formed a multi-step plan for answering the user question.
Answers that involved particular points in space or time were cited with clickable links that caused the viewer to jump to that particular moment, improving confidence & observability. Rerun.io kept all data in memory, so jumping to linked moments was essentially instant.
One-minute mockup to win buy-in
The mock-up that won buy-in was produced in less than a minute - I took a screenshot of ChatGPT overlaid on a Rerun.io dashboard and pitched it as “ChatGPT for your private multimodal spatial data”.
Capturing multimodal spatial data
My team captured multimodal sample recordings in a mock cabin interior, including eye-tracking data, 4D point clouds, static 3D gaussian splats, speech transcripts, and multiview synchronized video streams. SpaceSense brought all this data together into a single unified dashboard where users could ask specific questions like “What objects were looked at the most?”, “What posted signage was most commonly overlooked?”, “Where did users tend to congregate?”, and even open-ended questions like “Were there any potential points of frustration or confusion?”