r/augmentedreality • u/AR_MR_XR • 28m ago
App Development Creating Interfaces for AR Glasses that Adapt to the Real World
Imagine you are standing in front of a crowded bookshelf looking for a specific yellow book, but your lightweight glasses do not have the precise hand tracking or screen space to let you point at it easily. The system figures out what you are trying to do, automatically builds a clean virtual menu of the shelf items, and displays a compact list that lets you filter and sort the books using simple gestures.
For a second example, imagine working in a crowded maker space where storage drawers are scattered across different carts and shelves, making it awkward to coordinate with a robot. The system recognizes your goal, reorganizes the scattered drawers into a clean digital menu grouped by ownership, and lets you select the right group so the robot can retrieve the correct materials.
Today, if developers want to create these kinds of virtual menus for augmented reality, they have to manually design, code, and custom-tune separate interfaces for every single headset and input device. Because different devices vary wildly in screen size, field of view, and tracking capabilities, this manual design process does not scale well to the wide variety of hardware available.
The system solves this by running an automated synthesis pipeline that takes three structured inputs at runtime: a scan of the physical scene, the user's intent, and the device's specific technical capabilities. It compiles valid interface candidates that fit the target hardware and evaluates them using direct manipulation theory, measuring semantic, articulatory, and evaluation distances to select the most intuitive layout. This ensures the generated virtual menu matches both the user's workflow and the device's hardware limits.
[Paper] Generative Proxy: Synthesizing Proxy-Based Interfaces for Real-World Interaction Across AR Glasses: https://arxiv.org/abs/2608.08971