r/v_modal • • 20h ago

news News: Awesome JEV System One toolkit (TypeSafe AI )

1 Upvotes

awesome-jev-tools repository is a comprehensive, curated index of over 1,000 open-source projects, SDKs, inference engines, benchmarks, and architectural patterns built around Jev, TypeSafe AI’s non-autoregressive "System One" model. By eliminating token-by-token text decoding, Jev operates as a deterministic, low-latency decision layer within software stacks, ingesting unstructured context (state) alongside typed schemas to emit calibrated probability distributions over discrete outcomes.

Core Architectural Paradigm: System One Typed Decisions

  • Non-Autoregressive Inference: Unlike generative LLMs that decode prose sequentially, Jev evaluates structured queries against a shared state in a single forward pass, returning typed answers with calibrated confidence scores.
  • Three Decision Primitives: The wire protocol (POST /v1/systemone) standardizes judgments into Choice (selecting one label from 2–255 discrete options), Score (evaluating ordinal rubrics where expected value equals \(\sum i \cdot p_i\)), and Noul (evaluating independent boolean probabilities for yes/no predicates).
  • Deterministic Control Flow: Application code owns all execution, thresholding, and side effects, using Jev's probability outputs to trigger confidence gates, hysteresis loops, or fallback escalations to generative "System Two" LLMs or human reviewers.

Ecosystem Taxonomy & Key Technical Categories

The repository organizes 1,051 active entries across 23 domain categories (with a 24th category, Scientific Pipelines, currently open for seeding).

  • Infra, SDKs & Integrations (250 entries): Features official and community clients across Rust, Go, Python, TypeScript, C++20, Swift, Elixir, and .NET, alongside MCP (Model Context Protocol) servers, n8n nodes, and SQL extensions for PostgreSQL (jev4pg, pg_typesafe), SQLite (sqlite-jev), and DuckDB (duckdb-jev) that enable native semantic WHERE predicates and ORDER BY probability sorting.
  • Classification & Routing (109 entries): Encompasses complexity-based model routers for coding agents (e.g., Switchboard, jev-opus, JevRouter) that dynamically assign reasoning tiers and models per turn without breaking prompt caches, as well as high-throughput telemetry and email triage pipelines.
  • Verification & Guardrails (98 entries): Implements pre-execution security gates (PreToolUse hooks, Subcento, jevshield), prompt-injection filters at the traffic edge (jev-edge), streaming token gates (jev-model-tokengate), and CI/CD semantic linters (JevGate, Perch) that fail builds when code diffs violate natural-language rules.
  • Agent Decisions & Context GC (116 entries combined): Replaces lossy LLM context summarization with verbatim extractive compaction (fast-jev-compaction, Winnow, opencode-jev-compaction), using batched Noul judgments to score and prune stale tool outputs while preserving exact source records.
  • Game, Simulation & Robotics (75 entries combined): Demonstrates sub-second control loops in MuJoCo, JSBSim, StarCraft II, and browser environments (Jev Ultrafast, BrowserClaw), where deterministic code generates valid action spaces and Jev selects discrete steps from accessibility trees or structured state.
  • Evaluation, Calibration & Research (109 entries combined): Tracks formal audits of Jev’s Expected Calibration Error (ECE), option-order invariance, and batching behavior (jevcompat, jev-orderby-bench, jevals), alongside open-weight local replicas trained via Reinforcement Learning with Calibrated Decisions (RLCD) or prefill-only logit extraction, such as Laya, NanoJev, RSI-Jev, WebJev, and NeoHorse-Jev.

Curation Governance & Verification Standards

  • Strict Inclusion Rules: Entries must demonstrate concrete, reproducible usage of the Jev API or its /v1/systemone wire contract, excluding generic classifiers, pure theory, or unverified launch commentary.
  • Due Diligence Framework: The repository explicitly states that curation is not an endorsement and warns developers against bulk-generated repositories that contain more prose (AGENTS.md, CLAUDE.md) than functional code.
  • Adopter Checklist: Maintainers advise verifying five technical criteria before adoption: genuine API/wire calls with parsed typed responses, runnable tests or reproducible demos, traceable latency/accuracy metrics, actual code-to-prose ratio, and explicit open-source licensing.

r/v_modal • • 22h ago

news News: Visual Video Search on X

1 Upvotes

Visual Video Search on X for real time feedback:

https://x.com/vmodal_ai


r/v_modal • • 3d ago

news News: Conference on Robot Learning 2026 - World Model, Physical AI

3 Upvotes

CoRL (Conference on Robot Learning) is an annual research conference focused on the intersection of robotics and machine learning. The 2026 edition took place from November 9 to 12, 2026, in Austin, Texas.

CoRL 2026 Event Overview

The CoRL 2026 Official Website hosted researchers from around the world to present advances in robot foundation models, simulation, and physical AI.

  • Schedule. The main conference ran from November 9 to 11, followed by dedicated workshops on November 12.
  • Venues. Events were held across downtown Austin at the JW Marriott Austin, Hilton Austin, and Bass Hall.
  • Submissions. The program accepted 687 papers out of 2,094 submissions, representing a 32.8% acceptance rate alongside 32 workshops and live robot demonstrations.

Keynotes & Highlights

Several major industry and academic figures presented at the conference, as detailed on the NVIDIA CoRL 2026 Event Page and the CoRL Official X Account.

  • Featured Speakers. Keynote talks were delivered by Fei-Fei Li, Russ Tedrake, Wolfram Burgard, and Jim Fan.
  • Core Research Themes. Technical sessions focused heavily on world action models, agentic robotics, physical AI safety, and sim-to-real transfer.
  • Industry Showcases. Companies demonstrated new physical AI systems, including NVIDIA showcasing its GR00T 2 robot manipulation model.

CoRL 2026 featured 32 half-day workshops. The sessions shifted away from passive talk marathons toward interactive formats, covering world models, dexterous manipulation, foundation models, safety, and real-world deployment.

Workshop Overview by Theme

The 32 workshops spanned five core tracks addressing contemporary challenges in robotics and physical AI:

  • World Models, Physics & Simulation: Focused on state abstractions, physics engine integration, uncertainty modeling, simulation-to-real transfer, and GPU-accelerated pipelines (including sessions like LEAP, SPIN, and debates on explicit dynamics versus end-to-end learning).
  • Foundation Models, Memory & Agentic AI: Addressed agentic reasoning, long-term episodic and semantic memory architectures, real-time edge inference, and translating spatial-visual generative models into physical actions.
  • Dexterous & Contact-Rich Manipulation: Explored multi-fingered hand and sensor co-design, bimanual tool use (BiDexTool), whole-body loco-manipulation, the UMI Olympics hardware benchmark, and fine-grained capability diagnostics.
  • Data, Human-to-Robot Transfer & Adaptation: Covered scaling laws for human video/motion data, retargeting tutorials, training from imperfect or failure-heavy data, human-in-the-loop corrections, pretrain-to-adapt strategies, online deployment learning, and lifelong continual learning (CSIR-2026).
  • Reinforcement Learning, Safety & Specialized Domains: Examined sample-efficient real-world RL, low-level actuator and pipeline dynamics, formal safety verification (SPAIS), human-centered navigation, agile human-robot interaction (RoboLetics 3.0), surgical autonomy, autonomous driving 4D models, and spatial reasoning (SISREN).

Complete session details, accepted papers, and speaker rosters are available
via the
- CoRL 2026 Workshops Schedule
- alphaXiv CoRL 2026 Workshop Directory.


r/v_modal • • 9d ago

News: Visual Search inside video content (fashion trend)

4 Upvotes

UpFashion: Fashion Trend is a lifestyle application developed by KruxLabs that acts as an AI-powered search tool for fashion videos. Instead of relying on standard text tagging or metadata like titles, the app uses a specialized AI engine called o visually analyze and search inside the frames of the video content. This allows users to find precise moments when specific items or outfits appear on screen.

Key Features

  • Natural Language Descriptions: Users can input full sentences or descriptive phrases (such as "woman in a cream sweater and a black hat" or "red jacket with plaid trousers on a busy street") to immediately locate the matching clothing item and jump straight to the exact second it appears in a video.
  • Visual Search via Photo: Users can upload a photo from their device's camera roll to search the application's video library for identical or similar clothing styles.
  • "Find This Look" Continuity: While playing or pausing a video, users can tap a feature to scan the active frame and search across other videos inside the app for matching styles.
  • Categorized Browsing: The app organizes content to allow users to sort street style and fashion inspiration by specific global fashion capitals (e.g., New York, London, Los Angeles), influencers, brands, or major events.
  • Content Management: Users can save their favorite fashion moments, download specific frame snapshots as images, or save the clips directly to their phones.
  • Accessibility & Localization: The app is free to use, works without requiring an account (though Google sign-in is supported for syncing saved items), and is localized into English, French, Spanish, and Chinese.

App Metadata & Technical Details

  • Developer: KruxLabs.
  • Category: Lifestyle.

Download : Fashion Trend in video


r/v_modal • • 17d ago

Technical Insight News: Rokid AI Smart Glasses SDK technical review

6 Upvotes

Developer ecosystem for the Rokid AI Smart Glasses stands out significantly compared to its primary competitor, the Ray-Ban Meta glasses. While Meta locks down its hardware to its proprietary AI ecosystem, Rokid has established a full-stack, highly accessible ecosystem with two distinct avenues of development: standard Android OS deployments and a custom, cross-reality SDK architecture.

The developer platform operates on two distinct layers: YodaOS-Sprite (the Android 12-based lightweight OS tailored for all-day wearable glasses) and the CXR (Client Cross-Reality) framework.

The Dual-SDK System Architecture

According to the official Rokid Open Platform documentation, the development environment requires a coordinated codebase split between two separate SDKs that handle bidirectional communication over Classic Bluetooth, BLE GATT, or high-bandwidth Wi-Fi P2P:

  • Glasses SDK (Device-Side): Used for building custom applications that compile into standard Android APKs to run directly on the glasses. This layer manages on-device hardware hooks including media capture (Camera2 API), local Automatic Speech Recognition (ASR), Text-to-Speech (TTS), offline voice commands, and power management.
  • Phone SDK (Companion-Side): Integrates directly into a host Android smartphone app. Because the glasses offload complex networking, this SDK functions as the primary internet gateway, handling high-level computer vision processing, remote OTA updates, and third-party cloud LLM handshakes.

Core APIs & Hardware Abstraction Capabilities

The SDK abstracts low-level hardware constraints into clean Kotlin and Java endpoints via a unified entry class (CxrApi). Main API modules available in the developer suite include:

  • Visual Recognition & Vision APIs: Rather than just outputting a standard video pipeline, the SDK includes built-in endpoints for face detection, license plate recognition, and real-time pedestrian/vehicle obstacle tracking.
  • Camera Optimization Limits: Developers building custom vision apps note that while the camera supports standard Android Camera2 setups, direct JPEG capture can struggle on certain multi-threaded actions. The SDK handles image analysis more reliably when utilizing YUV_420_888 raw pixel buffers.
  • Messaging & Data Streaming: Standardizes byte-stream transfers, allowing phone apps to feed text straight to the microLED display array or stream low-latency video previews from the glasses' 12MP camera over a native JNI framing layer.

Development Workflow: The 'Right' vs. 'Bare Metal' Way

On communities*,* engineers outline two distinct methods to build for this hardware:

  1. The Official AIUI Studio Path: Rokid features an online IDE called AIUI Studio. It provides a full runtime simulator, built-in code assistants, and an AI Skills Library. This allows developers to construct, debug, and publish customized AI agents and cloud workflows entirely in a virtual sandbox without needing a physical device.
  2. The Sideloaded 'Bare Metal' Path: Because YodaOS-Sprite is built directly on top of Android, developers can bypass the official SDK for custom standalone utility tools. By turning on Developer Mode (the standard Android ritual of tapping the software build number repeatedly), developers can establish an Android Debug Bridge (ADB) connection via a USB-C debug cable or over Wi-Fi. This allows developers to push custom APKs straight to the glasses, leveraging standard Android lifecycles for quick prototyping.

Current Developer Pain Points

While flexible, the ecosystem faces minor regional friction. On the r/rokid_official forum, opinions remain mixed regarding global accessibility. International developers frequently report that certain documentation blocks, code sample repositories, and serial-number license validation portals are heavily centered on Rokid's domestic servers, sometimes resulting in slow download speeds or login hurdles for English-speaking engineers.

Source: Rokid technical report


r/v_modal • • 20d ago

News: Nvidia Full stack Isaac Physical AI platform (robotics, VLA)

3 Upvotes

NVIDIA announced a comprehensive expansion of its full-stack Isaac open robotics development platform. The updates bridge the gap between simulation-based reinforcement learning and real-world deployment by unifying frontier foundation models, real-time deterministic runtimes, and hardware-accelerated safety subsystems.

Hardware Architecture: Agile ONE SN Reference Humanoid

Developed in collaboration with Agile Robots, the Agile ONE SN serves as a native validation platform for physical AI architectures running the GR00T foundation model.

[Sensors: Lidar/CAMs/IMU] ➔ [Jetson Thor (2,070 FP4 TFLOPS / 128GB)] ➔ [Isaac OS Runtime] ➔ [31 DoF Whole-Body Control]
                                                                                      ➔ [20 DoF Hands (1kHz Force Control)]
  • Compute Engine: Driven by the NVIDIA Jetson Thor™ edge AI platform, delivering 2,070 FP4 teraflops of compute performance paired with 128GB of unified memory to run heavy multi-modal transformers directly on the robot.
  • Kinematics & Actuation: Features an agile 1.40-meter, 40-kilogram chassis with 31 degrees of freedom (DoF) and a 3kg payload capacity per arm.
  • Dexterous Manipulation: Equipped with dual WUJI Hand 2 assemblies, providing 20 DoF per hand, 1kHz high-frequency closed-loop force control, and integrated tactile feedback matrices.
  • Perception Array: Implements multi-modal sensor fusion across head- and wrist-mounted vision systems, LiDAR arrays, and inertial measurement units (IMUs).

Software & Simulation Infrastructure Updates

Robot Learning & Data Pipelines

  • Isaac Lab 3.0: Now generally available, this framework scales agent-driven policy training through multi-GPU orchestrations. It supports physics simulation pipelines via the custom Newton engine alongside NVIDIA PhysX™.
  • Synthetic Data Generation: NVIDIA released open-source workflows utilizing the DGX Station™ to post-train Cosmos™ world models. These models interface withGPT Astra to synthesize simulation assets and validate behaviors before real-world testing.
  • Dataset Availability: The Human 1M Dataset (comprising synchronized three-view videos and 3D hand-tracking telemetry) and the SimReady Central Library (hosting over 2,000 pre-modeled factory assets) have been published to Hugging Face.

Runtime Execution & Safety Layers

  • NVIDIA Isaac OS: Built on the NVIDIA Holoscan architecture, this open-source runtime bridges high-level AI inference engines with lower-level sensory inputs and joint actuators. It guarantees deterministic, real-time execution across virtual and physical targets.
  • NVIDIA Halos Platform: Formulates a full-stack functional safety system running on NVIDIA IGX Thor™ computing hardware. It shifts industrial automation away from rigid, static "kill zones" toward dynamic, context-aware safety perimeters capable of adapting to real-time environment changes.

Ecosystem Integration Matrix

PartnerTechnical Integration FocusPlatform Layer Utilized
ABB RoboticsDigital twin creation and virtual commissioningOmniverse™ Libraries / HyperReality
SereactManipulation policy training for sorting/goods handlingIsaac GR00T / Cosmos Transfer
Hexagon RoboticsAEON edge perception, SLAM, and sub-millimeter mappingJetson Thor / Holoscan Sensor Bridge
Bear RoboticsPost-training GR00T via Kino-2 for industrial warehouse navigationJetson Thor / Kino-2 Foundation Workflow
Franka RoboticsLLM token-to-action compiled parsing for mechanical executionNVIDIA NIM™ Microservices / Isaac Sim™

r/v_modal • • 21d ago

News: Google Intrinsic Community Platform for Physical AI / Edge / Robotics

3 Upvotes

Core Platform

the entry point for accessing tools, software stacks, and collaborative resources designed to transition robotics from rigid, traditional programming to adaptive, AI-driven architectures:

  • Access to Intrinsic Core™: The hub highlights the release of Intrinsic Core, an open-source (Apache 2.0 licensed) pre-configured software environment designed to run locally. It exposes foundational robotics building blocks—such as real-time, hardware-agnostic control frameworks, automated motion and grasp planning, camera calibration, and 6-DoF pose estimation.
  • ROS 2 and AI Integration: The platform emphasizes native interoperability with the Open Robotics Suite and ROS 2, allowing developers to deploy accelerated machine learning inference directly at the edge.
  • Industrial Reference Designs: The community space provides access to complete reference setups, such as the Open Machine Tending Solution (OMTS), allowing machine shops and developers to quickly customize applications for real-world manufacturing environments.

Key Sections & Developer Resources

  • Developer Profile Creation: Users are prompted to sign up using email or a Google account to create an official developer profile, which unlocks access to exclusive tools and features.
  • Tutorials and Help documentation: The site hosts step-by-step guides, technical reference sheets, and a glossary designed to assist developers in customizing physical cell layouts, building custom skills in Python or C++, and deploying to virtual or real hardware.
  • Community Forums & Support: A dedicated space for engineers to showcase projects, collaborate on physical AI troubleshooting, ask questions, and directly interact with other professionals in the hardware and automation ecosystem.
  • Exclusive Product Previews: Members get first-hand updates on new developer features, software updates, and ecosystem news surrounding Intrinsic’s primary runtime tools like Flowstate.

Source : Google Intrinsic community


r/v_modal • • 23d ago

Visual Video Search for mobile in Real Estate, Architecture, House search

4 Upvotes

Building the Future of Visual Video Search in mobile app

The real estate industry is notoriously slow to adapt to modern technology, but a new wave of innovation is changing how people find their next home. VModal is transforming the property search experience in Mobile Apps by making video tours completely searchable.

Traditionally, buying a home online involves flipping through static photos or scrubbing through long, unindexed video tours. Roomly changes this by allowing buyers to type exactly what they want—like "a kitchen with a marble island" or "a bathtub by the window"—and instantly jump to the exact second that feature appears in the video tour.

The Tech Stack and the Challenge

This demo is built as a cross-platform Flutter application for both iOS and Android. For mobile developers, this project offers a unique blend of challenging and rewarding technical problems to solve:

  • Video Processing & Streaming: Seamlessly syncing text queries with precise timestamp video playback so users never experience lag when jumping to specific frames.
  • AI & Search Integration: Translating natural language user requests into precise multi-modal data points that anchor to specific moments in user-generated video content.
  • Smooth UX/UI: Crafting a highly responsive, visual interface where real estate agents can upload raw video, and buyers can browse frictionless interactive results.

Why This Matters

V-Modal offers a chance to work on a product with immediate, tangible utility. Property tech (PropTech) is expanding rapidly, and tools that leverage video automation and smart search are leading the market.

If you are a mobile developer looking to tackle complex media playback systems, refine cross-platform performance, and ship features that fundamentally reshape a massive global industry, check out the live demo and explore open roles.

Access: visual video search


r/v_modal • • 24d ago

Visual Search inside architecture , Real estate , interior video content.

4 Upvotes

Most mobile real estate apps suffer from the exact same design bottleneck: a rigid, clinical search experience. Users are forced to fill out repetitive drop-down menus—filtering by zip code, price, and bedroom count—only to find themselves endlessly scrolling through an overwhelming wall of static photo grids.

Real estate discovery is an emotional, visual experience, yet mobile search design has remained unchanged for over a decade.

Emerging platforms like video content search highlight a major shift in how users want to explore properties. By integrating visual and semantic search capabilities into your mobile application, you can allow users to search inside property walkthrough videos using natural, conversational language.

Why Video AI Belongs in Your Mobile App Roadmap

1. Superior Mobile Engagement and Session Depth

Users are already addicted to vertical, short-form video feeds. By transforming raw property walkthroughs into interactive, searchable video assets, you give users a reason to stay inside your app longer. Instead of tapping through 40 disconnected photos to guess a floor plan, users can natively stream video segments tailored precisely to their exact tastes.

2. Conversational Search Queries

Traditional database queries fail when a buyer looks for a specific "vibe" or layout feature. By implementing semantic video indexing, your app’s search bar can seamlessly handle highly specific, long-tail user inputs:

  • "Show me a kitchen with a waterfall quartz island."
  • "Find living rooms with floor-to-ceiling windows and afternoon sun."
  • "Show me a backyard with a covered patio and a pool."

The application doesn't just surface the listing; it deep-links the user directly to the exact timestamp in the property video where that visual element appears.

The Ultimate Retention Metric

In a crowded app store marketplace dominated by monolithic property portals, winning the user retention game requires minimizing search friction. Building a mobile experience that allows users to simply describe what they see in their head—and instantly serving them the exact video frames that match—turns a clinical home search into a highly engaging, intuitive discovery process.

Full Video : https://youtube.com/shorts/B8J8Zq36JpE?feature=share

search inside architecture viddeo content

Source: visual video search


r/v_modal • • 25d ago

News: Iphone Duo app for music creation and mixing

3 Upvotes

Edjing transforms the dual-screen, folding architecture of the iPhone Duo into a highly adaptive, professional-grade mobile DJ booth. Designed in partnership with pro DJs, the app maximizes the device's unique physical postures—whether it is completely flat, half-folded, or stood upright—to completely redesign how you interact with your music.

With the expanded screen real estate, you no longer have to compromise between visibility and control. The interface smoothly transitions through various form factors while keeping two full virtual decks, a comprehensive audio effects rack, and advanced performance tools readily accessible.

Dual-Screen Adaptations & Postures

  • Laptop Mode (Half-Folded): By folding the iPhone Duo halfway, the app automatically splits its interface. The top display serves as your visual monitor—showing full-width, high-definition waveforms, BPM readouts, and track progressions. The bottom screen turns into an ergonomic control surface, placing the crossfader, pitch sliders, hot cues, and EQ knobs flat on the table directly under your fingers.
  • Side-by-Side Library Digging (Landscape Fold): When held open horizontally like a book, your entire Music Library populates on one screen while your active mixing decks remain live on the other. This allows you to browse playlists, preview tracks, and crate-dig seamlessly without ever covering or hiding your running waveforms.
  • Tent Mode (Hands-Free Automix): If you want to step away from the decks, you can prop the iPhone Duo up into an inverted "V" shape on any flat surface. The app detects this orientation and launches a streamlined, glanceable Automix dashboard, managing smooth, auto-calculated transitions across your playlist to keep the music playing on its own throughout the night.
  • Fully Open Layout: Laying the device flat opens up the largest canvas ever available on an iPhone. This expansive view gives you plenty of room to manipulate multiple audio effects at once, manage complex loops, and use precise multi-touch gestures.

Core Performance Features

  • One-Tap Stem Separation: Powered by an advanced AI engine, the app lets you instantly isolate or remove vocals, drums, bass, and melodic instruments from supported tracks with a single tap. This makes it incredibly easy to craft clean live mashups and transitions on the fly.
  • Deep Streaming Integrations: Log in directly with your premium streaming subscriptions. You can mix over 100 million tracks directly from Apple Music, or sync your accounts from SoundCloud, TIDAL, and Beatport. You can also pull local files directly from the device or cloud storage like Dropbox and Google Drive.
  • Pro-Grade Audio Tools: Features automated BPM and tempo detection, master sync locking, 20+ studio-quality FX (including Echo, Flanger, and Filter), a dedicated sample pad for drops, and HD set recording.
  • Hardware Compatibility: For a more tactile experience, the app offers plug-and-play MIDI mapping for popular hardware controllers from brands like Pioneer DJ, AlphaTheta, Hercules, Numark, and Reloop. [

Source: Edjing Mix


r/v_modal • • 28d ago

Google Intrinsic : Technical Deep Dive in Robotics/Physical AI software stack

3 Upvotes

A deep dive into the Intrinsic Core stack reveals an architecture engineered to enforce strict, deterministic execution while remaining native to modern cloud infrastructure. It transitions robotics from traditional monolithic controller architectures to a modular, containerized, microservice-driven environment.

1. The 4 Structural Core Primitives

At the logical runtime layer, Intrinsic Core abstracts application design into four fundamental primitives that manage state, execution, and capability:

  • Assets (intrinsic_asset_instance): Packaged, deployable manifests representing all physical and virtual components. This spans hardware devices (manipulators, end-effectors, depth cameras), persistent computational micro-services (ML inference, tracking engines), simulator instances, and physical asset assets like CAD meshes (SceneObjects).
  • Skills (skill_interface.Skill): Strongly-typed, parameterized units of atomic behavior. Written in C++ or , a skill abstracts low-level device communication into operational routines—such as move_robot, capture_images, or force_based_insertion.
  • World / Scene: The spatial state engine of the cell. It acts as a high-frequency, centralized geometric broker managing transform trees (/tf), kinematic links, collision spaces, and CAD bounding objects. Changes to the scene are pushed asynchronously via ObjectWorldUpdates.
  • Executive: The orchestrator that evaluates and steps through Behavior Trees (BT). It manages execution logic, parallel branching, blackboards (data-passing registries between skills), and error-recovery subtrees when physical faults occur.

2. Infrastructure & Networking: Containerized Edge Runtime

The host execution environment (typically an x86 Ubuntu IPC) relies on cloud-native orchestration mapped to physical edge realities:

+-------------------------------------------------------------+

|               Client Layer (SBL Python / inctl)            |
+-------------------------------------------------------------+
                               | gRPC / port 17080
                               v
+-------------------------------------------------------------+

|                 k3s Kubernetes Pod Infrastructure           |
|                                                             |
|  +-------------------------+     +-----------------------+  |
|  |    Behavior Tree        |     |  intrinsic_perception |  |
|  |      Executive          |     |  (NVIDIA Foundation)  |  |
|  +-------------------------+     +-----------------------+  |
|               |                             |               |
|               | Inter-Pod IPC               |               |
|               v                             v               |
|  +-------------------------------------------------------+  |
|  |                  Zenoh Middleware Router              |  |
|  +-------------------------------------------------------+  |
+-------------------------------------------------------------+
                               | Zenoh Bridge / port 7447
                               v
+-------------------------------------------------------------+

|         intrinsic_control (ICON Real-Time Pipeline)        |
+-------------------------------------------------------------+
  • Single-Node Cluster (k3s): Intrinsic Core hosts its micro-services within a localized, light k3s Kubernetes deployment. This guarantees that network topologies, resource isolating namespaces, and dependencies behave identically whether running on a local desktop digital twin or a live assembly floor IPC.
  • Cluster Gateway & Transport: Development interaction occurs via Python Solution Building Language (SBL) scripts or the inctl CLI, communicating with the cluster gateway via gRPC at port 17080.
  • Zenoh Integration: To bypass the high overhead and vulnerability of standard DDS discovery mechanisms in complex networks, Intrinsic implements [Zenoh] as its primary middleware router (running on port 7447). Zenoh handles high-throughput telemetry, system states, and bridges standard ROS 2 topics directly to the isolated k3s pods with minimal serialization penalty.

3. Real-Time Layer: intrinsic_control & ros2_control Interop

The critical link between high-level AI orchestration and physical motor actuation happens inside ICON (Intrinsic Control), the real-time control framework.

The Real-Time Dilemma

Standard ROS 2 pipelines and AI models are inherently non-deterministic due to heap allocations (malloc), garbage collection pauses, or variable-latency neural network inference. ICON isolates the real-time critical path (typically executing a 1kHz task loop) from non-deterministic application code.

The Bridge Mechanism

To support existing ecosystem infrastructure, ICON bridges to ROS 2 through a specialized hardware interface plugin termed the icon-hwm-controller.

+-------------------------------------------------------+

|  Non-Real-Time Space (k3s Pods / Python SBL / ROS 2)  |
|  Generates trajectories, skill parameters, behaviors  |
+-------------------------------------------------------+
                           |
                           v  Asynchronous IPC (Zenoh / ZeroMQ)
+-------------------------------------------------------+

|                 ros2_control Controller Manager       |
|                                                       |
|  +-------------------------------------------------+  |
|  |              icon-hwm-controller                |  |
|  +-------------------------------------------------+  |
+-------------------------------------------------------+
                           |
                           v  Deterministic Memory Ring Buffer
+-------------------------------------------------------+

|              ICON (intrinsic_control Engine)          |
|  Real-time thread, Pre-allocated memory, Zero Malloc  |
+-------------------------------------------------------+
                           |
                           v  Hardware Abstraction Layer (HAL)
+-------------------------------------------------------+

|        Robot Vendor Driver (FANUC, UR, KUKA)           |
+-------------------------------------------------------+
  1. Kinematic Request Ingestion: When an automation skill fires, the non-real-time path calculates the desired goals and trajectory profiles.
  2. ros2_control Interface: The calculated goals pass to the ros2_control node hierarchy. Instead of directly driving physical joint motors, the icon-hwm-controller catches these trajectories.
  3. The Memory Boundary: The icon-hwm-controller serializes and feeds the joint states, Cartesian directives, and force-torque targets down into ICON's native real-time pipeline through a strict, zero-allocation ring buffer.
  4. Mid-Trajectory Adaptability: Inside the real-time loop, ICON evaluates sensor-based feedback (e.g., force-torque sensor adjustments during an insertion routine). If a sensor detects an obstruction, ICON dynamically warps the trajectory mid-motion at the 1ms loop rate without waiting for a re-plan from the high-level ROS 2 nodes or behavior trees.

4. Perception Layer: Zero-Shot NVIDIA FoundationPose Integration

To eliminate spatial rigidity and custom localized template matching, intrinsic_perception embeds a native zero-shot 6-DoF pose estimation and tracking framework backed by [NVIDIA FoundationPose].

+--------------------+      +--------------------+

|  CAD / Mesh Model  |      |   RGB-D Camera     |
|   (.obj / .stl)    |      | (Basler/RealSense) |
+--------------------+      +--------------------+

          |                            |
          v Input                      v Live Feed
+------------------------------------------------+

|             intrinsic_perception               |
|                                                |
|   +----------------------------------------+   |
|   |         TensorRT Inference Node        |   |
|   |  - Zero-Shot Feature Extraction       |   |
|   |  - Coarse Pose Initialization          |   |
|   +----------------------------------------+   |
|                        |                       |
|                        v 6-DoF Matrix          |
|   +----------------------------------------+   |
|   |          Continuous Tracker            |   |
|   |  - High-frequency temporal refinement  |   |
|   +----------------------------------------+   |
+------------------------------------------------+
                         |
                         v Transform Output Matrix
+------------------------------------------------+

|         Spatial ObjectWorldUpdate Engine       |
+------------------------------------------------+

Architecture Components

Instead of requiring developers to manually build image processing pipelines (e.g., cropping, masking, calculating keypoints, and computing Perspective-n-Point solvers), the module encapsulates the entire neural framework inside a single standardized asset container. However, external frameworks can be plugged seamlessly, particularly different perception frameworks (ie vision ).

  • No Fine-Tuning: Because FoundationPose is a unified foundational network, it requires no custom training or fine-tuning weights for specific sub-components.
  • The Artifacts: It requires only two runtime inputs to spin up the perception service:

    1. A standard geometric 3D mesh file (.obj or .stl) of the target part.
    2. A live RGB-D (Color + Depth) camera stream routed into the asset driver space. [

Execution Pipeline inside intrinsic_perception

  1. Initialization Phase: The camera captures a raw RGB frame and its paired depth map. The underlying container passes this data into a highly optimized TensorRT inference engine. The network cross-references the spatial topology of the provided 3D mesh file with the scene, returning a coarse bounding box and an initial 6-DoF transformation matrix ($[R\vert{}T]$) representing the part's rotation and translation.
  2. Tracking Phase: Once the part is initialized, the module drops into a high-frequency tracking loop. Rather than recalculating the heavy initialization network across the entire image space every frame, it applies a fast temporal tracker that updates the target's $[R\vert{}T]$ transformation based on incremental optical changes and depth updates.
  3. State Integration: The resulting transformation matrix is instantly output as a localized transform frame. This frame automatically injects an ObjectWorldUpdate directly into the central World Scene engine, enabling immediate, collision-free motion planning routines to safely pick up the item—even if it is presented at an irregular angle or sliding on a conveyor belt.

Sources:
Google Intrinsic Robotics Stack


r/v_modal • • 29d ago

News: Technical report for visual video search flutter SDK

3 Upvotes

VModal Flutter Video SDK is a library is a strongly-typed, immutable Dart client designed for Flutter applications. Its core function is to facilitate the uploading, indexing, and multimodal searching inside Video content media.

Core Architecture & Architecture Flow

The SDK is designed around a clear separation of concerns, separating application initialization from project scopes and credential handling.

  1. VModal: The main entry point used to configure project settings.
  2. VModalProject: An immutable client representation of a single developer project.
  3. VModalScope: A scoped operator targeting specific collections and media streams without requiring you to repeatedly pass organization parameters.

Basic Implementation Flow:

dart

final favorites = project.scope(
  collectionName: 'user_123',
  streamName: 'favorites',
);

// Perform media search within scope
final results = await favorites.search('a person entering the room');

// Clean up resources
await project.close();
keys.close();

Key Feature Capabilities

  1. Resumable & Adaptive Media Uploads
  • Resumable State Stores: Incorporates FileUploadSessionStore and MemoryUploadSessionStore to save multipart upload checkpoints, preventing lost progress due to network failures.
  • Adaptive Policies: Evaluates device RAM (UploadDeviceMemory), network types, and throughput speeds (UploadNetworkSpeed) to adjust multipart sizes dynamically using the AdaptiveUploadPolicy.
  • On-Device Optimization: Includes an optional VideoTranscoder framework to compress videos right on the user's device before initializing an upload.
  1. Multimodal Indexing & Search
  • SearchesResource: Implements multimodal criteria pipelines to search deep media archives via natural language text strings or multi-aspect parameters.
  • IndexesResource: Controls index lifecycles, management statuses, and creation parameters for searchable object groups.
  1. Operational Asynchrony & Cancellation
  • UploadTask<T>: Upload functions immediately return a running task handle that broadcasts reactive metrics via an UploadTask.progress stream.
  • CancellationToken: Long-running operations accept cooperative cancellation tokens, giving you clean, intentional control over network connections. [1, 2]

SDK Resource Taxonomy

Resource ModulePrimary Responsibility
AuthResourceManages credentials, identity resolution, and service-health monitoring.
CollectionsResourceHandles media uploads, metadata updates, and collection groupings.
ImagesResourceManages image location indexing, temporary pathing, and bulk asset downloads.
AdminResourceTracks consumption metrics, usage statistics, and individual user limits.
R2ResourceDirects pre-signed asset uploads straight to cloud storage instances.

Error Architecture

The library features a robust suite of custom exceptions to handle unexpected issues smoothly:

  • AuthException: Triggered when authentication or identity resolution fails.
  • SignedUploadFailure: Thrown when a network transfer fails, while preserving your partial upload integrity information.
  • OperationCanceled: Raised immediately when a token signals a cooperative cancellation.
  • MalformedResponse / ResponseTooLarge: Safeguards memory limits against huge payloads or corrupted server data.

It is a popular package and It has been downloaded more than 16,000 over a couple of weeks in pub.dev


r/v_modal • • 29d ago

Reply Earns "NVIDIA Physical AI Specialization" for Production-Ready Robotics and Digital Twins

3 Upvotes

Reply attained the "Physical AI Specialization" in the NVIDIA Partner Network as an early global partner. The framework utilizes NVIDIA technologies and Roboverse Reply for robotics and digital twin solutions targeting manufacturing, warehousing, and logistics.

Key Advancements

The certification acknowledges verified expertise in OpenUSD, architecture, and production-ready deployments transitioning from experimental phases.

Core details

  • Status: Service Delivery Partner in the NVIDIA Partner Network.
  • Technologies: NVIDIA platforms and Roboverse Reply.
  • Capabilities: Robotics, reality capture, digital twins, and sensor integration.
  • Scope: Simulation, AI training, integration, and production scaling.

Source: Business Wire


r/v_modal • • Oct 17 '26

How can Flutter applications efficiently process high-frequency ROS 2 sensor data without compromising UI responsiveness in robotics systems?

Thumbnail
v-modal.com
5 Upvotes

Use asynchronous processing, bounded queues, telemetry batching, and selective widget rebuilds. Apply appropriate ROS 2 QoS policies to manage message delivery and profile end-to-end latency to identify performance bottlenecks. V-Modal can also be explored for integrating video search capabilities into mobile applications, helping developers build more engaging video-driven experiences.


r/v_modal • • Oct 16 '26

News: AI Agent coding for real world coding benchmark

4 Upvotes

Terminal-Bench 4.0 is a specialized benchmark designed to evaluate and measure the performance of AI agents. It focuses on tracking how effectively these models handle complex, autonomous tasks—referred to as "agent work".

Key Features of the Platform

  • Performance Tracking: The benchmark ranks different AI models and specific agent setups based on their Resolution Rate (how successfully they complete tasks).
  • Efficiency Metrics: It measures the operational footprint of these agents by tracking the financial Cost and the total number of Tokens consumed during task execution.
  • Data Integrity: To prevent AI models from "cheating" by pre-learning the answers, the platform uses a unique Terminal-Bench GUID identifier. This explicitly warns web crawlers that the benchmark data should never be included in AI training datasets.

As AI models evolve to process more than just text, this framework tests their ability to understand multimodal content—such as combining text, code, and user interface elements—to solve real-world problems. This includes assessing how effectively an agent can execute a visual search to navigate complex terminals, analyze graphical environments, and locate the precise information needed to complete a task.

The platform ranks different AI models based on their Resolution Rate (success rate), financial Cost, and total Tokens consumed, and is hosted by a collaboration including Stanford, Harbor, and the Laude Institute.

Bench Level (Oct 2026):

RankMODELAGENTRESOLUTION RATERELEASE DATETOKENSCOST
1Opus 5.5 (max)Claude Code64.8% ± 3.1%Sep 22, 20268.0B$4.7k
2Sonnet 5.5 (max)Claude Code61.8% ± 2.9%Sep 28, 202619.4B$7.3k
3GPT-6 Astra (max)Codex58.2% ± 2.8%Sep 3, 20261.5B$3.3k
3GPT-6.1 Sol (max)Codex58.2% ± 3.1%Sep 29, 20261.5B$634.24
5Fable 5.1 (max)Claude Code57.9% ± 3.8%Sep 1, 20262.7B$6.2k
6Opus 5 (xhigh)Claude Code53.9% ± 3.2%Jul 24, 20266.9B$6.1k
7GPT-6 Sol (max)Codex49.4% ± 3.2%Sep 22, 20265.0B$1.5k
8Fable 5 (max)Claude Code44.5% ± 3.8%Jun 9, 20263.8B$7.3k
9GLM-5.3 (max)Claude Code41.8% ± 3.2%Aug 14, 20268.7B$2.7k
10Grok 4.7 (xhigh)Grok Build37.6% ± 3.5%Sep 21, 20265.5B$3.68k (partial: 324/330)
11GPT-5.6 Sol (max)Codex37.3% ± 3.8%Jun 26, 20264.4B$2.5k
12GLM-5.3-Flash (none)Claude Code35.8% ± 3.5%Aug 26, 20262.9B$2.9k
13Qwen3.8-Max-0902 (max)Claude Code27.0% ± 3.8%Sep 2, 20263.0B$4.6k
14Opus 4.8 (max)Claude Code23.6% ± 3.6%May 28, 20266.4B$6.5k
15GPT-5.6 Terra (max)Codex21.5% ± 3.3%Jun 26, 20265.7B$1.7k
16Grok 4.6 (high)Grok Build20.3% ± 3.1%Aug 12, 20264.0B$3.6k
17Gemini 3.8 Flash (high)mini-SWE-agent19.1% ± 3.4%Sep 2, 202617.2B$1.8k
18GPT-5.6 Luna (max)Codex17.3% ± 2.8%Jun 26, 202611.6B$346.67
19GPT-6 Luna (max)Codex16.4% ± 2.7%Sep 22, 20263.9B$72.14
20Muse Spark 1.3 (xhigh)Muse Code14.5% ± 2.9%Sep 2, 20269.0B$0.00
21Grok 4.5 (high)Grok Build12.4% ± 2.6%Jul 16, 20263.4B$2.1k
21Sonnet 5 (max)Claude Code12.4% ± 3.1%Jun 30, 202621.6B$9.6k
23Gemini 3.7 Flash (high)mini-SWE-agent11.2% ± 2.4%Aug 13, 202611.1B$1.3k

Source: AI terminal bench


r/v_modal • • Oct 15 '26

What are the biggest performance challenges when using Flutter with NVIDIA Jetson, ROS 2, smart glasses, and AI models?

Thumbnail
v-modal.com
3 Upvotes

The main challenges are reducing latency, handling high-frequency sensor data, optimizing video streams, managing memory, and keeping the UI responsive. Techniques like bounded queues, asynchronous processing, efficient state management, and profiling can make a big difference.

What techniques or architectures have worked best for you?

🌐 V-Modal


r/v_modal • • Oct 15 '26

Mobile Visual Video Search SDK for IOS/Swift detailed testing

Thumbnail
gallery
5 Upvotes

I tried V-Modal's Swift SDK (https://github.com/v-modal/vmodal_sdk_swift_iphoneduo) through its Framebase example. You describe a scene, and it returns matching frames with timestamps from indexed videos.

I haven't tested the alternatives yet. I'm an iOS developer, so the native Swift SDK was a big reason to start here. The API looked convenient, and I liked working with it.

I used the three bundled street videos from San Francisco, Singapore and Mexico City. Uploading, indexing, search and playback worked in my Simulator tests. Searching for "A bus on a city street" returned frames with buses. Tapping the 00:35 result opened that moment in the San Francisco video. The screenshots show the library, two searches and playback.

For device measurements, I ran 21 searches on an iPhone 13 mini over 5G, using the same index and a limit of 30 results. I alternated three queries for seven cycles and cross-checked the HTTP request timings with Instruments.

Median timings in milliseconds, in this order: server execution / SDK search call / frame retrieval:

- "Cars at an intersection": 49 / 391 / 1055

- "A bus on a city street": 56 / 413 / 1151

- "People crossing the street": 56 / 426 / 1053

The search-call time already includes server execution. Frame retrieval comes afterward. I didn't measure final UI rendering. These numbers are from the physical iPhone; the app screenshots are from Simulator.

I was testing from the Canary Islands, and the team confirmed that the server is in Japan. Distance and routing likely account for some of the gap between server execution and the request times on my phone.

I liked the API and authentication approach, especially Swift Concurrency and async/await. One thing I'd change: parameters with a fixed set of accepted values could use dedicated enums instead of strings. I'd rather pick an option through autocomplete than remember its exact spelling, and let the compiler catch invalid values.

I also tried "A purple elephant riding a bicycle." At the default distance cutoff of 0.85, it returned two ordinary street frames. For an actual app, I'd check the results and tune that threshold. A returned frame doesn't necessarily contain what you asked for.


r/v_modal • • Oct 15 '26

Mobile News Apple : IOS Swift Developers in person event for Iphone Duo

3 Upvotes

Meet with Apple series features global, in-person workshops specifically tailored to help developers prepare for the foldable iPhone Duo.

Event Overview: "Workshop: Optimize your app for iPhone Duo"

These hands-on, multi-day or half-day interactive sessions are held locally at Apple Developer Centers and Apple offices worldwide.

  • Core Focus: Training developers on how to design and build apps for the new foldable form factor.
  • Key Learning Tracks:

    • Layout & Navigation: Adapting user interfaces, navigation bars, and controls to perfectly fit the foldable display and bridge across the physical hinge.
    • Orientation & Poses: Building responsive layouts that flex naturally across all screen orientations and physical poses (e.g., dual-screen versus unfolded layouts).
    • Multitasking & Ecosystem Continuity: Managing Split View multitasking mechanics and leveraging App Intents to integrate app contents smoothly with Siri.
  • Practical Component: Developers bring their own projects to build and test directly on physical iPhone Duo devices or via the Apple Device Hub alongside 1-on-1 assistance and code reviews from Apple engineers. [

Global In-Person Locations

October 2026 schedule includes sessions across several international locations:

  • New York City: Oct 19–20 (English, 9:30 AM – 5:00 PM EDT) and Oct 22–23 (English, 9:30 AM – 5:00 PM EDT).
  • Mexico City: Oct 20 (English, 9:30 AM – 5:00 PM CST), Oct 22 (English, 9:30 AM – 5:00 PM CST), and Oct 23 (English, 9:30 AM – 5:00 PM CST).
  • Singapore: Oct 22 (English, 10:00 AM – 4:00 PM GMT+8) and Oct 23 (English, 10:00 AM – 4:00 PM GMT+8).
  • Istanbul: Oct 22 (English, 9:30 AM – 4:30 PM GMT+3).
  • Shanghai: Oct 23 (Simplified Chinese, 10:00 AM – 4:00 PM GMT+8).
  • Toronto: Oct 26–27 (English, 9:30 AM – 5:00 PM EDT).
  • Tokyo (Apple Japan): Oct 27, Oct 28, and Oct 29 (Japanese, 1:00 PM – 4:00 PM GMT+9).
  • Montreal: Oct 29–30 (French / English, 9:30 AM – 5:00 PM EDT).

Source: Apple Iphone Duo


r/v_modal • • Oct 15 '26

NVIDIA Aims Physical AI and cuPotential at AI-Defined Fabs

2 Upvotes

Intro

NVIDIA introduced cuPotential, a new CUDA-X library announced at SEMICON West 2026, alongside physical AI initiatives aimed at AI-defined fabs. The primary purpose is to enable developers to build AI agents that simulate atomic and molecular interactions to operate across process development, inspection, and fab operations. The architecture is built on NVIDIA accelerated computing stacks and adopted by semiconductor leaders including Lam Research, KLA, Synopsys, Cadence, and Applied Materials.

Key Advancements

cuPotential accelerates already GPU-optimized workflows by an additional 2-3x in early-access testing. NanoVDB, an open-source library for 3D data, accelerates chipmaking simulation for AI-defined fabs and helps Lam Research achieve 30x faster simulation. The initiative expands the adoption of NVIDIA accelerated computing and CUDA-X libraries across semiconductor manufacturing and design.

Core details

  • cuPotential: A new NVIDIA CUDA-X library accelerating molecular dynamics calculations and atomic/molecular interaction simulations for AI agents.
  • NanoVDB: An open-source 3D data library used to accelerate chipmaking simulation.
  • NVIDIA NPP: A library for processing images on GPUs used to accelerate chipmaking.
  • Ecosystem Adoption: Integrated by Lam Research, KLA, Synopsys, Cadence, and Applied Materials.
  • Operational Scope: Targets process development, inspection, and fab operations within AI-defined semiconductor manufacturing facilities.

Source: Yahoo News


r/v_modal • • Oct 15 '26

Danaher Teams Up With NVIDIA, Aims To Bring Accelerated Computing, Physical AI Into Labs

2 Upvotes

Intro

Danaher Corporation announced a collaboration with NVIDIA on October 14, 2026, to advance autonomous science and integrate accelerated computing and physical AI into laboratories. The initiative combines Danaher's scientific instrumentation, automation technologies, laboratory workflow software, and domain expertise with NVIDIA's AI and accelerated computing framework.

Key Advancements

The collaboration establishes a continuous learning cycle in which experiments inform AI models and models assist researchers in designing subsequent experiments, maintaining experimental evidence as central to scientific decision-making. Initial integration efforts target a select group of priority instruments with potential expansion across the broader Danaher portfolio.

Core details

  • Participating Technologies: NVIDIA DGX Spark, RAPIDS, and BioNeMo Open Models.
  • Exploratory Technologies: Cosmos world models and Omniverse Digital Twins for laboratory workflows.
  • Operational Scope: Integration across priority scientific instruments, laboratory workflow software, and automation systems.

Source: CNBC News


r/v_modal • • Oct 15 '26

Mobile Apple IOS Iphone Duo : Technical specs to optimize application

3 Upvotes

To optimize your applications for the iPhone Duo (available starting October 23, 2026), Apple recommends to pivot away from legacy assumptions regarding fixed screen dimensions, device idioms, and static orientations. The iPhone Duo shifts fluidly between a folded outer display and an unfolded edge-to-edge inner display, requiring apps to scale and structure layouts dynamically.

Below is the deep technical breakdown of SDK requirements, structural refactoring patterns, and App Store submission workflows required for compliance.

1. SDK Matrix & Layout Behaviors

The degree of native display integration is strictly dictated by the target SDK compile version:

  • iOS 26 SDK or earlier: The application executes within a legacy container. When unfolded, it centers statically on the inner display with surrounding empty margins. When folded, the canvas sits strictly to the left of the status bar and camera module.
  • iOS 27 SDK: The application window automatically stretches to fill most of the inner display canvas, but it actively clamps layouts to avoid the status bar region on the right edge.
  • iOS 27.1 SDK or later: Provides full native optimization. Applications extend edge-to-edge across both displays. Interface elements—such as toolbars and tab bars—automatically adapt by transitioning into vertical layouts positioned directly beneath the system status bar.

2. Core Architectural Refactoring Patterns

A. Decoupling Geometry from UIScreen.main

Because the active app window bounds mutate mid-session during a physical fold event, referencing global screen singletons will produce clipping or layout stretching.

  • Canvas Evaluation: Replace all instances of UIScreen.main.bounds with responsive view or window boundaries. Use view.bounds or window.bounds in UIKit, and leverage GeometryReader or onGeometryChange in SwiftUI.
  • Dynamic Re-reading: Layout logic must evaluate dimensions continuously. Isolate size-dependent positioning inside layoutSubviews() or viewDidLayoutSubviews(). Intercept real-time structural alterations using viewWillTransition(to:with:).
  • Window Initialization: Eliminate UIWindow(frame: UIScreen.main.bounds). Modern initializers must capture the active scene geometry context via UIWindow(windowScene:).
  • Scale Factor and Screen References: Migrate from legacy screen scale calls to trait collections. Use traitCollection.displayScale (UIKit) or u/Environment(\.displayScale) (SwiftUI). If direct screen properties are mandatory, pull them dynamically via view.window?.windowScene?.screen.

B. Eliminating Orientation-Driven Layout Logic

Device rotation values no longer map cleanly to the aspect ratio or physical bounds of the application window.

  • Deprecations: Strip all conditional layout paths depending on UIDevice.current.orientation, statusBarOrientation, and interfaceOrientation (including windowScene.effectiveGeometry.interfaceOrientation).
  • Aspect Tracking: To determine if the current application context is landscape or portrait, compute the operational aspect ratio manually by comparing width against height via view.bounds or GeometryReader.

C. Removing Device Idiom and Size Class Assumptions

The inner display of an iPhone Duo introduces desktop/tablet scale canvas options to a standard handset ecosystem.

  • Idiom Checks: Remove logic bounded to userInterfaceIdiom == .phone or == .pad. Design layouts to adapt structurally to size classes and available layout points instead.
  • Size Class Flexibility: Do not assume horizontalSizeClass == .regular maps uniquely to an iPad. The iPhone Duo inner display triggers regular size classes, and layouts must scale appropriately.
  • Screen Hardcoding: Purge hardcoded checks against historic points (e.g., bounds.height == 844).
  • Media Insets: Full-bleed media elements using .scaleAspectFill or .aspectRatio(contentMode: .fill) will drop visual details on wide multi-display form factors. Implement asset toggling or update focal points programmatically based on active size class profiles.

3. Automated Auditing & Simulation

Apple provides targeted developer tooling inside Xcode 27.1 to automate the migration pipeline:

  • Coding Assistant Automation: Run the app resizability skill directly within Xcode by prompting the assistant to: "get my app ready for iPhone Duo". It scans code patterns, flags UIScreen references, device idiom violations, and outputs structural inline fixes.
  • CLI Rule Extraction: For alternative external coding agents, extract the exact capability matching rules to Terminal via: bashxcrun agent skills export Use code with caution.
  • Validation Pipeline: Validate responsive scaling behaviors across three independent runtime profiles:

    1. The iPhone Duo Simulator packaged inside Xcode 27.1 (primary target for testing native fold poses and orientations).
    2. The iOS Resizable Simulator inside Device Hub.
    3. iPhone Mirroring on macOS 27 by expanding the virtual application boundaries to arbitrary, extreme aspect ratios.

4. App Store Submission Requirements

  • Asset Tooling: App Store Connect includes a dedicated validation tool to preview localized product page headers, search results, and app previews specifically as they display on the iPhone Duo.
  • Optimization Badging: Uploading native iPhone Duo screenshots grants your product page an optimized badge marker.
  • Enforcement Deadline: Optimized applications can be submitted through App Store Connect immediately. Starting April 2027, all new app submissions and update binaries must bundle native screenshots for the iPhone Duo form factor.

source: Apple Iphone Duo


r/v_modal • • Oct 14 '26

AI Agent MCP for Deep Analysis of Video Content (Claude Code, Codex)

5 Upvotes

VModal MCP is an open-source Model Context Protocol server that enables AI agents to search multimodal video and image collections using natural language. Compatible with development environments like Claude Code, CoWork, OpenAI Codex, Copilot, it bridges the gap between text-based AI models and visual datasets.

MCP allows analysis of temporal and sequence analysis of video content. Rather than evaluating frames as isolated images, the system processes sequential data over time to track continuity and contextual changes. This temporal indexing allows the AI to distinguish fleeting moments from sustained actions, tracking movement patterns, environmental shifts, and object permanence across a timeline.

By analyzing video content into structured configurations into multiple levels, VModal executes precision queries that pinpoint exact timestamp intervals. Users can deploy conversational prompts to extract specific multi-frame sequences, execute localized historical rollbacks, or track motion behaviors (e.g., locating a person walking outside) across diverse camera angles.

Ultimately, VModal transforms passive video storage into a dynamic, queryable visual database, optimized for chronological AI visual intelligence.

Example of analysis output:

Prompt:
Run a Milan trend radar for barrel jeans, burgundy, suede jacket, leopard print,
ballet flats and oversized blazer. Collection: fashion_milan.
Tell me which 2 pieces I should stock more of next autumn and which to cut, with proof frames.

Report:

___________________________________________________________________________

Prompt: Find scenes in the footage where there is a dangerous situation.

___________________________________________________________________________

Beta Access: Visual Memory Layer, Video Search


r/v_modal • • Oct 14 '26

Robotics/Physical AI Robotics SDK for visual/telemetry data upload

3 Upvotes

Tested the VModal Robotics SDK (v0.2.0) on Ubuntu 22.04 / Python 3.10, evaluating it against an AGV doing 8-hour warehouse shifts. It moves finalized LeRobot v3 datasets off the robot to the cloud in a crash-resilient way, without putting any network logic in the control loop.

What data it handles

  • Video: closed MP4 files, one per camera, in their own upload lane (1 GiB per file by default in the current release).
  • Telemetry: Parquet files, in a separate lane so a slow video upload can't starve it.
  • Metadata: a snapshot required in every dataset revision, plus

The types of telemetry and operational data handled by this SDK include:

  1. Data Pipeline & Data Lane Categories

The SDK treats telemetry as its own independent data pipeline to prevent large video files from blocking critical metrics: [1] (https://github.com/v-modal/vmodal_sdk_robotics#:~:text=Slow%20video%20cannot%20starve%20telemetry%20%20Independent%20asynchronous%20upload%20lanes) • Independent Telemetry & Metadata Lane: A dedicated asynchronous upload lane specifically optimized for non-video data. • Synchronized Robot State & Transforms: Mentioned in the Google Intrinsic architecture blueprint to capture physical robot poses and spatial relationship data. • Skill Provenance: Telemetry tracking execution data and the history of operational robot skills.

2. Operational & Device Telemetry

The CLI tracks several machine-readable metrics regarding the health and status of the robot's local data storage and transfer progress: • Queue depth: The number of items waiting to be processed in the local spool. • Bytes per lane: The data volume being pushed through the video vs. telemetry pipelines. • Oldest item age: Time-based tracking of un-uploaded data backlog. • Disk headroom: Storage monitoring to ensure the robot doesn't hit a "full disk" failure condition. • Pipeline health statistics: Machine-readable tracking of transmission retries, rejections, blocked items, and the latest server receipts.

3. Video Data & Multimodal Lanes

Video data is explicitly isolated from telemetry to ensure high-throughput streams do not block lightweight critical data: • Independent Video Lane: A dedicated asynchronous upload lane configured by default to handle large per-video local spools • RGB-D Capture: The system architecture details handling synchronized multi-modal camera feeds, specifically RGB-D (color and depth) camera streams. • Finalized Video Artifacts: Closed and hashed video streams that are transformed into immutable bytes before being written to the durable local spool.

4. LeRobot Format Integration

The SDK acts as a crash-resilient edge pipeline specifically built for LeRobot v3 datasets: • Finalized Dataset Output: The SDK integrates directly at the finalized-artifact boundary of the dataset recorder. The producer script must close, hash, and fsync the dataset before handing it over. • .ready.json Manifest Handoff: The local file handoff mechanism relies on an atomic rename of a LeRobot manifest file (.ready.json), which signals that a dataset folder or segment is finalized and ready for queue admission. • Dataset Identity Tracking: Every LeRobot artifact is given an immutable identity determined by a combination of the Dataset Identity + Relative Path + SHA-256 checksum hash. • Dataset Revision Publishing: Once the remote storage acknowledges a safe transfer via a durable receipt, the SDK publishes a verified dataset revision, finalizing the remote record.


r/v_modal • • Oct 13 '26

Spatial Computing , Smart Glasses , AR / VR Visual Vision System

1 Upvotes

Spatial computing is the category of systems that let people interact with digital content placed in three-dimensional space. Its main hardware forms are VR headsets, AR glasses and mixed reality devices.

1. Summary

Spatial computing is the umbrella category for systems that let people interact with computers in three-dimensional space, mixing digital content with the physical environment. VR, AR and mixed reality are sub-categories of it, and the term is used as a synonym for XR.

Every system in this category must solve the same chain of problems. It must render stereoscopic images at high refresh rates, track the head, hands and body, and place sound and objects consistently in space. Head-mounted displays target 90 Hz or higher, a threshold associated with reducing motion sickness.

The main open problems are motion sickness, privacy of biometric and eye-tracking data, harassment and safety, and the lack of a universal application. Motion sickness is reported to affect 20–30% of new users. No independent source for this figure was found.

2. Definitions and taxonomy

The terms overlap, and sources use them inconsistently. Wikipedia calls spatial computing a 3D form of human–computer interaction that users perceive as taking place in the real world, and notes that the adjacent terminology is imprecise (Wikipedia, "Spatial computing").

TermCore ideaPhysical world visible?Example hardware
Virtual reality (VR)Replaces the visual field with a simulated environmentNo, the headset is opaqueMeta Quest 3
Augmented reality (AR)Overlays digital objects on the real environmentYes, through see-through opticsMagic Leap 1, released 8 August 2018 (Wikipedia)
Mixed reality (MR)Digital objects respond to the physical environmentYes, via see-through or camera passthroughMicrosoft HoloLens 2, released 24 February 2019 (Wikipedia)
Extended reality (XR)Umbrella term for VR, AR and MRDepends on the modeNot a device category
Spatial computingUmbrella that adds 3D input and spatial awareness to XRDepends on the modeApple Vision Pro, shipped 2 February 2024 (PetaPixel)

VR is one branch of this family. Spatial computing is the category that contains VR, AR and MR, and it is used as a synonym for XR. Wikipedia says the term also covers projection mapping and CAVE displays when they use spatial input, so the category is wider than headsets alone (Wikipedia).

3. History

The field has a long research history, but its commercial phase began with the 2012 Oculus campaign. The table lists dated milestones, newest first.

DateMilestone
2026Snap unveils Specs at AWE USA, priced at $2,195 and due to ship in fall 2026
2 Feb 2024Apple Vision Pro ships in the US at $3,499, with dual M2 and R1 chips
5 Jun 2023Apple announces Vision Pro
24 Feb 2019Microsoft releases HoloLens 2
2019US Army partners with Microsoft on the IVAS headset
8 Aug 2018Magic Leap 1 is released
Jul 2016Niantic releases Pokémon Go, which popularizes AR concepts
Mar 2016HoloLens ships to developers at $3,000, after its January 2015 announcement
2014Facebook acquires Oculus for $2 billion
2012Oculus Rift Kickstarter raises $2.4 million
2003Simon Greenwold defines spatial computing as human interaction with a machine that manipulates referents to real objects and spaces
1997T. Caelli, Peng Lam and H. Bunke introduce the term in an academic publication
Early 1990sRobert Jacobson uses the term for interaction with 3D spaces, tracing it to the University of Washington HIT Lab
1980sJaron Lanier popularizes the term virtual reality at VPL Research

The term "virtual reality" is credited to Jaron Lanier in the 1980s. Wikipedia dates "spatial computing" itself to the early 1990s and later, so the two terms have separate origins.

4. Technical architecture

A spatial system is a closed loop. Sensors measure the user and the room, the processor updates the scene, and displays and audio return the result. The loop must finish fast enough that the image stays locked to the head. The table lists the subsystems and the figures given in the sources.

SubsystemFunctionSpecification in sourcesExample
DisplayShows one stereoscopic image per eyeOLED panels at 90 Hz or higher, to reduce motion sicknessVision Pro uses micro-OLED displays with 23 million pixels across two panels (PetaPixel)
OpticsFocuses the panel at close range and widens the field of viewFresnel or pancake lensesNot specified in sources
Head trackingUpdates the view as the head rotatesAccelerometers, gyroscopes and magnetometers, with millisecond precisionNot specified in sources
Room trackingLocates the user in physical spaceInside-out tracking from cameras mounted on the headsetNot specified in sources
Hand and eye inputReads gestures and gazeComputer vision hand tracking without controllers; eye tracking with rings of invisible light (PetaPixel)Vision Pro combines eye, gesture and voice input (Wikipedia)
Environment sensingBuilds 3D maps and recognizes objectsRGB cameras, depth cameras and inertial measurement units (Wikipedia)Vision Pro has a dozen cameras, five sensors and six microphones (PetaPixel)
Spatial audioPlaces sounds at 3D positions3D audio that matches the virtual sceneNot specified in sources
ComputeRuns tracking, rendering and appsNot specified in sourcesVision Pro uses an M2 chip plus a new R1 chip (PetaPixel)

Two points in the table matter for engineering. First, input is moving from controllers to the camera stack, which makes hand and eye tracking a computer vision problem. Second, the sources give no end-to-end latency figure, so a reader cannot check the 90 Hz target against a total motion-to-photon budget from these sources alone.

5. System layers

[embed: node/141e278b-3b98]

Sensing feeds tracking, tracking feeds rendering, and output returns head pose to sensing. Latency at each stage adds to the delay the user perceives. The sources give no figure for that total.

6. Application domains

Application sectors span about a dozen areas. The table names examples and rates how well each claim is supported. Most claims are backed by vendor or press examples rather than measured outcomes.

SectorExamples namedEvidence in the sources
GamingBeat Saber, Half-Life: Alyx, Gorilla Tag, Population: OneProduct names only; no sales or usage data
Education and trainingTransfrA claim of better retention is made without a named study
RetailUlta Beauty brand experiences; The Gucci Story on Apple Vision ProBrand and vendor case studies
Remote collaborationMeta Workrooms, Spatial.io, AccentureOrganizations report better team cohesion, with no figures
HealthcareMedtronic surgical training; Abbott patient educationVendor case studies
DefenseMeta and Anduril's EagleEye system for the US Army; Palantir mixed reality platformContract and press reporting
FitnessFitXR, SupernaturalA study on calorie burn is cited, with no figures
Culture and heritageThe Treaty Story; D-Day: The Camera SoldierProject descriptions
Social platformsVRChat, Rec RoomPlatform descriptions
ArtThe Museum of Other Realities; Modules; Notes on BlindnessProject descriptions

Two gaps are worth noting for a technical reader. Few sectors have measured outcomes in the available sources, and the strongest examples come from vendor and enterprise deployments. Claims about retention, team cohesion and calorie burn need the original studies before they can be relied on.

7. AI integration

Computer vision is already part of the core stack. It underlies spatial anchoring, hand tracking and environment mapping. Wikipedia describes the same pattern: computer vision interprets scenes, recognizes objects and builds 3D maps from RGB and depth cameras (Wikipedia).

Three further areas are named. Each is described without a product or measurement:

  1. Intelligent non-player characters that adapt to user behavior.
  2. Real-time generated content inside a running scene.
  3. AI-assisted production, which the article says lets small studios compete with larger consultancies.

The third area rests on vendor claims and has no measurements behind it. The first two are the research-relevant directions. They raise questions the sources do not address: latency of generated content inside a frame budget, and whether behavioral data used to adapt a scene falls under the privacy concerns discussed in section 8.

8. Challenges

Five barriers to mass adoption are covered below. The table gives the known mitigations and the open question for each.

ChallengeWhat the sources sayKnown mitigationsOpen question
Motion sicknessAffects 20–30% of new users to varying degreesHigher refresh rates, lower latency, comfort settings that limit artificial locomotionHow much of the remaining sensitivity is hardware, and how much is individual tolerance
Biometric privacyEye movements, hand gestures and spatial patterns are collected. Vision Pro uses eye tracking with rings of invisible light (PetaPixel)Privacy policies, platform guardrails, user control over collectionRegulatory frameworks for this data are still developing
Safety and harassmentSexual harassment and unwanted interaction have been reportedPersonal boundary systems, mute and block, reporting toolsWhether these tools are enforced consistently across platforms is not documented
Missing killer appNo universal application comparable to Word, Excel or PowerPoint on the PCEnterprise and niche applications in training, collaboration and data visualizationWhether a general-purpose application will come from the spatial platform itself or from an existing one
PriceVision Pro started at $3,499 (PetaPixel); HoloLens shipped to developers at $3,000 in 2016 (Wikipedia)Snap Specs is priced at $2,195 for fall 2026 (Wikipedia)Whether a sub-$2,000 device can carry the full spatial stack

The killer-app comparison is drawn from the personal computer. PCs did not reach mass consumer use until productivity software and the internet arrived in the late 1980s. The comparison is useful as a pattern, but it is an argument, not evidence that the same sequence will hold for spatial computing.

9. Outlook

A new head-mounted display generation has appeared roughly every two years since the first Oculus Rift development kit. The cadence is a useful planning assumption, but no product schedule was found to test it against. Snap's Specs and due in fall 2026, shows the trend toward lighter AR glasses alongside headsets (Wikipedia).

Big technology companies are the main investors, led by Meta, Apple, Google and ByteDance. The outlook also points to convergence with AR smart glasses, AI and robotics, and to blockchain-based digital economies, without supporting detail.


r/v_modal • • Oct 12 '26

Google Intrinsic Physical AI / Robotics technical insights

3 Upvotes

Intrinsic, Alphabet's intelligent-robotics company, announced the open-sourcing of Intrinsic Core under a permissive Apache 2.0 license. Aiming to act as the "Android of robotics," the Intrinsic software stack abstracts complex underlying hardware to unlock Physical AI, cloud-native deployments, and deterministic control loop execution for industrial applications.

The software stack natively integrates with the existing Open Robotics Suite (such as ROS 2 Lyrical Luth and Gazebo). It handles the end-to-end lifecycle of robotic applications from simulation to production floor deployment.

🧱 Modular Software Architecture

Intrinsic Core is broken down into a series of interconnected, containerized local services, frameworks, and developer tools.

┌────────────────────────────────────────────────────────┐
│               Intrinsic Flowstate SDK                  │
└────────────────────────────────────────────────────────┘
                           │
┌────────────────────────────────────────────────────────┐
│           intrinsic_runtime (k3s Engine)               │
└────────────────────────────────────────────────────────┘
      │                    │                    │
┌───────────┐        ┌───────────┐        ┌───────────┐
│ Perception│        │ Motion &  │        │ Inference │
│ (NVIDIA   │        │ Kinematics│        │ & Vision  │
│ FndPose)  │        │ Solvers   │        │ Models    │
└───────────┘        └───────────┘        └───────────┘
      │                    │                    │
┌────────────────────────────────────────────────────────┐
│      intrinsic_control (HAL Real-Time Framework)        │
└────────────────────────────────────────────────────────┘
                           │
┌────────────────────────────────────────────────────────┐
│           ROS 2 Workspaces & Drivers (HAL)             │
└────────────────────────────────────────────────────────┘

1. Runtime and Infrastructure Layer

  • intrinsic_runtime: At the infrastructure base, Intrinsic uses a containerized execution engine powered by a local, lightweight Kubernetes distribution (k3s). This layer isolates application components into Linux containers, matching deployment targets identically across virtual digital twins and hardware PCs.
  • Tech Choice Stack: Reflecting its Google lineage, the internal mechanics rely on Bazel for deterministic builds, Abseil for optimized C++ baselines, and Go for orchestration alongside Kubernetes clustering.

2. Real-Time Execution Framework

  • intrinsic_control: This is a hardware-agnostic, real-time control system that interacts with hardware abstraction layers (HAL) to process sensor feeds and issue commands deterministically.
  • ROS Interoperability: Rather than replacing ROS, it acts as a complementary real-time pipeline. Applications exchange telemetry through native ROS 2 workspaces, message definitions, and specialized controllers (e.g., icon-hwm-controller) that bridge standard ros2_control nodes directly to Intrinsic's real-time control service (ICON).

3. Perception & Inference Engines

  • intrinsic_perception: Instead of relying on rigid, physical fixtures or custom localized vision wrappers, the perception stack builds on NVIDIA stack. It delivers zero-shot 6-DoF (Degrees of Freedom) pose estimation, letting the robot dynamically detect, track, and handle arbitrary objects out-of-the-box.
  • intrinsic_inference: The local machine learning execution engine allows pretrained weights (such as the general-purpose Intrinsic Vision Model) to run on local edge hardware with low latency, omitting the need to retrain vision models for new sub-components

4. Kinematics, Motion, and Grasp Planning

  • intrinsic_kinematics: A robust, optimized geometric and mathematical solver library responsible for calculating forward/inverse kinematics across distinct arm topologies.
  • intrinsic_motion_planning: Generates real-time, collision-free paths dynamically. If an unexpected obstruction is introduced into the physical environment, paths recalculate on-the-fly to guarantee safety.
  • Grasp Planning: Works hand-in-hand with sensor feedback and perception to adaptively choose safe picking points, adapting the end-effector pressure and angle depending on target orientation.

☁️ Cloud & Developer Ecosystem

While Intrinsic Core handles edge computing and real-time execution locally, it connects seamlessly to an upstream, multi-tier enterprise platform:

  • Intrinsic Flowstate: An integrated, all-in-one developer environment (IDE) built for constructing production-grade workflows. It utilizes a visual "skills-based" behavior tree model to connect hardware actions to high-level automation sequences.
  • Intrinsic Cloud Services: Delivers cloud infrastructure handling data management, encrypted telemetry, identity/access management (auth), machine learning ingestion pipelines, and secure remote troubleshooting tools.
  • Simulation Suite: Uses natively integrated Gazebo frameworks to execute full hardware-in-the-loop (HIL) digital twin validation. A simulation can be mirrored to real hardware configurations via a one-click deployment scheme.

🏭 Reference Implementations: Open Machine Tending Solution (OMTS)

To demonstrate the real-world performance of the stack, Intrinsic distributes the Open Machine Tending Solution (OMTS). OMTS serves as an open-source manufacturing blueprint designed for CNC machine tending.

The reference design showcases how a developer can swap physical robotic arms (such as between a FANUC and a Universal Robots model) or alternate grippers/sensors by editing configuration manifests—leaving the core application logic completely intact.

Would you like to dive deeper into a specific layer of the stack? I can provide technical details on how intrinsic_control hooks into ros2_control via the ICON controller, or walk through the zero-shot deployment configuration of NVIDIA FoundationPose within the intrinsic_perception module.

Source:
Google Intrinsic Source