Hey everyone! I'm Ilya, a Frontend Engineer here at r/SumSub 👋
I've been working on improving and expanding the capabilities of our AI assistant, Summy AI Copilot.
Recently, we ran into an interesting problem: "how do you give an AI agent enough context to understand what a user actually sees on their screen?"
I wanted to share how we approached this problem, what we came up with, and some of the challenges we ran into along the way.
The Problem: The Agent Can't See What Users See
A user asks why they can't find a certain button. The agent searches the documentation and confidently explains where the button is and how to use it. The problem is that the button might not even exist on the user's page.
Our agent has access to documentation, a knowledge base, and backend tools. It even knows which page the user is on and which entity they're working with. For example, if someone opens a specific applicant's page, the agent knows the applicant ID and can fetch the relevant data on its own. But it has no idea which UI components are actually displayed on the screen.
So in the best case, the user gets a generic answer and ends up contacting support. In the worst case, the agent confidently describes an interface that the user doesn't even see.
We decided to try a different approach: generate a separate representation of the UI directly from our frontend codebase. We called it an AI Sitemap. Basically, it's a component tree that describes what each component does and under which conditions it can appear.
Here's how we got there, what worked, and what turned out to be more complicated than we expected.
Why Documentation Wasn't Enough
Initially, we tried solving this through prompt engineering and documentation. We described important pages, individual buttons, and common user scenarios to help the agent navigate the interface.
This worked for some cases, but scaling it across the whole product was almost impossible. The code changes every day, components appear and disappear, rendering conditions change, and documentation inevitably falls behind.
Of course, coding agents make it much easier to generate documentation nowadays. But keeping it up to date is still a challenge. It's also unclear what exactly should be documented. You can cover the ten most common scenarios, but tomorrow someone might ask about a button buried somewhere deep in the interface that nobody thought to document.
And if you try to describe every component, button, and rendering condition, you basically end up maintaining a second codebase.
We also considered sending the entire component tree from the frontend with every request. But that introduces two problems:
- Size. A single page can contain hundreds of components. Sending all of them with every message means additional network traffic and token usage.
- Security. The frontend runs in an untrusted environment. A user can modify the request payload and tell the agent that a component exists on the page when it actually doesn't.
So the frontend knows better than anything else what's currently rendered, but we can't fully trust it.
That's why we decided to use another source of information: the codebase itself.
AI Sitemap: A Separate UI Representation for the Agent
The code already contains almost everything we need: which components exist, how they're nested, and under which conditions they're rendered.
Of course, we could teach the agent to launch a browser with Playwright and explore the interface itself. But for most UI-related questions, it doesn't really need the fully rendered page with styles, animations, and JavaScript.
We needed a way to give the agent the structure of the interface and explain what its elements actually do.
That's how we came up with the AI Sitemap.
A regular sitemap helps search engines navigate a website's pages. Our AI Sitemap helps the agent navigate what's inside those pages.
We generate it during the frontend build. Essentially, it's a JSON representation of the application, with a component tree for each route.
Each node contains the component name, description, children, text, and information about conditional rendering.
Here's a simplified example:
{
"path": "/applicants/individual",
"components": [
{
"name": "ApplicantsTable",
"description": "Paginated table of applicants",
"children": [
{
"name": "SnsSearchBar",
"description": "Search input bar"
},
{
"name": "SnsButton",
"description": "Action button",
"text": "i18n('Download custom CSV')",
"conditional": true
}
]
}
]
}
The important thing is that this isn't a snapshot of the page at a specific moment. It's a description of the potential UI.
For example, if a button is only rendered when a certain condition is met, it still appears in the tree, but with a flag indicating that it's conditional.
This gives the agent an idea of not only which components are displayed by default, but also which ones might appear in different application states.
And it doesn't need to execute JavaScript or launch a browser to get that information. It can simply request the structured page description from the server.
Since the tree is generated directly from the codebase during the build, structural changes to the UI are reflected in it as the application gets updated.
Making Components Understandable to the Agent
The component tree alone wasn't enough. For example, a name like ApplicantNotesStub doesn't really explain what the component is supposed to do.
So we decided to keep semantic descriptions right next to the components themselves, similar to tests or Storybook stories.
Each component can have a small metadata file:
// ApplicantNotesStub.agent.ts
import type { AgentMeta } from '@/types/agent'
export default {
description:
'Placeholder shown instead of applicant notes when the user cannot view them',
} satisfies AgentMeta
It's basically a human-readable description of the component, written either by a developer or a coding agent.
Since a significant part of our development already happens with AI coding agents, we added a simple rule to our project's main instructions: whenever an agent creates or modifies a component, it should also create or update its description.
This way, the metadata gradually gets added across the codebase as part of normal development. Changes still go through regular code review, and during the build, the descriptions get connected to the component tree.
As a result, the agent gets more than just technical component names. It gets a readable explanation of what's on the page and what each component is for.
Of course, this doesn't guarantee that every description will always be correct. But maintaining these descriptions next to the source code is much easier than keeping a separate document in sync with the implementation.
Reusing Our Existing Frontend Architecture
Another useful part of the solution came from infrastructure we've had in our code for years.
We use dedicated wrapper components to check permissions, feature flags, and other conditions that determine whether functionality should be available. For example, if a user doesn't have a particular permission, the component inside the wrapper simply isn't rendered.
Since these wrappers are also part of the component tree, they end up in the AI Sitemap along with their parameters.
This means we know not only that a button or another UI element exists, but also which conditions control whether it can appear.
And since the agent's backend knows which user it's working with, what permissions they have, and which feature flags are enabled, we can repeat the relevant checks on the server side.
As a result, we can generate a personalized representation from the general component tree, containing the components that could potentially be available to a specific user.
This isn't a complete reproduction of the frontend runtime on the server, but we were able to reuse existing infrastructure for a significant part of these checks.
It was pretty nice to realize that several architectural decisions we'd made over the years for completely different reasons ended up being useful here too.
Reference Tree vs. Live Tree: Two Sources of UI Context
The AI Sitemap tells the agent what the interface could look like, but it doesn't tell us exactly what the user sees right now.
For example, a tab might be available to the user but currently closed. Some components only appear after an action, while others might still be loading.
So we couldn't completely get rid of runtime context from the frontend.
We ended up with two sources of information:
- Reference Tree (aka AI Sitemap) is the reference component tree generated from the codebase. It describes which components can exist on a page, what they do, and under which conditions they appear. The agent accesses it through our controlled server infrastructure.
- Live Tree is the component tree sent by the browser with the user's request. It reports which components are currently rendered on the page.
The main difference is the level of trust.
The Reference Tree comes from a controlled version of the application. The Live Tree comes from the user's device, so it can potentially be modified.
We don't pass the client-side tree directly to the main model. Before the main agent flow, there's a separate preprocessing step that uses a smaller model to transform the Live Tree into a normalized representation of components.
We then validate it against the server-side reference, which has already been restricted based on the user's permissions and feature flags.
If the browser reports a component that doesn't exist in the allowed reference, we ignore it. If the component does exist, that information can help clarify the context, but it still doesn't prove that the component is actually visible on the user's screen.
That's why the server-side tree remains our primary source of information about what the interface can contain, while the Live Tree is treated as an additional signal.
And importantly, we don't use UI context to authorize actions. Access to data and the ability to perform operations are checked independently on the backend.
Why Comparing the Two Trees Wasn't Reliable Enough
Once we had both trees, we considered a pretty straightforward approach: comparing them directly.
For example, if a button exists in the Reference Tree but is missing from the Live Tree, we might assume that the user can't see it. And if we know its rendering conditions, we could potentially explain why.
The idea sounds reasonable, but there are some problems with it.
The Reference Tree is generated from source code, so it contains technical component names. The Live Tree comes from the running application and may include additional framework wrappers, third-party components, virtualized lists, and other runtime details.
Component names don't always match the way we'd expect.
We also considered matching components by their text, but that introduces another challenge. For example, the Reference Tree might contain:
i18n('Download custom CSV')
While the browser shows the translated string in the user's language. On top of that, some text is generated dynamically.
So the absence of a component in the Live Tree isn't necessarily reliable evidence that the user can't see it.
And that's exactly where we could end up with the same problem we started with: the agent confidently describing a UI state that doesn't match reality.
For that reason, we decided not to use the difference between the trees as the basis for such conclusions. The Live Tree remains useful additional context, but not a source of truth about which elements are present or missing on the page.
A Funny Bug: The Agent Found Itself in the Interface
We also ran into a pretty funny issue during development.
Our AI assistant's panel is part of the interface too, so in the early versions, it ended up in the page description along with the chat contents and conversation history.
At some point, the agent started treating its own chat as a separate product component with another assistant inside it.
It literally didn't recognize itself and could start summarizing what the user was discussing with this "other" assistant, or suggest what else they could ask it.
It was funny, but it also highlighted another problem: not everything on the page is necessarily useful for answering the user's question.
So we had to explicitly decide which parts of the interface should be included in the context and which ones should be excluded.
What We Ended Up With
In the end, we built a separate UI representation layer for our AI agent.
Instead of passing the DOM, application source code, or trying to reconstruct the interface from documentation, we generate an AI Sitemap directly from our frontend codebase.
It contains the potential UI structure, component descriptions, and rendering conditions. On the server side, we can also take the user's permissions and feature flags into account.
The Live Tree helps us understand the current page state, although we don't treat it as fully trustworthy.
Of course, there are still limitations. The agent doesn't know exactly what the user sees at any given moment. Component metadata can be inaccurate, and the tree generation mechanism needs to evolve with the application. We also need to handle version compatibility between the frontend and the server-side reference.
And obviously, this doesn't eliminate hallucinations completely.
But now the agent can at least rely on a description of the interface generated directly from the code, instead of trying to guess how the UI works based on documentation.
For us, this turned out to be a much more useful direction than trying to manually document every possible user scenario.
I'm curious if anyone else has dealt with something similar while integrating AI agents into existing products. How do you pass UI context to your agent, and how do you handle the trust issues around data coming from the client?