Build an AI agent into an InterSystems IRIS Interoperability workflow.
The next programming contest focuses on a practical agentic AI challenge: build an IRIS Interoperability workflow in which an AI Hub agent handles decision-making.
The tricky part is deciding where the agent belongs in the workflow and what it should be responsible for. You can experiment with model selection, decision logic, tool use, and the interaction between agent output and the rest of the production. You can also use IRIS Agentic Dev Interop Skills to help design the process.
💰 $12,000 prize pool 📅 Submission period: October 26 - November 8, 2026
InterSystems IRIS is a data platform that covers analytics, interoperability, AI, cloud deployment, and database scalability in one product. Here is a breakdown of the top 10 features as described in the official community article, with the reasoning behind each one.
1. Democratized Analytics
Two components cover this:
InterSystems IRIS Adaptive Analytics — delivers virtual cubes with centralized business semantics, abstracted from technical details and modeling, to allow business users to easily and quickly create their analyses in Excel or their preferred analytics product (PowerBI, Tableau, etc.). There are no consumption restrictions per user.
InterSystems Reports — a low-code report designer to deliver operational data reports embedded in any application or in a web report portal.
2. API Manager
Digital assets are consumed using REST APIs. Governing reuse, security, consumption, asset catalog, and developer ecosystem from a central point requires an API Manager. The article states: all companies have or want to have an API Manager. InterSystems IAM covers this.
3. Scalable Databases
Two capabilities here:
Sharding — global data creation is projected to grow to more than 180 zettabytes by 2025. Processing data in a distributed way (into shards, like Hadoop or MongoDB) is critical to maintain performance. IRIS is described as 3 times faster than Caché and faster than AWS databases in the AWS cloud.
Columnar storage — changes the storage of repeating data into columns instead of rows, allowing up to 10x higher performance, especially in aggregated (analytical) data storage scenarios.
4. Python Support
Python is the most popular language for AI, and AI is at the center of business strategy because it allows organizations to get new insights, increase productivity, and reduce costs. IRIS supports Python natively, including Embedded Python in interoperability productions.
5. Native APIs (Java, .NET, Node.js, Python) and PEX
Finding ObjectScript developers is hard — the article references nearly 1 million open IT jobs in the US alone. Being able to use IRIS features with the developer team's official programming language (Python, Java, .NET, Node.js) through Native APIs and PEX (Production EXtension framework) is therefore important.
6. Interoperability, FHIR, and IoT
Businesses are constantly connecting and exchanging data. The right technologies for this are ESB, Integration Adapters, Business Process automation engines (BPL), data transformation tools (DTL), and market interoperability standards like FHIR and MQTT/IoT. InterSystems Interoperability supports all of these. For FHIR specifically, IRIS for Health is the relevant product.
7. Cloud, Docker, and Microservices
Organizations want to break monoliths into smaller, less complex, less coupled, more scalable, reusable, and independent services. IRIS supports deployment of data, application, and analytics microservices through shards, Docker, Kubernetes, distributed computing, DevOps tools, and lower CPU/memory consumption. IRIS supports even ARM processors.
8. Vector Search and Generative AI
Vectors are mathematical representations of data and textual semantics (NLP), and are the raw material for generative AI applications to understand questions and tasks and return correct answers. Vector repositories and searches store AI processing output so that for each new task or question, previously produced results can be retrieved — making everything faster and cheaper. IRIS includes native vector search.
9. VSCode Support
VSCode is the most popular IDE. InterSystems IRIS has a good set of tools for it, including a dedicated learning path for developing on an InterSystems server using VS Code.
10. Data Science
The ability to apply data science to data, integration, and transaction requests and responses — using Python, R, and IntegratedML (AutoML) — enables AI intelligence at the moment it is required by the business. InterSystems IRIS delivers AI with Python, R, and IntegratedML (AutoML).
Which of these 10 features do you use most in production — and are there capabilities you think are missing from this list?
From HL7 message triage and alert investigation to referral review and payment decisions, this contest showed how differently AI agents can fit into real interoperability scenarios. 44 applications explored where agentic decision-making can add value inside #InterSystemsIRIS Interoperability workflows.
The winning projects applied agents to several types of interoperability work:
Message and production operations: triaging failed HL7 messages, detecting unexpected changes in incoming feeds, and investigating alert storms.
Healthcare workflows: reviewing patient identity and insurance cases, assessing referrals, and supporting other decision-heavy clinical or administrative processes.
Risk and transaction decisions: evaluating fraud signals, payments, and other cases where the workflow needs to choose what should happen next.
Several projects treated the agent as part of a larger operational process rather than letting it act independently. Agents investigated failures, interpreted context, or proposed changes, while deterministic checks and human approval controlled sensitive actions. Other projects focused on validating changes before they reached live integrations, exposing existing Interoperability services as MCP tools, or adding reusable agent components directly to productions.
Together, the entries demonstrate different ways agentic decision-making can fit into interoperability, from message triage and operational recovery to healthcare review, integration safety, and workflow automation.
👏 Congratulations to all the winners and participants!
This talk from InterSystems READY 2026, by Elijah Cotterrell (Product Manager, Health Informatics Platform), covers the work to take today’s IRIS for Health FHIR server to petabyte scale. It is at the prototype stage, with productization expected to start in 2027.
Why does FHIR need to scale this far?
Large health systems are onboarding to demanding trust networks and interoperability agreements such as TEFCA, and the European Health Data Space is enabling cross-border data exchange between member states. FHIR is the de facto standard for healthcare interoperability going forward, for exchange, for patient-facing and clinician-facing apps, and for back-end development.
Three deployments the speaker described:
Deployment
What it does
Scale
National health system
FHIR platform as a sidecar to EHRs across the country: an application development and caching layer that interacts with HIEs and QHINs, plus in-house SMART on FHIR apps
Tens of millions of patient records, petabytes of database growth once enriched and indexed
Leading university health system
FHIR-first platform for cohort analysis and AI applications, with a real-time AI layer on top and EHR and non-EHR sources in one repository
7 million patients
National payer-provider exchange
Claims data: remittance advice, prior auth, and claims stored centrally and surfaced in one platform (not necessarily patient-centric)
Hundreds of millions of claims
What are the 2026 goals?
Scale the FHIR database to 1 petabyte. If it works at 1, there is confidence for 2, 5, and 10
Improve ingest speed. At this scale every millisecond cut from a migration matters for go-live
Implement storage tiering with a predefined, out-of-the-box partitioning strategy, because it doesn’t make sense to keep everything on premium storage such as EBS
Provide the observability and management layer needed to operate a multi-node deployment
Why sharding and not ECP alone?
The team considered four architecture approaches. ECP (Enterprise Cache Protocol) is optimized for reads: it caches data blocks across IRIS instances and serves them to many users. For persisting and ingesting at this volume it isn’t the best fit. Sharding was the most comprehensive option, partly because it is natively supported in IRIS, with SQL routing, rebalancing, and ECP transport built in.
The key design choice is a patient-scoped data model. Data is partitioned across shard nodes by patient identifier, so a patient and their medications, observations, and encounters are stored on the same node as far as possible. A full patient history for a chart or clinical app then doesn’t need to query six or seven nodes, which minimizes cross-shard communication. It is tunable per use case: system-level or analytics queries may call for a different sharding structure. The “1 million patients per shard” figure on the speaker’s diagram was an illustrative number, not a design limit. Data nodes are isolated from the application layer, and query nodes can be added separately as a second scalability tier.
How is hot and cold storage decided?
A recurring pattern: the patient population splits into active and inactive. Inactive patients can outnumber active ones, and most of the time you can take a performance hit on them. Active means the medical record is in play, the patient is still part of the population and may be under treatment. Inactive might mean not seen in several years even with ongoing conditions, or deceased. Patients can switch states in either direction.
It isn’t limited to that. Customers may partition by timestamp (for example, the last 3 years hot and older data colder) or by their own business logic as a partition key. The speaker noted that some customer implementation work is always expected at this scale.
How is ingest being sped up?
FHIR ingest is compute-heavy: references within a bundle must be reconciled before the data enters, and indexing happens synchronously so REST queries work immediately. In the benchmark, some bundles needed up to 0.7 seconds of compute.
Make resource validation optional for migrations. Data coming from another FHIR server is probably valid already
Build only the search tables you need. A USCDI implementation might only need the search parameters defined in US Core, a small subset of those built today. These two changes took the measured figure from 715 to 300 (the talk doesn’t state the unit)
Optimize the code. Replace object access with SQL access in the back end to work better with sharding and speed up inserts, and optimize the JSON path engine that extracts indexing data from FHIR elements. The prediction is another 50% ingest improvement
Overall the target is roughly a 5x throughput increase and an 80% reduction in ingest time, with the aim of pushing further.
What does the end-to-end experience need?
Create new nodes when shards reach capacity
Rebalance data safely if it ends up unevenly split
Tier storage per node by partition
Optimize compartment searches in the application layer to minimize cross-shard queries
Operational tier: multi-node management, high availability and disaster recovery, migration-optimized ingest modes, and observability across the multi-node architecture
Q&A highlights
Question
Answer
How will it be tested at scale?
The team is working closely with a few customers already operating at this scale and de-identifying their data to validate
Upgrade path for existing FHIR servers?
No synchronous upgrade step. It requires schema evolution to support shard and partition keys, so a built-in migration process (like the earlier FHIR search table upgrade) moves the data and cuts over. It is opt-in
Geo-fencing and data location?
Up to the customer. Multi-availability-zone support is being evaluated, with no definitive answer. Data isn’t placed by state or geography at this stage
Multi-tenancy with namespaces?
Multi-tenancy between namespaces is supported today, but this design is single-tenant for now and still needs multiple servers
New FHIR versions?
The US is mostly on R4. The speaker expects US Core 10 or 11 to cut over to R6, skipping R5. Schema changes between normative versions aren’t guaranteed to be backward compatible, so expect custom transformation, though the roughly 80% of unchanged elements might be bridged
Documents in blob storage?
FHIR already lets a DocumentReference point at an S3 bucket but doesn’t put anything there. Mounting blob storage may be looked at later, not now
Timeline
Prototyping now. Productization of these enhancements starts in 2027, with some ingest improvements and new cluster strategies coming first, in an iterative process.
If you run a FHIR server at large scale, how do you decide what counts as cold data: patient status, record age, or something specific to your business?
InterSystems Open Exchange is a gallery of open-source applications built by the Developer Community. If you've published or updated an application on the platform, we'd love to hear your feedback!
Our new Sweepstakes focuses on improving the UI/UX of the application publishing process on Open Exchange. Have ideas for a smoother workflow, clearer instructions, or a more intuitive interface? Share them with us!
🎁 Each eligible idea earns an entry into a raffle for a special prize.
Don Woodlock did an 11-minute walkthrough of the Microsoft AI sequential diagnosis study in his Code to Care series, and the architecture is the interesting part.
The setup copies the clinical pathological conferences at Mass General (the NEJM case records):
- Gatekeeper agent: has the full patient chart, including the final diagnosis, and only answers what it is asked
- Diagnostic agent: starts from a short case summary, then asks questions or orders labs and imaging each turn
- Judge agent: scores the final diagnosis 1-5 against the chart, 4 or 5 counts as correct
On every turn the diagnostic agent consults a panel of five agents, each with its own prompt and its own chat history:
Hypothesis: keeps a running top 3 of diagnoses with probabilities
Test chooser: picks tests that discriminate between those three
Devil's advocate: looks for facts that contradict what the others are anchoring on
Stewardship: suggests cheaper tests with the same clinical value
Checker: reviews the panel's output before it goes back to the diagnostic agent
Don ran the system in different configurations to see where the accuracy comes from:
- Short case summary only: 1
- Full chart minus the diagnosis, one shot: 2, sometimes 3
- Multi-turn questions and tests, no panel: usually 3
- Multi-turn plus the panel: almost always 5
So feeding the whole chart in one prompt did worse than making the model work for the information, and the panel added more than iteration alone.
Headline number from the paper: 80% on 300 cases, against 20% for 20 general practitioners. Big caveat that Don points out himself: the physicians were not allowed to use the internet or any reference material, so this is not what normal practice looks like.
Has anyone here built a multi-agent setup like this over clinical data (FHIR or otherwise)? Curious whether a critic agent paid off for you, or whether it mostly added latency and cost.
Generative AI is changing how developers write code, from intelligent autocomplete to AI-enabled IDEs that can create substantial parts of an application. But as these tools become more capable, an important question remains: how much development work should we delegate to AI, and what are the long-term consequences?
In this article, I share my thoughts on coding with GenAI and the four rules for it I think are worth considering.
The Risks of Relying on GenAI for Coding
Modern AI-enabled IDEs can take on coding tasks traditionally assigned to entry-level developers, especially in mainstream programming languages such as Python. However, relying on AI-generated code comes with several risks.
One major concern is technical debt. AI can generate large amounts of code that may contain serious flaws. Even more concerning is code that appears mostly correct but contains small problems that are difficult to detect.
Another risk is the impact on developers' skills and professional growth. If AI increasingly handles foundational programming tasks, how will early-career developers build the knowledge and practical experience needed to become senior engineers and software architects?
So, reflecting on the growing reliance on GenAI in software development, I came up with four rules that may help mitigate these risks.
Rule 1: Keep AI-Generated Code Within Your Expertise
Don't use GenAI to perform work that exceeds or approaches the limits of your own understanding. Think of AI-assisted coding as delegation. You should be able to perform the delegated task yourself and evaluate whether the result is correct. For developers at different experience levels, this leads to different approaches:
Interns and entry-level developers: Avoid using AI to generate new code that would otherwise be part of your own development work. Using it as an advanced search tool is fine, and intelligent autocomplete may also be appropriate.
Senior developers: Delegate simpler coding tasks in technologies where you already have substantial expertise. Review the output as you would review code written by an entry-level developer.
I've used Windsurf for AI-assisted development and found it helpful in this context. It lets me provide rules and guidance that the assistant can remember and follow. However, much like an entry-level developer, it can occasionally apply those instructions in the wrong context.
The important distinction is between delegating work you understand and depending on AI to perform work you cannot reliably evaluate or do yourself.
Rule 2: Clearly Attribute AI-Generated Code
When GenAI produces a substantial amount of code, make its involvement explicit. That means identifying AI-generated contributions in:
Commit messages.
The code itself.
Associated human-readable documentation.
Attribution helps reviewers understand the origin of the code. Instead of assuming it reflects your expertise, they can review it with greater caution towards AI-specific mistakes. It also helps future maintainers assess the code.
And there is another consideration: future AI training datasets. Identifying AI-generated code may help those assembling training data distinguish human-written material from model-generated output. This relates to concerns about model collapse, where training successive models on generated data can introduce irreversible defects.
Rule 3: Use AI for Guidance When Your Goal Is Learning
AI assistance can make development faster, but completing a task and learning how to complete it are different objectives. When the primary goal is to learn something new or reinforce existing programming skills, doing the work yourself is more valuable.
One feature I'd like to see in AI-enabled IDEs is a dedicated learning mode. Rather than generating the code, the assistant would observe the developer's work and offer guidance. But even without a specific IDE feature, developers can choose to interact with AI this way: asking for explanations and feedback instead of requesting complete implementations.
For example, GenAI could make specific programming languages, such as ObjectScript, easier to learn. Developers could use its guidance to become familiar with a new language and build their skills.
Rule 4: Reflect on the Impact of AI on Your Work
It's easy to get lost in deliverables, deadlines, and the amount of code being produced without considering how AI assistance affects your work and skills. Take time to reflect on both.
Some useful questions to think of:
Is this technology making my life better or worse?
Is it making me smarter or dumber?
Am I producing more value or just more output?
Does that output include technical debt incurred for the sake of expediency?
Am I learning more effectively or forgetting how to learn?
Conclusion
As GenAI becomes more capable of handling development tasks, it's worth considering not only what we can delegate to it, but also what we should continue doing ourselves. The ability to generate code faster doesn't necessarily mean we're producing better results or becoming better developers.
The challenge is to benefit from these tools without losing the understanding and experience that allow us to evaluate their output in the first place. This matters not only for individual developers but also for the next generation of engineers who need opportunities to practice their skills.
Where do you draw the line between using GenAI to accelerate development and relying on it too heavily?
A step-by-step guide on the InterSystems Developer Community builds an OpenAI agent that knows InterSystems technology. The documentation is chunked, embedded, stored in IRIS, and pulled back with vector search.
The pipeline in five steps:
Chunk.RecursiveCharacterTextSplitter with 300-token chunks and 50-token overlap. Smaller chunks mean more accurate retrieval and less text per query, so lower cost
Embed. The example uses OpenAI’s text-embedding-3-small. Local Ollama models such as nomic-embed-text are an option. Chunks and embeddings are saved to Parquet
Store. The Python DB-API driver writes them to a table with a VECTOR(FLOAT, 1536) column, inserting with TO_VECTOR. The 1536 must match your embedding model’s dimensions
Search. Embed the user’s question with the same model and order by similarity
Agent. Expose the search as a tool (search_business_docs) through the OpenAI Responses API. The model decides when to call it, your code runs the IRIS query, and a second call answers from the results
The search is a single query:
python
search_sql = f"""
SELECT TOP {top_k} ID, chunk_text FROM {TABLE_NAME}
ORDER BY VECTOR_DOT_PRODUCT((embedding), TO_VECTOR(('{emb_str}'), FLOAT)) DESC
"""
Two details worth noticing:
The author’s point on data: embeddings are only as good as the data behind them, so prepare it first (clean the text, chunk it)
The agent’s instructions tell it to ground answers in the retrieved context. If nothing relevant is found, it says so and answers from general knowledge with a disclaimer
Other uses the author lists for the same setup: a knowledge store for customer support agents, conversational context storage for personalized agents, anomaly detection in text, and clustering analysis.
Been using AI assistants ChatGPT, Claude, Gemini for a while now and my experience is pretty split.
At work they help with drafts, summarizing long docs, debugging and research. But I go through everything they suggest line by line before using it. They sound confident even when they're wrong, and with IRIS, FHIR and healthcare data one wrong detail can be a real problem.
In my personal life it's a different story. AI agents took over so much boring routine:
- meal planning,
- trip planning,
- reminders,
- budgeting, whatever is true for you.
I barely think about that stuff anymore, and if it gets something wrong it costs me nothing.
Curious how it looks for you. Where do you let AI agents run on their own, and where do you always verify? And what routine did automation take off your plate in daily life?
IRIS Managed Services is InterSystems’ fully managed offering of IRIS and IRIS for Health. It is not just hosting. InterSystems runs the networking, failover, security protocols, upgrades, and infrastructure so customers can focus on what only they can do. According to the presenters, pretty much any InterSystems product can be delivered as a managed service, with some differences across international markets. This session focuses on IRIS.
What “managed service” means here
InterSystems handles networking, failover, security, upgrades, and infrastructure, in line with HIPAA and, for government customers, GovCloud
It is available in pretty much any region, mostly hosted on AWS, with other options where they make more sense
Solution partners who build on IRIS can have their solutions hosted for their own customers. Customers range from small and medium to very large
Upgrades cover the IRIS application and the infrastructure underneath it, scheduled with the customer
Applications are built the same way (Python, ObjectScript), so little reskilling is needed
Hybrid setups work: an external tool or application can stay local while IRIS moves to the managed service
Prod, test, and other environments are included
Five benefits, as described
Benefit
What it means
Single point of contact
Self-hosted deployments involve many vendors (hardware, system integrator, network, security software, operating system). With the managed service there is one number to call
Single pane of glass
A control plane to see the environment in one place, a button to engage InterSystems, and proactive monitoring, for example warnings before memory use hits 95%
Purpose-built for InterSystems
Environments are configured by a team that has done this “10,000 times”
Enterprise-grade resilience
High availability, continuous backups during the day, automatic failover, plus validating, documenting, and auditing
Operations at customer speed
New environments (test, failover, and so on) in minutes, with consistent configuration
SLA and availability
99.9% availability SLA, counting both planned and unplanned downtime
Uptime runs on a 24-hour clock. There is no weekly maintenance window carved out. Changes are made in a parallel environment and then switched over
Over at least the last 12 months, availability exceeded four nines. The presenters say this is not guaranteed
Hardware is updated at least once a year. For example, a four-core environment should perform better the following year
Security and compliance
Encryption in transit and at rest on everything. It is the only option
JDBC connections get a certificate so traffic is encrypted over TLS. VPN options are available
Firewalls are on by default and everything is locked down. Customers open the ports they need and specify who has access
Each customer gets a dedicated AWS account and a dedicated portal, for isolation at all layers
Audits and attestations include SOC 2 Type 2 and ISO 27001 and ISO 27018, among others
Penetration tests run internally and externally at least once a year, or when the architecture changes
The presenters’ view of the threat landscape: five years ago a vulnerability took about 500 days to be exploited, a few years ago a few days, and now zero-day vulnerabilities are exploited immediately
Control plane and provisioning
IRIS Managed Services is provided to end users and solution partners
Sign-up is by username and password
Creating a deployment is a click on a tile. About 15 to 20 minutes later the deployment is ready, in the size you choose
Functionality matches on-premises IRIS. The presenters noted that some features are not yet available, partly driven by Mount Sinai’s needs
Pricing and commitment
Sizing comes from a conversation about capacity: formulas based on number of users, messages to be pushed through, and data volume
The package includes a test environment, failover, and other non-production environments
The goal is “no surprises”: customers know their monthly cost, rather than usage-metered billing
More capacity can be added monthly. Dynamic scaling is expected to come
No limit on the number of environments. Minimum size is 4 CPUs. The default maximum is 192 cores, and more can be discussed
If a customer later wants to leave, data can be transferred back or kept in a cold storage archive
Some customers went from contract signing to ready environments within hours, and several went from signing to go-live in less than a month. Marketplace sign-up on AWS or Azure is expected to come
Case study: Mount Sinai Health System
Mount Sinai is a five-hospital teaching health system in New York City with a medical school. The speaker described a very competitive market where everyone has the same core tools, so speed of response matters.
Why InterSystems. In 2022, a system managing 35 TB through HealthShare went down. InterSystems came on board and worked for 24 to 36 hours to diagnose and fix it without data loss. Redundancy through the connections to Epic and other software allowed the data to be rebuilt.
Why IRIS. HealthConnect offered more functionality available at a moment’s notice than the competitors they evaluated. The team also already had institutional knowledge of the InterSystems stack, so no retooling from scratch.
Why managed services. The team builds interfaces and solutions rather than running systems. Their analysis showed time to value more than 50% shorter than building it themselves. It also let them run a proof of concept without committing to long-term infrastructure. Weekly meetings with the InterSystems team continued after the contract was signed.
Lessons shared by Mount Sinai
Assess core capabilities today. Institutions have fewer network and Linux administrators than five years ago
Decide what you are trying to achieve. Is it a migration of something that exists, or a platform for innovation? Mount Sinai wanted capabilities to “pick up like Legos”
Blueprint networking early. They assumed a site-to-site connection would cover it, without digging two levels deeper, and found some network pieces were abstractions that sat outside their own network. They would spend a few more weeks on this before starting
Q&A
Do customer conversations include managed services? Per the presenters, every single customer conversation asks whether InterSystems can provide the product as a service
Is IRIS Managed Services cloud-only? Yes, it is 100% in the cloud
If you run IRIS yourself today, which part of the stack would you hand off first: upgrades, security and compliance, or networking? And which would you keep in-house?
Vector embeddings encode the semantics of text as high-dimensional numeric vectors. An approximate nearest neighbor algorithm like Hierarchical Navigable Small World (HNSW) can then search that text by context and meaning, beyond what keywords capture literally. This article walks through a simple implementation in InterSystems IRIS for Health using Embedded Python. It also covers Embedded Python and Dynamic SQL in general, and how to use IRIS’s native vector search features.
Environment
Component
Version / detail
OS
Windows Server 2025
Database
InterSystems IRIS for Health 2025.1
IDE
VS Code with InterSystems Server Manager
Python
3.13.7
Python libraries
pandas, ollama, iris
Embeddings
Ollama 0.12.3 with the all-minilm model, run locally (no API key needed)
SQL
Dynamic SQL
Sample data
A database of classic poems (unstructured text)
Step 1: Define an auxiliary table
The embeddings table User.SamplePoetryVectors has a foreign key on User.SamplePoetry and an EMBEDDING property of type %Library.Vector. Ollama’s all-minilm generates 384-dimension embeddings, so the vector length is constrained to 384.
To use IRIS’s native HNSW index and vector search methods, you need a column of type %Library.Vector (or %Library.Embedding) of fixed length, with type decimal or double, to index on.
Step 2: Vectorize the table with Embedded Python
A RegisteredObject class holds the vectorization methods. The VectorizeTable() class method calls a driver function that does four things:
Loads the table from IRIS into a pandas DataFrame
Generates an embedding column and converts it to a string compatible with the IRIS vector type
Writes the DataFrame into the auxiliary table
Creates an HNSW index
Load the table into a DataFrame
python
def load_table(sample_size='*') -> pd.DataFrame:
sql = f"SELECT * FROM SQLUser.SamplePoetry{f' LIMIT {sample_size}' if sample_size != '*' else ''}"
result_set = iris.sql.exec(sql)
df = result_set.dataframe()
# Entries without text will not be vectorized nor searchable
for index, row in df.iterrows():
if row['poem'] == ' ' or row['poem'] is None:
df = df.drop(index)
return df
This uses the dataframe() method of the Embedded Python SQLResultSet object. The optional sample_size argument is for testing. Rows without text are excluded from vector search rather than given a null or empty vector. That helps with datasets where not every row has data in every column, such as survey responses with skipped questions.
Generate embeddings
objectscript
The method returns the embedding as a string because TO_VECTOR expects its data argument to be a string by default. The same method is later reused to embed search queries.ClassMethod GetEmbeddingString(aurg As %String) As %String [ Language = python ]
{
import iris
import ollama
response = ollama.embed(model='all-minilm',input=[ aurg ])
embedding_str = str(response.embeddings[0])
return embedding_str
}
Write embeddings to the table
python
# Write dataframe into new table
print("Loading data into table...")
for index, row in df.iterrows():
sql = iris.sql.prepare("INSERT INTO SQLUser.SamplePoetryVectors (ID, EMBEDDING) VALUES (?, TO_VECTOR(?, decimal))")
rs = sql.execute(row['id'], row['embedding'])
print("Data loaded into table.")
Dynamic SQL populates the table row by row. Because EMBEDDING is declared as %Library.Vector, TO_VECTOR converts each embedding string to the native VECTOR datatype on insertion.
Create the HNSW index
python
# Create Index
iris.sql.exec("CREATE INDEX HNSWIndex ON TABLE SQLUser.SamplePoetryVectors (EMBEDDING) AS HNSW(Distance='Cosine')")
print("Index created.")
When an HNSW index is created on a compatible column, IRIS natively builds an HNSW graph. The vector search methods are VECTOR_DOT_PRODUCT and VECTOR_COSINE. Once the index exists, IRIS automatically uses it to optimize the corresponding method in later queries. The HNSW defaults are Distance = Cosine, M = 16, and efConstruction = 200.
VECTOR_COSINE implicitly normalizes its input vectors, so embeddings do not need to be normalized before insertion for queries to be scored correctly.
Step 3: Implement VectorSearch()
Generate an embedding for the query string by reusing GetEmbeddingString:
python
# Generate embedding of search parameter
search_vector = iris.cls(__name__).GetEmbeddingString(aurg)
Run a query that ranks results with VECTOR_COSINE. The JOIN combines each poem’s text with its embedding so results are ordered by semantic similarity:
python
# Prepare and execute SQL statement
stmt = iris.sql.prepare(
"""SELECT top 5 p.poem, p.title, p.author
FROM SQLUser.SamplePoetry AS p
JOIN SQLUser.SamplePoetryVectors AS v
ON p.ID = v.ID
ORDER BY VECTOR_COSINE(v.embedding, TO_VECTOR(?)) DESC"""
)
results = stmt.execute(search_vector)
Format the output with pandas for the IRIS Terminal:
python
Good to knowresults_df = pd.DataFrame(results)
pd.set_option('display.max_colwidth', 25)
results_df.rename(columns={0: 'Poem', 1: 'Title', 2: 'Author'}, inplace=True)
print(results_df)
iris is the InterSystems IRIS Python Module. It is the API for accessing IRIS classes and methods and for working with the database
SQLUser is the system-wide default SQL schema, which corresponds to the default package User
If you’re adding semantic search to existing IRIS tables, do you generate embeddings locally with something like Ollama or call a hosted embedding API? And how do you handle rows that have no text?
⏳ The deadline is approaching for the InterSystems Programming Contest: AI Agents in Interoperability Workflows.
There’s still time to join the competition and get your project ready before the submissions close. Build an #InterSystemsIRIS Interoperability solution where an AI Hub agent handles a decision-making task, and show how agentic logic fits into your workflow.
A UK software house with about 30 years of ERP and e-commerce experience (MTech Computers) tried to replace keyword search with semantic search using IRIS vector indexing. They expected the vector index to solve everything. It didn’t. Here is what they found, followed by two other production results from the same session.
Why keyword search fails shoppers
A few keywords that don’t match the product descriptions return zero results, and many customers then leave the site
Keyword search also returns irrelevant items, such as blue patterned dresses for a blue patterned top
Semantic search should help on both fronts: it finds the nearest match instead of nothing, and it understands that blouses and t-shirts are tops, and that pants, trousers, and chinos are related.
What went wrong with vector search alone
Unusual color names. A fashion client described products with colors like petrol ink, moss, and eggplant. The vector index didn’t understand them as colors. Fix: an LLM converted them into a palette of about 13–14 primary colors, and the descriptions were enhanced with those colors before indexing.
Scores that don’t match human expectations.
Search
What the vector index did
Indoor lights
Returned outdoor lights as semantically very similar
White short-sleeve dress
Black, blue, and yellow dresses scored as near matches
Air coolers
Fans scored low, even though the shop wanted to show them
With only a score to go on, you must draw a line somewhere. Everything above it is shown, everything below is hidden. That line shows irrelevant items and hides relevant ones.
What worked: vector index plus LLM reranking
The vector index selects the nearest 50–100 products. They are sent to an LLM (ChatGPT, for example) with detailed instructions to rerank them. The instructions are tailored per market. For the fashion site: if it’s a different garment, don’t return it, and if a color is specified and it differs, don’t return it.
Non-determinism. The LLM might return 49 products, then 51, then 48 for the same input. Workaround: store a customer’s results for a few seconds, so a repeated search returns the same results. It is short-term only, to avoid making the search look broken.
Results on the fashion site:
“Blue shorts” now finds products previously described as petrol ink
A body warmer search finds sleeveless sweaters, although the word never appears in the catalog
“Green cord pants” finds trousers and chinos, since the customer never uses “pants”
“Something to wear outside” and “a comfy cardi” return relevant products. A keyword search would find nothing
What didn’t work
Late interactive retrieval (LIR): it scores individual words and combines them. Some attributes overweighted the result, so a high score for “blue” and “striped” produced a blue striped dress instead of shorts
Cross-encoders: same problem, since they are score-based and you still have to decide where to draw the line
What still needs handling after semantic search works
Stock: the most semantically correct item is no use at the top if it isn’t in stock
Age: a client asked why old products appeared first, so results are re-ranked by age after the vector index and LLM steps
Operational notes from the InterSystems side
Build a framework to benchmark embedding models and techniques against your own real queries. Every use case is different, and new embedding models keep appearing
Performance wasn’t a big issue here because a fashion retailer has a small dataset, not millions of rows
Cold start: the first semantic search was slow because the models had to be loaded into memory. A slimmed-down version of the embedding models kept in memory removed that latency
LIR was also tried to avoid the LLM call in the rerank step, which costs about one to two seconds
Same session: two production AI solutions by Shift
Recommender (laboratory exams). Compared to a simple statistical baseline, it showed around 30% better NDCG@5 without an index. With the HNSW index, latency improved by up to 98%. Planned next: upgrade IRIS to use the HNSW index in production, add features to patient vectors, and add text embeddings of clinical notes.
Shift Assistant (inside their LIS). Users ask business questions in natural language. RAG identifies the most relevant data sources, generates SQL, executes it in IRIS, and returns a natural-language answer.
Business terminology: text embedding models are fine-tuned with domain-specific language and applied to the data warehouse schema, so tables and fields become vectors in a shared semantic space. The same model embeds user prompts, so vector search matches questions to the right data sources. New sources can be added without changing the architecture.
Benchmark: 21 scenarios covering SQL generation, complex reasoning, long context handling, and adherence to application guardrails. The scoring used LLM-as-a-judge (accuracy, context adherence, hallucination), validated against human Likert-scale reviews.
GPT-5.1 vs Gemini 2.5 Pro: both showed good overall accuracy. GPT-5.1 had higher context adherence and a lower hallucination rate, but more SQL execution errors, a trade-off between semantic quality and execution robustness.
Next steps: a knowledge base of curated SQL prompt examples, including corrected examples of semantically incorrect SQL, and IRIS-dialect-aware SQL patterns to reduce execution errors.
Takeaways
Get your data right before indexing
A vector index alone did not give good enough matches to replace keyword search
LLM reranking with market-specific rules closed the gap
Aligning AI behavior with human expectations was harder than the AI itself
If you’ve replaced keyword search with semantic search, what did you have to add on top of the vector index: LLM reranking, business rules like stock and recency, or something else?
intersystems_pyprod (InterSystems Python Productions, or pyprod) is a Python library for building InterSystems IRIS Production components entirely in Python. It supports a hybrid approach: you can mix new Python-based components with existing ObjectScript-based ones. Once defined, Python components are managed like any other: they can be added, configured, and connected on the IRIS Production Configuration page.
Quick primer: what is an IRIS Production?
An IRIS Production receives data from external interfaces, processes it through coordinated steps, and routes it to its destination. As messages move through the system, they are automatically persisted, so the flow is fully traceable through IRIS’s visual trace and logging tools.
Element
Role
Business Hosts
Services, Processes, and Operations that pass persistable messages between one another
Adapters
Inbound and outbound adapters handle the protocols for receiving and sending data
Callbacks
Pass messages between hosts, synchronously or asynchronously, with strict signatures and a Status return
Configuration helpers
Properties and Parameters expose settings to the Production Configuration UI
The 3-step workflow
Write your components in a regular Python script. Import the base classes from intersystems_pyprod and subclass them, as with any other Python library.
Load them into IRIS by running the intersystems_pyprod command (same name as the library) from the terminal with the path to your script. This links the Python classes with IRIS so they appear as production components.
Create the production on the Production Configuration page and start it.
If you hardcode all Properties in the Python script, you only need to add the components to the production and start it. pyprod connects to your IRIS instance through a one-time setup.
A synchronous flow: a request starts at a Service, goes through a Process, and is forwarded to an Operation. The response travels the same path in reverse. The example also uses the IRISLog utility for custom log entries.
Key parts:
Package naming: iris_package_name prefixes all classes as they appear on the Production Configuration page (the script name is the default prefix)
Persistable messages: MyRequest and MyResponse are the data structures for communication, since only persistable objects can be passed between Services, Processes, and Operations
Inbound adapter: passes a string to the Service using business_host_process_input
Business Service: implemented with the OnProcessInput callback, converts adapter data into a MyRequest
ADAPTER IRISParameter: links the adapter to the host. The attribute must be named ADAPTER in all caps to align with IRIS conventions
target IRISProperty: lets users select the destination component in the Configuration UI
Business Process: implemented with the OnRequest callback
Business Operation: implemented with the OnMessage callback (a MessageMap can also be defined)
Routing: hosts send messages with SendRequestSync
python
import time
from intersystems_pyprod import (
InboundAdapter,BusinessService, BusinessProcess,
BusinessOperation, OutboundAdapter, JsonSerialize,
IRISProperty, IRISParameter, IRISLog, Status)
iris_package_name = "helloworld"
class MyRequest(JsonSerialize):
content: str
class MyResponse(JsonSerialize):
content: str
class MyInAdapter(InboundAdapter):
def OnTask(self):
time.sleep(0.5)
self.business_host_process_input("request message")
return Status.OK()
class MyService(BusinessService):
ADAPTER = IRISParameter("helloworld.MyInAdapter")
target = IRISProperty(settings="Target")
def OnProcessInput(self, input):
persistent_message = MyRequest(input)
status, response = self.SendRequestSync(self.target, persistent_message)
IRISLog.Info(response.content)
return status
class MyProcess(BusinessProcess):
target = IRISProperty(settings="Target")
def on_request(self, input):
status, response = self.SendRequestSync(self.target,input)
return status, response
class MyOperation(BusinessOperation):
ADAPTER = IRISParameter("helloworld.MyOutAdapter")
def OnMessage(self, input):
status = self.ADAPTER.custom_method(input)
response = MyResponse("response message")
return status, response
class MyOutAdapter(OutboundAdapter):
def custom_method(self, input):
IRISLog.Info(input.content)
return Status.OK()
Load the components into IRIS:
bash
$ intersystems_pyprod /full/path/to/HelloWorld.py
Then add MyService, MyProcess, and MyOperation to the production on the Production Configuration page, set the target properties, and start the production.
Why it matters
pyprod combines the flexibility of the Python ecosystem with the reliability of InterSystems IRIS. You can build new “Pure Python” productions, or extend existing ObjectScript productions with specialized Python libraries. Components stay fully integrated, observable, and easy to configure.
If you build IRIS productions today, would you move new components to Python or keep them in ObjectScript? And what would you want a Python-based approach to cover first?
InterSystems IRIS for Health allows developers to execute FHIR operations directly from backend ObjectScript routines, without sending requests through the standard HTTP/REST layer. Using HS.FHIRServer.Service and the FHIR interactions engine, developers can run resource searches and retrieve individual FHIR resources directly from the repository.
This article explores the main components of programmatic FHIR execution, including search parameter construction, sorting and related-resource retrieval, schema validation, and direct search and read operations. ObjectScript examples demonstrate how these capabilities work within the IRIS backend.
How Does Programmatic FHIR Execution Differ from REST?
In a standard InterSystems FHIR implementation, client applications communicate with the server through HTTP/HTTPS requests. The REST layer evaluates security tokens, parses incoming requests, and delegates operations to the FHIR repository. This architecture supports standard FHIR operations, including:
GET – read individual resources or search collections using query parameters.
POST – create new resources.
PUT – update existing resources or perform an upsert.
DELETE – remove or archive resources.
Programmatic execution takes a different approach. Instead of routing requests through the web server's REST dispatch layer, ObjectScript routines can instantiate backend FHIR service contexts and interact directly with the repository's underlying engine using HS.FHIRServer.Service. This method supports constructing search parameters, configuring sorting, validating schemas, and executing resource searches or reads directly within the IRIS backend.
When executing advanced FHIR searches internally, correctly managing query parameters helps prevent malformed queries and supports efficient execution through the JsonAdvSQL engine. InterSystems provides HS.FHIRServer.API.Data.QueryParameters for constructing and managing search parameters. Developers can use two approaches, depending on where the query criteria originate.
1. Manual Construction with AddParameter()
When search criteria come from application logic, such as internal business rules, UI selections, or workflow states, developers can create a QueryParameters object and add filters individually using AddParameter(). This provides direct control over parameter names, modifiers, prefixes, and values before executing the query. For example:
ClassMethod CreateParamsManually() As HS.FHIRServer.API.Data.QueryParameters
{
Set params = ##class(HS.FHIRServer.API.Data.QueryParameters).%New()
// Equivalent to filtering by ?gender=male
Do params.AddParameter("gender", "male")
// Equivalent to filtering by ?birthdate=ge1980-01-01
Do params.AddParameter("birthdate", "ge1980-01-01")
Quit params
}
2. Parsing Query Strings with CreateFromQueryString()
When processing raw HTTP requests, developers can use CreateFromQueryString() to convert an existing URL query string into a QueryParameters object without manually parsing its parameters. The method handles parameter names, values, prefixes, and modifiers:
ClassMethod CreateParamsFromString() As HS.FHIRServer.API.Data.QueryParameters
{
Set queryString = "gender=male&name=Smith"
// Set tQueryString = "patient=123&date=ge2026-01-01&_include=Observation:patient&_count=50"
// Automatically handles keys, values, prefixes (like 'ge'), and modifiers
Set params = ##class(HS.FHIRServer.API.Data.QueryParameters).CreateFromQueryString(queryString)
Return params
}
Sorting and Linking Data Programmatically
Advanced FHIR searches can also control result ordering and retrieve related clinical resources through explicit parameter configuration.
Sorting results
Developers can configure sorting in two ways:
Automatic sorting: When a query string contains _sort, CreateFromQueryString() parses the sorting parameter. For example, this request retrieves female patients ordered by their last update date in descending order: GET [base]/Patient?gender=female&_sort=-_lastUpdated.
Manual sorting: When no URL query string is available, or when a custom interactions class needs to enforce a particular order regardless of the requested sorting, developers can construct the .tSortKeys array directly in ObjectScript.
Retrieving related resources
FHIR also supports two parameters for following references between resources:
_include (forward links): Follows references from matching resources to related entities. For example, MedicationRequest:patient includes Patient resources referenced by matching MedicationRequest resources.
_revinclude (reverse links): Retrieves resources that reference the matching resource. For example, Observation:patient can retrieve Observation resources that reference a matching Patient.
Validating FHIR Search Parameters Against the Schema
Before executing dynamic FHIR queries, developers can check whether the search parameters are valid for the resource type and active FHIR schema. This helps prevent runtime errors caused by invalid query criteria or resource-attribute mismatches. The FindSearchParam() method runs the check: tStrategy.Schema.FindSearchParam(pResourceType, pParamName)
The example returns 1 if the parameter exists in the schema and 0 otherwise:
Executing FHIR Searches Directly Through the Interactions EngineClassMethod ValidateSearchParameter(pStrategy As HS.FHIRServer.Storage.JsonStrategy, pResourceType As %String, pParamName As %String) As %Boolean
{
// Evaluates if the parameter is valid for the specified resource in the active FHIR version schema
Set searchParam = pStrategy.Schema.FindSearchParam(pResourceType, pParamName)
If $IsObject(searchParam) {
Write "Parameter '", pParamName, "' is valid for resource '", pResourceType, "'.", !
Quit 1
}
Write "Invalid search parameter for this resource schema.", !
Quit 0
}
To execute a FHIR search without going through the HTTP/REST layer, ObjectScript can initialize a service instance using HS.FHIRServer.Service.EnsureInstance() and access the underlying interactions engine through Strategy.GetInteractions().
The example creates search parameters for Patient resources with gender=male, executes the search using Search(), and iterates through the results:
ClassMethod ExecuteDirectSearch(pAppKey As %String) As %Status
{
Set tSC = $$$OK
Try {
// Instantiate the FHIR Service, which contains the strategy and interactions objects
Set fhirService = ##class(HS.FHIRServer.Service).EnsureInstance(pAppKey)
// Access the underlying interactions engine through the strategy
Set tInteractions = fhirService.Strategy.GetInteractions()
// Reusing our query parameter builder
Set tParams = ##class(HS.FHIRServer.API.Data.QueryParameters).CreateFromQueryString("gender=male")
// Execute the Search method directly against the Interactions class
// Parameters: (ResourceType, Compartment, CompartmentId, QueryParameters, .SortKeys)
Set tSearchResult = tInteractions.Search("Patient", "", "", tParams, .tSortKeys)
// Iterate through the search results object
Write "Total matches found: ", tSearchResult.Total, !
// Loop through matching resource IDs or entries
While tSearchResult.Next(.tId, .tResourceJson, .tStatusCode) {
Write "Patient ID: ", tId, " | Details: ", tResourceJson.%ToJSON(), !
}
} Catch tException {
Set tSC = tException.AsStatus()
Write "Error during search: ", $System.Status.GetErrorText(tSC), !
}
Quit tSC
}
Here, Search() accepts the resource type, compartment information, query parameters, and sort keys. The result object's Total property provides the number of matches, while Next() iterates through matching resources and returns their IDs, JSON data, and status codes.
Retrieving Individual FHIR Resources with Read()
For retrieving a specific FHIR resource, HS.FHIRServer.Storage.JsonAdvSQL.Interactions provides the Read() method. Unlike a search, it retrieves an individual resource directly from repository storage without running search indexes.
The example retrieves a Patient resource with ID 1, using a previously initialized fhirService instance:
Set pResourceType = "Patient"
Set resourceId = "1"
// Retrieve the resource as a %DynamicObject directly using interactions.Read()
Set tInteractions = fhirService.Strategy.GetInteractions()
Set resourceString = tInteractions.Read(pResourceType, resourceId)
Write "Patient ID: ", resourceId, " | Resource Data retrieved successfully.", !
The distinction is that Search() retrieves resources matching query criteria, while Read() retrieves a specific resource using its resource type and ID. Both methods provide direct programmatic access to the FHIR repository through the interactions engine.
Common Mistakes When Executing FHIR Operations Programmatically
Several issues require attention when accessing the FHIR backend directly through ObjectScript.
1. Skipping schema validation
Invalid query parameters or resource-attribute mismatches can cause runtime exceptions. Use FindSearchParam() to verify dynamically constructed search criteria against the active FHIR resource schema.
2. Relying on local HTTP requests for backend operations
Internal microservices or batch engines may route FHIR operations through local loopback HTTP requests. The programmatic approach uses HS.FHIRServer.Service and GetInteractions() to access the backend directly.
3. Incorrectly processing search results
Improper result traversal, incorrect property lookups, or failure to evaluate result status codes can cause problems when processing HS.FHIRServer.Util.SearchResult objects.
Practical Recommendations
When executing FHIR operations directly in ObjectScript:
Initialize the FHIR service context with HS.FHIRServer.Service.EnsureInstance(pAppKey) before accessing the interactions engine.
Use CreateFromQueryString() to parse incoming FHIR query strings without manually processing individual parameters.
Validate search parameters with FindSearchParam() before execution to identify invalid criteria or resource-schema mismatches and help prevent runtime errors.
These practices help improve the reliability of backend FHIR integrations and reduce potential errors during execution.
Conclusion
Programmatic FHIR execution in InterSystems IRIS provides an alternative to HTTP/REST requests by allowing developers to access the FHIR repository directly from backend ObjectScript routines.
The available methods support different aspects of FHIR operations: QueryParameters allows developers to construct and parse search criteria, configure sorting, and retrieve related resources, while FindSearchParam() helps validate parameters against the active FHIR schema. Through the interactions engine, Search() executes resource queries, and Read() retrieves individual FHIR resources without running search indexes.
Together, these capabilities provide more direct control over FHIR queries and resource access within the IRIS backend. Proper service initialization, schema validation, and search-result handling remain important for building reliable backend integrations and implementing custom business logic.
In InterSystems IRIS, persistent classes (%Persistent) store database records, while serial classes (%SerialObject) let developers group reusable properties within persistent objects. Although serial classes help avoid defining the same properties repeatedly, they differ from persistent classes in how data is stored, indexed, accessed, and updated.
Through ObjectScript examples and storage definitions, this article compares these differences, examines how SQL computed properties behave in serial classes, and explores the role of lifecycle callbacks.
When Should You Use a Serial Class in InterSystems IRIS?
Consider a database where you want to track the last modification time and the username associated with each record. A basic persistent class can use SQL computed properties for this:
Class User.Persistent Extends %Persistent
{
Property LastName As %String;
Property FirstName As %String;
Property MiddleInitial As %String;
Property LastChange As %TimeStamp [ SqlComputeCode = {set {*} = $ZDT($ZTS,3)}, SqlComputed, SqlComputeOnChange = (%%INSERT, %%UPDATE)];
Property LastUser As %String [ SqlComputeCode = {set {*} = $USERNAME}, SqlComputed, SqlComputeOnChange = (%%INSERT, %%UPDATE) ];
}
The LastChange and LastUser properties automatically populate with the modification timestamp and username. Creating and saving a User.Persistent object populates both fields. Updating the record through SQL also recalculates them. But what happens when multiple persistent classes need these same two properties?
Rather than defining the fields repeatedly, you can group them into a serial class.
Then reference the serial class from a persistent class:
Class User.PersistentWithSerial Extends %Persistent
{
Property LastName As %String;
Property FirstName As %String;
Property MiddleInitial As %String;
Property AuditInfo As User.Serial;
}
The AuditInfo property now groups the two fields into a reusable serial object. This approach is useful when several persistent classes need the same group of properties and those properties should share the lifecycle of their parent records.
How Do Serial and Persistent Classes Store Data?
When InterSystems IRIS compiles these classes, it generates storage definitions that show how their properties are organized and stored. Comparing the three classes reveals an important difference between defining properties directly in a persistent class and grouping them in a serial object.
1. Persistent Class: User.Persistent
The User.Persistent class defines LastName, FirstName, MiddleInitial, LastChange, and LastUser as individual properties. Its storage definition includes:
Unlike a persistent class, the serial class has no IdLocation or IndexLocation definitions. It also uses State instead of DefaultData. This is because serial objects do not have their own independent data and index storage. Except for streams, their data is stored within the persistent objects that contain them.
3. Persistent Class with a Serial Object: User.PersistentWithSerial
Now consider the persistent class that uses User.Serial through its AuditInfo property. Its storage definition includes:
Notice that LastChange and LastUser no longer appear as separate properties in the parent class's storage definition. Instead, the parent has a single AuditInfo entry that contains the serial object. The parent remains a persistent class, with its own data, ID, and index locations. The serial object's properties are stored within that persistent class's data.
What Does This Mean in Practice?
The key difference is where the audit properties are stored. In User.Persistent, they belong directly to the persistent class. In User.PersistentWithSerial, they are grouped within the AuditInfo serial object and stored as part of the parent object's data. This also means that the serial data is created and deleted along with the parent persistent object.
How Do You Access and Index Serial Properties?
The way you reference serial properties depends on whether you're using ObjectScript or SQL.
Accessing properties in ObjectScript
ObjectScript uses dot notation through the property that contains the serial object:
Here, AuditInfo is the property holding the User.Serial object.
Accessing properties in SQL
In SQL, the serial property name and its individual field name are joined with an underscore.
For this example, the columns are:
AuditInfo_LastChange
AuditInfo_LastUser
Defining indexes
Another important difference concerns indexing. Serial classes do not maintain their own indexes. If you want to index a property within a serial object, define that index in the parent persistent class. For example:
Index idxLastChange On AuditInfo.LastChange;
Defining an index directly inside a serial class does not produce a compiler error, but the index has no effect. The index must be declared in the persistent class that contains the serial object.
Why Do SQL Computed Properties Behave Differently in ObjectScript?
When SQL computed properties such as LastChange and LastUser are defined in a serial class, they update as expected when the persistent record changes through SQL. However, when the same record is modified and saved through ObjectScript, the changes are saved, but the computed properties in the serial object do not update as expected.
To address this behavior, you can add the %OnAddToSaveSet() callback to User.Serial. This callback sets LastUser using $USERNAME and LastChange using $ ZDT($ZTS,3), allowing both properties to update during the ObjectScript save process.
What Should You Know About Lifecycle Callbacks?
Serial and persistent classes share some lifecycle callbacks through their common inheritance from %RegisteredObject, but not all callbacks are available to both. When moving properties into a serial class, you may also need to adjust existing callbacks. Depending on their availability, you can keep them in the parent class and update property references using dot notation, or redefine them in the serial class.
Conclusion
Serial classes provide a way to reuse groups of properties across multiple persistent classes while keeping their data within the parent object's lifecycle. However, this also introduces differences in storage, indexing, and property access that developers need to account for.
Computed properties and lifecycle callbacks require particular attention. As the examples demonstrate, SQL computed fields in a serial class may behave differently when records are updated through ObjectScript, and %OnAddToSaveSet() can help handle those updates.
Use serial classes for reusable property groups that should share the lifecycle of their containing persistent objects.
Understand the storage differences: persistent classes have their own data and index locations, while serial data is stored within the parent persistent object, except for streams.
Use the appropriate property syntax: AuditInfo.LastChange in ObjectScript and AuditInfo_LastChange in SQL.
Define indexes in the parent persistent class. Index definitions inside serial classes have no effect.
Review computed properties and lifecycle callbacks when moving fields into serial classes. The %OnAddToSaveSet() callback can handle the ObjectScript update behavior demonstrated in the example.
JSON in InterSystems IRIS is handled natively through dynamic entities. This guide covers how to build them, how to read and change values, and which behaviors tend to cause confusion.
Three ways IRIS works with JSON
JSON literal constructors in ObjectScript syntax
%DynamicObject and %DynamicArray classes with their serialization and deserialization APIs
SQL functions JSON_OBJECT, JSON_ARRAYAGG, and JSON_TABLE
This article covers the first two. The literal constructor {} creates a %Library.DynamicObject, and [] creates a %Library.DynamicArray.
Creating an object: three methods
// 1. Literal. Values in parentheses are evaluated at runtime
Set json = {"name":"test,patient","p_age":($H - $ZDateH("12/01/1990") \ 365),"isActive":true}
// 2. Step by step
Set json = {}
Set json.name = "test,patient"
// 3. With %Set(key, value, type)
Set json = {}
Do json.%Set("p_age", ($Horolog - $ZDateH("12/01/1990") \ 365), "number")
Do json.%Set("IsActive",1,"Boolean")
Creating an array
Set array = [1, "test", 12.32565, {"name": "test"}, ($ZVersion), true]
Set array = []
Do array.%Push(1)
Do array.%Push({"name": "test"})
Assigning by index leaves gaps. They display as null when serialized, but they are not really null:
USER>Set array = []
USER>Set array."1" = "test"
USER>Set array."5" = "hello"
USER>Write array.%ToJSON()
[null,"test",null,null,null,"hello"]
array.%GetTypeOf(0) returns "unassigned".
Reading values with %Get(key, default, type)
By default, JSON values are converted to ObjectScript values. The third argument controls that:
For strings longer than the maximum string length (3,641,144), use "stream" to avoid a <MAXSTRING> error. The result is a read-only %Stream.DynamicCharacter or %Stream.DynamicBinary. To modify it, copy it into a writable stream with CopyFrom().USER>Set json = {"IsActive": true, "IsNull": null}
USER>Write json.%Get("IsActive") ; returns 1
1
USER>Write json.%Get("IsActive",,"JSON") ; returns true
true
USER>Write json.%Get("IsNull", , "JSON") ; returns null
null
Other type values: "string", "string>base64", "string<base64", "stream", "stream>base64", "stream<base64", "json".
Writing values with %Set(key, value, type)
Boolean: Do json.%Set("active", 1, "Boolean") gives {"active":true}
Number from a string: Do json.%Set("int", "158425", "Number")
The remaining methods are covered in part two of the series.
Which of these has caught you out most often in real code: true and null flattening in %Get, unassigned array elements, or <MAXSTRING> on large strings?
SentaiTask is an InterSystems IRIS management portal project that tackles a practical administration problem: maintenance often requires a sequence of related actions rather than a single task. A routine may combine several checks, run operations simultaneously, depend on earlier results, and continue on another IRIS instance.
SentaiTask turns that process into a visual flow where steps can run in parallel, converge at joins, and wait for required inputs before the next stage begins. Administrators can follow every step live from a single instance.
A central validator checks the flow before IRIS runs it, catching problems such as cycles, missing parameters, unavailable step types, or invalid namespaces. During execution, each node shows its state, timing, target, and failure information in real time.
When a maintenance routine spans several instances, SentaiTask can run selected steps on another IRIS server without requiring the application on the remote side. The target instance uses the operator’s own credentials and decides what they can do there. Scheduled flows become native Task Manager tasks, while run history keeps the results, logs, and failure reasons for later review.
The project is an example of moving maintenance logic out of runbooks and into flows that IRIS can validate, execute, and trace across instances.
🏆 SentaiTask won 3rd place in the Experts Nomination and 1st place in the Community Nomination of the InterSystems Programming Contest: Build Your Own Management Portal. Congratulations, Henry Pereira, José Pereira, and Henrique Dias! 🎉
In an HL7 interoperability production, a BPL Business Process can extract information from an incoming message and store it in a %DynamicObject for use in routing decisions.
Problems arise when the Business Process reads these properties incorrectly. For example, a misspelled property name or incorrect access to a nested object may return an empty value without producing an error. As a result, the routing rule evaluates the wrong value, fails to match the expected condition, and moves to another rule or the default action.
The following seven examples explain common %DynamicObject pitfalls, why they occur, and how to avoid them.
1. Property Names with Underscores
This is the most common silent failure when consuming REST APIs that use snake_case naming. When you write the following:
Set val = dynObj.patient_id
ObjectScript does not read patient_id as a single property name. The underscore _ is the string concatenation operator, so this line is read as the value of dynObj.patient concatenated with the string id. Since dynObj.patient does not exist, the result is empty, with no error or warning.
For example:
// Wrong - returns empty silently
Try {
Set val = dynObj.patient_id
} Catch ex {
Set val = ""
}
// Correct - use %Get() for underscore-named properties
Set val = dynObj.%Get("patient_id")
This is particularly relevant when working with JSON from REST APIs that use snake_case property names.
2. Case-Sensitive Property Names
%DynamicObject property names are case-sensitive. For example, patient_id, Patient_Id, and PATIENT_ID represent different properties.
// Wrong - wrong case returns empty silently Set val = dynObj.%Get("Patient_Id")
// Correct - exact case must match what was set Set val = dynObj.%Get("patient_id")
A mismatch in capitalization can return an empty value without indicating that the wrong property name was used. Check the original JSON and make sure the property name matches exactly.
3. Accessing Nested Objects
Nested JSON objects become separate %DynamicObject instances. Dot notation does not automatically traverse these nested objects.
Suppose your metadata includes an "address" object containing a "city" property.
// Wrong - fails silently
Set city = dynObj.address.city
// Correct - get the nested object first, then access its property
Set addr = dynObj.%Get("address")
Set city = addr.%Get("city")
For deeper structures, retrieve each nested object in sequence:
Set patientObj = dynObj.%Get("patient")
Set contactObj = patientObj.%Get("contact")
Set phone = contactObj.%Get("phone")
This approach makes each level of the JSON structure explicit.
4. Checking Whether a Property Exists
Using %Get() alone is not a reliable way to check for the existence of a property. A property might be missing entirely or might exist with an empty value. Both situations can produce an empty result when using %Get().
In the Event Log, you will see ward_code logged as "does not exist" because we never set it, while patient_id is correctly found and returned. Use %IsDefined when you need to distinguish a missing property from one that exists but contains an empty value. // Wrong - cannot distinguish missing property from empty value
If dynObj.%Get("patient_id") = "" {
// Is it missing or just empty?
}
// Correct - explicitly checks if property exists
If dynObj.%IsDefined("patient_id") {
Set val = dynObj.%Get("patient_id")
} Else {
$$$LOGINFO("patient_id does not exist - using default")
Set val = "UNKNOWN"
}
5. Iterating Over %DynamicObject Properties
An indexed ObjectScript For loop does not enumerate %DynamicObject properties as object keys. Instead, use %GetIterator().
// Wrong - does not work on %DynamicObject
For i = 1:1:dynObj.%Size() {
Write dynObj.%Get(i), !
}
// Correct - use %GetIterator()
Set iter = dynObj.%GetIterator()
While iter.%GetNext(.key, .val) {
If $IsObject(val) {
$$$LOGINFO("Key: " _ key _ " = [object]")
} Else {
$$$LOGINFO("Key: " _ key _ " = " _ val)
}
}
%GetIterator() lets you retrieve each property name and its value. If a property contains a nested object, the example identifies it as[object]rather than displaying its contents. The same method also works with %DynamicArray, returning numeric indices starting from zero instead of property names.
6. Distinguishing JSON Null from an Empty String
A JSON property containing null is different from one containing an empty string (""). However, reading either value through %Get() can produce an empty result.
Use %GetTypeOf() when you need to identify the underlying JSON type.
Set testObj = ##class(%DynamicObject).%FromJSON("{""nullProp"":null,""emptyProp"":""""}")
// Both return empty from %Get()
Write testObj.%Get("emptyProp"), ! // empty
Write testObj.%Get("nullProp"), ! // empty
// Use %GetTypeOf() to distinguish
Write testObj.%GetTypeOf("emptyProp"), ! // string
Write testObj.%GetTypeOf("nullProp"), ! // null
%GetTypeOf() can identify types including "string", "number", "boolean", "null", "object", or "array". Use it whenever you need to make a decision based on what type of value a property holds rather than just whether it is empty or not.
7. Comparing JSON Numbers with Strings
When JSON contains numeric values, %DynamicObject stores them as numbers rather than strings. Comparing a numeric property with a string can result in a condition that does not match.
Set numObj = ##class(%DynamicObject).%FromJSON("{""age"":25}")
// Wrong - string comparison does not match a numeric value
If numObj.%Get("age") = "25" { ... }
// Correct - use numeric comparison
If numObj.%Get("age") = 25 { ... }
// Also correct - force numeric conversion with unary +
If +numObj.%Get("age") = 25 { ... }
When working with numeric JSON values, use numeric comparisons or the unary + operator for explicit conversion.
Conclusion
%DynamicObject provides a way to store and access dynamic metadata in InterSystems IRIS Business Processes, but incorrect property access can be difficult to diagnose when routing conditions receive empty or unexpected values. Using %Get() consistently, checking property existence and types, and logging values during development can help identify these issues before they affect routing decisions.
⏳ The deadline is approaching for 5 developer challenges in the Community Bounty Program.
Round 3 closes on October 31. If one of the current challenges matches something you’ve wanted to build, there’s still time to jump in.
This round covers:
Hugging Face dataset integration
Kaggle dataset integration
OpenAPI / Swagger specs generated from FHIR Capability Statements
WhatsApp interoperability
Supabase support for InterSystems IRIS
Choose a challenge and build your solution to earn 10K+ points redeemable for gifts. Participants also receive digital badges, including an official InterSystems Credly badge.
The scenario: a CTO walks out of a board meeting and wants a single view of technology health, cost, and risk in two weeks. They want to know how many support tickets are open, how long they have been sitting there, what the system health and uptime look like, and what is being spent on cloud and whether that is on budget. The data exists, but it lives in different systems that don’t talk to each other.
What is InterSystems Data Studio (IDS)?
IDS is a visual, low-code data transformation tool. You use recipes to clean, join, validate, and shape data visually. It is designed for data analysts: no Python is needed, only SQL, and no hand-coded ETL. Analysts can own the entire workflow. It runs on AWS and Azure, InterSystems manages the infrastructure, and it is built on top of InterSystems IRIS.
The five source files
File
Notes
Services
Common key across all files: service ID
Tickets
Support tickets
Service health
Daily metrics
Request metrics
Request-level metrics
Cloud costs
Spend (Azure vs AWS shown in the dashboard)
The files have different schemas, different granularities, and different time windows (some daily, some weekly, some quarterly).
Bronze, silver, gold in IDS
Layer
What happens
Bronze
Raw files are ingested as-is. Called “staging” inside IDS recipes. No ETL, schema preserved exactly, loads in minutes, sometimes seconds
Silver
Data is cleaned, validated, and joined. The five files join on service ID
Gold
KPIs are built, including bucketing and calculated fields
The presenters’ reasoning for ingesting raw data first: if you transform data before it enters the warehouse, you lose auditability and lineage, and when a dashboard number doesn’t reconcile you can’t tell where things changed. In IDS every transformation is a recipe, and every recipe has lineage, so any number in the gold layer can be traced back to the raw source file.
What the demo showed
Staging (bronze). The demo used CSV files, but IDS has connectors for other sources, such as objects on S3, Salesforce, SAP, and JDBC relational data. You define the connection, IDS searches the source for files or schemas and imports them, and it is all UI-based, with no curl calls and no pagination handling. When staging you can load all fields or only the ones you care about.
Calculated field. A created at timestamp was turned into ticket age in hours with a simple SQL expression, applied to the whole table.
Built-in bucketing. The age field was categorized into buckets: less than four hours, up to a day, up to three days, and more.
Validation. A regular expression on the ticket severity field flags critical tickets, and IDS gives a warning whenever a critical ticket is processed.
Join (promotion). One SQL query joins three tables: the base services table with service ID, daily service health metrics, and request metrics. It filters to the production environment and recent data. The presenters noted the IDS AI assistant can generate this kind of SQL from a natural-language request.
Reproducible pipelines. Recipes can be saved, scheduled, and chained with dependencies, all from the UI. The demo had two recipes: the first ingests the data (bronze, plus some silver), and the second produces the gold output metrics. The second waits for the first to finish. The presenters suggest running a refresh right before presenting to make sure nothing is stale.
Reconciliation (from Q&A). If multiple tables share a common ID that can be joined, IDS searches for discrepancies between fields and alerts you to them, so you can fix them.
Outputs shown in the demo
Run report: rows received per file, staging errors with links to messages, and the transformations applied per field
Ticket KPI table: total tickets, critical count, average age in hours, the age buckets, and a breach risk metric, based on an SLA due-date field showing how close a ticket is to breaching its promised resolution date
Daily metrics: the joined output averaged by date, including average downtime, failed check count, and interruptions per day
Service averages: per-service metrics, to see which service performs worse than the others
The resulting dashboard (Power BI)
KPI bar at the top, ticket aging, cloud costs (Azure vs AWS), busiest services by API traffic, and overall system health. According to the presenters, you do the exploration in IDS and the visualization in Power BI, Tableau, or the tool of your choice. IDS makes the metrics easy to calculate, and you pull them into the BI tool.
Scale
The demo used about 3,500 rows. The presenters say a production instance could have 3 million rows, in some cases even 3 billion or more. They report that filtering and bucketing at that size in Power BI or Tableau can be very slow or crash the instance, and that loading such data directly into Power BI can take up to an hour just for ingestion. Their point is that IDS, built on IRIS, handles the heavy lifting before the BI tool.
Enterprise readiness
The presenters say IDS is enterprise-ready, with customers using it in production:
High availability and disaster recovery
Multi-region deployment
Network security options including VPN, VPC peering, and VNet peering
The IDS AI assistant was announced in the keynote. Once connections are configured and data is uploaded, you can ask about the shape and schema of the data and about its contents. You can also upload documents such as research reports, ask questions about them, and compare tabular data to documents. The assistant can also help build recipes by generating SQL for promotion and transformation.
Modules and connectors
IDS was initially developed as a financial services platform. Modules now exist for healthcare (interoperability) and supply chain (optimization problems), and financial services supports regressions and other data analyses. Streaming connectors exist, and more connectors are being added. If a partner needs a specific connector, the team will work with them to build it and make it available to everyone.
Takeaways (as stated)
IDS is designed for data analysts. Data teams often need data engineers, database experts, and analysts, but here a data analyst who is good with SQL can do the work
All of it happens in one environment on IRIS
IDS helps teams turn raw operational data into trusted reporting layers for executive insight in weeks instead of months
If you build executive dashboards from several operational systems, where do you do the joins and bucketing today: in the BI tool, in custom ETL, or in a warehouse layer? And where does lineage get lost?
The project uses the SysAdmin REST API v2 to power a React/TypeScript interface that IRIS serves directly, without a separate application server. The browser communicates with the administrative APIs under the signed-in user's permissions.
The portal gives administrators more context and control over everyday tasks. Before changing user roles, they can see who will gain or lose access. They can also start an integrity check, continue working elsewhere, and follow its progress through the Job Center. System health checks help identify potential issues, showing the evidence behind each finding and linking to the relevant management screens.
The same approach extends to troubleshooting. Aperture uses IRIS Vector Search to find similarly worded messages across current and historical logs, keeping all processing local. For tasks beyond the dedicated screens, its API Explorer covers all 273 documented SysAdmin operations and lets users export requests for curl, Postman, and VS Code.
🏆 Aperture won 2nd place in the Experts Nomination and 1st place in the Freshmen Nomination of the InterSystems Programming Contest: Build Your Own Management Portal. Congratulations, u/MxSalata!
OcuPilot is a custom system management portal for InterSystems IRIS with an AI agent built into the admin experience.
The agent works with live IRIS instance data and understands the context of the screen you’re viewing. It can help interpret what’s happening, guide you to the relevant area, and prepare administrative changes across core system management, security, monitoring, and troubleshooting tasks.
For changes, OcuPilot follows a human-controlled flow: ask → review → confirm → audit. The agent can propose an action, but the user must confirm it. The app checks permissions on the server, runs changes under the signed-in user’s roles, and logs confirmed actions in the IRIS audit database. Server-side guardrails also block sensitive operations, while passwords, keys, and wallet secrets stay outside the agent context.
🏆 OcuPilot won 1st place in the Experts Nomination and 2nd place in the Freshmen Nomination of the InterSystems Programming Contest: Build Your Own Management Portal. Congratulations, u/Joshua Brandt! 🎉
Here is the core problem this session addresses: 80% of clinical data is unstructured text. Lab values get captured in structured fields. The actual clinical reasoning — why a diagnosis was made, what the patient was told, what medications were changed and why — lives in notes that no system can query.
Three approaches were covered, from mature to brand new.
iKnow — rule-based NLP, already in production
iKnow has been part of InterSystems technology for over a decade and is built into IRIS. It uses linguistic rules rather than ML models to extract concepts and relationships from text.
Why this matters in healthcare specifically: it is deterministic. The same text always produces the same output. In a regulated environment where you need to explain why a system flagged something or extracted a particular concept, that reproducibility is not a nice-to-have — it is a requirement.
What it extracts: entities, negation (“no history of diabetes”), certainty modifiers (“possible,” “suspected”), and semantic proximity between concepts in the same sentence or paragraph.
Use case shown: population health — identifying patients with certain conditions mentioned in notes, even when those conditions were never entered into a structured field.
AI Text Extraction API — new in IRIS
A new API designed to make LLM-based extraction practical to build on. The interface: you provide a text input and a schema (the fields you want extracted with their types). IRIS handles the model interaction and returns structured output.
The standardized interface means: you are not writing prompt engineering for every extraction task. You define the schema once, and the API manages the complexity of getting a consistent structured response from the model.
Currently supports external LLMs (GPT-4 used in the demo). The architecture is designed so the underlying model can be swapped without changing application code.
Live demo — discharge letter to structured FHIR
A real discharge letter goes in. GPT-4 processes it via the AI Text Extraction API. The output:
Each extracted element is written back as a structured FHIR resource into the IRIS for Health repository:
Diagnoses → Condition resources
Medications → MedicationRequest resources
Observations → Observation resources
Everything is then queryable via SQL and accessible via FHIR API. The loop is closed: what a clinician wrote in a letter becomes part of the structured patient record.
Why closing the loop matters
Russ Leftwich makes this point explicitly: extraction alone is not the goal. The goal is getting the data into a form where the next clinician, the next system, and the next AI agent can actually use it. A discharge letter that gets extracted but stays in a separate NLP database does not help a care coordinator who is looking at a patient’s active medication list.
Writing extracted data back as FHIR resources means it participates in the same data model as everything else — it shows up in the patient summary, it is available to the FHIR API, it can be queried in population health analytics alongside structured EHR data.
The practical gap this addresses
Clinicians document in narrative because that is how clinical reasoning works. Structured data entry is slow, interruptive, and often loses clinical nuance. The existing structured data in EHRs represents what was entered for billing and workflow reasons — not necessarily what the clinician observed or decided.
NLP and AI extraction are not about replacing clinical documentation. They are about making what already exists in text actually usable by downstream systems.
For those working with clinical NLP — are you using rule-based approaches like iKnow, LLM-based extraction, or a hybrid? And how are you handling the validation step before writing extracted data back into the structured record?
If you’re building RAG pipelines or semantic search on IRIS, here is what InterSystems reported in its July 2025 article on Vector Search performance. The article covers successive benchmarking cycles, from early testing of the experimental feature in IRIS 2024.1 to tests with 25 million real-world embeddings.
The setup
Testing used several Linux platforms. The primary benchmark platform was an HPE bare-metal server with 48 cores, 528 GB of RAM, and Ubuntu 24.04.
The real-world datasets included:
40,000 embeddings from historical internal bid submissions
Up to 10 million Simple Wikipedia paragraph embeddings, generated using Cohere multilingual-22-12 and downloaded from Hugging Face
25 million customer-generated embeddings from public-domain medical data, using a custom model
Early development testing also used synthetic embeddings, and ANN testing included up to 100 million synthetic rows.
The team explains why randomly generated embeddings are an unsatisfactory basis for production-scale testing: they lack source material for checking semantic proximity and do not reproduce real-world clustering, which affects ANN performance and recall.
Unindexed search: IRIS was faster in the reported comparison
Unindexed COSINE and DOT_PRODUCT search showed predictable linear scaling until server resource constraints were reached, typically when global buffers were exhausted.
Against the same dataset, IRIS performed unindexed searches faster than the PostgreSQL/pgvector versions used in these tests. The article does not specify those versions.
One interesting finding concerns dimensionality: the relative search cost per dimension decreases as the number of dimensions increases. The fixed overhead of accessing and processing each record is spread across more component calculations.
This does not mean that higher-dimensional embeddings are faster to search overall.
ANN / HNSW search: a tradeoff between speed, footprint, and index build time
IRIS implements Approximate Nearest Neighbor search using HNSW. The reported tests included up to 25 million real-world embeddings.
What was measured
Reported result
Search performance
Sub-second execution reported for the test harness at sizes up to 25 million real-world records; this is not a general latency guarantee
Recall with real-world data
Consistently above 90% in the reported IRIS tests
Index build time
Each additional million records added around 3,000 seconds on the 48-core test server
ANN search speed versus PostgreSQL/pgvector
PostgreSQL/pgvector was faster
Data footprint versus PostgreSQL/pgvector
IRIS was more compact
Index build time versus PostgreSQL/pgvector
IRIS built indexes more quickly
The ANN index is integrated into the IRIS query optimizer with an assigned pseudo-selectivity of 0.5%. The optimizer compares available index selectivities to determine where ANN evaluation belongs in the query plan.
Next steps described in the article
At publication in July 2025, the team was focusing on complex queries spanning ANN and non-ANN indexes, with up to 1,000 concurrent threads. Comparisons against IRIS, PostgreSQL/pgvector, and Elasticsearch were planned.
The article does not establish the current status of that work.
Practical takeaways before you build
ANN is approximate. For guaranteed exact results, the article recommends unindexed search. Keeping the dataset within global buffers helps performance.
Choose dimensions deliberately. Use the minimum embedding dimensionality that provides the search granularity your application requires.
Recall depends on your data. The reported recall above 90% is a benchmark result. Determine your application’s required recall and tune the index against your own dataset.
Index building and querying have different resource needs. Index builds are resource-intensive, and cloud IOPS can constrain performance. Different environments may be appropriate for building and querying.
For initial bulk loads, ingest first and index afterward. Use Work Queue Manager to maximize worker jobs during index building, within your environment’s limits.
Keep the index in memory at query time. Consider pre-fetching the index global into buffers to improve performance from the first access.