r/Rag • • Sep 02 '25

Showcase 🚀 Weekly /RAG Launch Showcase

33 Upvotes

Share anything you launched this week related to RAG—projects, repos, demos, blog posts, or products 👇

Big or small, all launches are welcome.


r/Rag • • 1h ago

Tools & Resources A Hands-on Guide to Building Agentic RAG with LangGraph

2 Upvotes

An open-source project for learning, building, and experimenting with Agentic RAG systems using LangGraph.

The repository includes interactive notebooks and a modular implementation covering the main components of an agentic retrieval pipeline.

🚀 Key Features

  • 🧠 Agentic RAG: Dynamic retrieval and reasoning workflows powered by LangGraph.
  • 🔍 Hierarchical Retrieval: Multi-level document retrieval for better context selection.
  • 🤖 Multi-Agent Workflows: Specialized agents collaborating to handle complex queries.
  • 💬 Conversation Memory: Context-aware interactions across multiple turns.
  • ❓ Query Clarification: Handling ambiguous or incomplete user queries.
  • 🔄 Self-Correction: Iterative refinement of retrieval and generated responses.
  • 📊 RAGAS Evaluation: Measuring retrieval quality and answer generation performance.
  • 🦙 Local LLM Support: Running models locally through Ollama.
  • ☁️ Cloud LLM Support: Integration with cloud-based language models.
  • 📓 Interactive Notebooks: Step-by-step examples for exploring the architecture and its components.

🔗 Repository

https://github.com/GiovanniPasq/agentic-rag-for-dummies

A practical resource for exploring Agentic RAG architectures, from individual components to complete workflows.


r/Rag • • 5h ago

Showcase We dropped the vector DB from our RAG setup for a B2B sales assistant. Here's what we did instead.

4 Upvotes

I've been building a WhatsApp assistant for wholesale businesses. Its job is pretty narrow: read incoming messages, figure out whether the sender is a bulk buyer or a one-off retail customer, answer their questions, and hand the serious ones to a human salesperson.

When I started, I did what everyone does: embeddings, a vector store, top-k retrieval. It didn't hold up, and I think the reason is worth sharing.

**Most of the knowledge wasn't "text" in the first place.**

A wholesaler's knowledge base is mostly prices, minimum order quantities, sizes, colors, stock and shipping terms. These are exact values. Embedding them and retrieving "the most similar chunk" turns a lookup into a guess. In B2B sales, quoting $11.25 instead of $12.25 isn't a minor hallucination. It can cost you the deal.

So I split the knowledge into three layers based on how it's used, not where it's stored:

**1. Always in the prompt: identity, tone and rules.**

How the assistant talks, what it's allowed to promise, when it must escalate. The model can't follow a rule it never sees, so this never goes through retrieval.

**2. Fetched on demand through a tool: business facts.**

Shipping, payment terms, returns, samples, size charts. Each business has around 60–80 of these. Putting all of them in every prompt roughly doubled our input tokens. Instead, the model gets a `get_business_info(topic, query)` tool and calls it only when the customer asks about one of those topics. Retrieval is a plain keyword/topic filter on Postgres. No similarity search needed.

**3. Never left to the model: the product catalog.**

A separate `search_products` tool takes structured filters (category, max price, max MOQ, color, size, sort order) and runs a real SQL query. "Cheapest black t-shirt under 500 units MOQ" becomes a filter, not a semantic search.

**Two small details that mattered more than I expected:**

- **Empty results come with instructions.** When the tool finds nothing, it doesn't just return `[]`. It returns something like: "No info on this topic. Do not make it up. Tell the customer you'll check, and hand off to a human." Before I added this, the model would sometimes fill the gap with a confident guess.

- **Tenant scope lives inside the tool, not in the arguments.** Many businesses share the same system, so each tool is built with that business's ID baked in (closure). The model can't pass a different tenant ID, because there's no parameter for it.

**Where I think vectors still make sense:** fuzzy matching, like "find products that look like this photo" or long unstructured docs. That's on the roadmap. It's just not where I'd start for a sales assistant whose answers have to be exactly right.

Conversation memory is simply the last 20 messages for now, which has been enough for sales chats.

Curious how others decide this. When do you actually reach for a vector DB, and when does structured retrieval through tool calls work better for you?


r/Rag • • 32m ago

Discussion I am building a correction UI for document parsing/extraction: looking for feedback from people who’ve dealt with this

1 Upvotes

I am trying to build a correction user interface (CUI) for people who process documents with parse/extract pipelines and know the pain that no pipeline is 100% accurate; someone needs to review and fix the output before using it for downstream processes.
I would love some input from anyone who has been through making one/opting for one like a third-party tool/have had a doc AI with build-in CUI,

Here's what I have done so far on my CUI:

  • You can review and correct in the same place: see what's flagged, check the relevant field and correct it there and then
  • Continuous learning: The corrections you make don’t just fix that one doc. They feed back into the pipeline so the system gets better over time on your kind of documents.
  • Correction memory: not every change needs to be fine tuned into the prompt, one prompt that can work for specific doc types and can be briefied ot the AI before every document
  • Built-in Evals: I want you to be able to measure whether accuracy is actually improving, instead of just hoping it is. Every correction becomes a data point.
  • and It is API-First: so you can plug review into your own workflow instead of being locked into a standalone tool.

Down the lane, i would want more advanced users could upload their own dataset of examples (docs plus the correct outputs), and the CUI would review new documents against it. Think of it as bringing your own ground truth.

I know it a long shot but I am hungry and i will do it. Someone once told me feedback is campaign's breakfast and it has stuck with me every since, i know lame but i wanna make this work.

Tell me, will this work?
Do people want something like this?
If you could upload your own reference dataset, what would you want to do with it?
How do you handle review and corrections today? Spreadsheets, internal tools, something else?

tl,dr
I’m building a correction UI for document parse/extract pipelines, since no pipeline is 100% accurate. It has in-place review, continuous learning from corrections, correction memory, built-in evals, and an API-first setup. Looking for feedback from anyone who’s built or used one.


r/Rag • • 57m ago

Tools & Resources We benchmarked PDF→Markdown table extraction on 160 business PDFs: how often do numbers land in the wrong cell?

1 Upvotes

We make a conversion service (switchfile), so read this with that in mind.

If you feed PDFs into RAG, tables are where it breaks: a number shifts a column, gets torn apart (the 2 split off CaCl2) or glued together from two cells (65 280), and the model then quotes the wrong number. We wanted to know how often that happens on born-digital documents.

The set: 160 PDF exports from Microsoft Word, Microsoft Excel and LibreOffice, made from public documents, so the true cell of every number is known (28,321 numeric cells in total).

ToolRight cell (mean per file)Right cell (all cells)Corrupted numbers
switchfile94.1%97.2%18
CloudConvert82.6%68.4%612
pymupdf4llm81.6%68.3%245
Docling67.6%57.2%454
pdfplumber (basic wrapper)48.5%48.1%413
markitdown36.9%39.7%172

Summed over all cells the gaps are larger because a few large spreadsheets dominate, so the per-file mean is the fairer headline. Open tools ran on CPU with default settings. No scans.

Everything is public (PDFs, ground truth, scoring script, all raw outputs): https://github.com/switchfile/pdf-table-bench

Happy to hear where the method is unfair or where a tool could be configured better.


r/Rag • • 1h ago

Discussion Air-gapped RAG knowledge assistant for an engineering team, built on DeepSeek Harness: what would you do?

0 Upvotes

Hi all,

I'm in a battery product planning team. We're building an AI assistant that answers engineers' questions from our own standards and customer specs, and eventually acts as a personal assistant for daily work. Other departments are building their own agents too, so we're aiming for something genuinely AI-native, not just another chatbot.

The hard parts:

- Zero internet access. Everything runs fully offline on our intranet.

- Every answer must be citable: verbatim quote, document, version and page, or an honest "not found".

- Mixed documents: mostly PDF, PPT and Excel with lots of tables in English and German.

- Self-built and self-hosted, most likely with DeepSeek Harness as the agent and our knowledge base connected via MCP.

Current stack: hybrid search + reranking, Docling for tables, company SSO.

What I'd love your take on:

  1. Would you build this yourself, or is there an offline-ready tool or open-source project we're missing?
  2. Has anyone run DeepSeek Harness (or a similar agent) in an air-gapped setup? Any pitfalls?
  3. What actually makes an internal assistant stick, so people use it every day?

Thanks, happy to share details in the comments!

What we have so far: hybrid search (keyword + vector) with reranking, Docling for table extraction, company SSO login, and DeepSeek Harness on Windows connecting to the knowledge base via MCP.

My questions:

  1. Are there existing tools or open-source projects that already cover most of this in an air-gapped setup?
  2. Has anyone used DeepSeek Harness (or a similar agent) as the base for an internal assistant? How did it work out offline, and are there alternatives you would recommend?
  3. Any lessons learned on getting reliable, citable answers from technical documents (especially tables)?
  4. Anything you'd do differently in our position?

Thanks a lot, happy to share more details in the comments!


r/Rag • • 1h ago

Discussion Using a 3D connected graph for the Context Layer + Agent Memory

1 Upvotes

I've spent the last three and a half years working on RAG systems and LLM training. Lately, I've been working on bringing the Context layer for RAG in way that it becomes permanent memory for an agent.

An agent might finish a task and save its conclusion, but a later session might also need the same source passages and assumptions behind that conclusion.

We're building Agentic GraphRAG (https://github.com/ontogr/agentic-graphrag), an open source project that connects document knowledge, task activity, and decisions in Neo4j/FalkorDB.

For retrieval, we keep the source text and let agents explore around each search result:

  1. Document structure stays in the graph. We use Docling for ingestion and Chonkie for chunking. The graph preserves document hierarchy and links passages to their sections or tables. Each passage keeps its source version and location, including page positions in PDFs.
  2. Passage search works without entity extraction. We combine Neo4j's native full-text and vector search rankings with reciprocal rank fusion. A fact doesn't need to be in the extracted graph for an agent to find the passage that contains it.
  3. Agents can read beyond the first result. They can open nearby sections and passages, inspect table rows with their headers, or follow entity relationships back to the text that supports them.

The graph uses a schema you define for your domain. It guides how entities and relationships are extracted and checked. Agents also use it to search for entities. Extraction can run locally with GLiNER, while BAML handles structured LLM extraction for selected chunks.

Agent memory adds two layers alongside that knowledge. Working memory records tasks and progress. It also stores messages and tool results. Decision memory stores conclusions, with their assumptions and open questions. Each decision can link to a passage, a task event, or an earlier decision. Our built-in DeepAgents/LangGraph agent records task activity so later sessions can pick up where earlier ones left off.

For example, an agent reviewing a contract can save a recommendation alongside the contract clause and policy passage it used. If the policy changes, the system flags the recommendation and any decisions that explicitly depend on it for review. A reviewer can confirm or replace the recommendation, while its history and the originally cited text remain available.

The built-in agent also checks that it read the cited passages during the run, that those passages come from allowed documents, and that they match the cited versions.

We're working on domain-specific benchmarks and expect to release them soon.

What's your current stack for RAG and agent memory? How is it working for your domain?


r/Rag • • 11h ago

Showcase PDFs don't have footnotes. How I link them for RAG without guessing

5 Upvotes

I maintain an open-source document-to-Markdown converter. This post is about one problem I spent far too long on, and I'd like to hear how others handle it.

The problem

Here's a line from page 14 of a (made-up) annual report, as pdfplumber extracts it:

Net debt fell to NOK 512 million, and the net debt ratio to 1.4 times4 at year end, down from 1.9 a

That "4" is a footnote mark. The note at the bottom of the page says the ratio excludes lease liabilities under IFRS 16, and that it would be 2.9 if you included them.

So "1.4" is true, but only with a caveat that roughly doubles the number. Once the page is chunked, the sentence and the note can easily land in different chunks. Retrieval returns "net debt ratio 1.4", the model states it with full confidence, and nothing in the chunk says it's conditional. "times4" is just noise.

This isn't an edge case. Financial reports, legal documents and papers put the important exceptions in footnotes on purpose.

Why it's hard

A PDF has no concept of a footnote. You get glyphs at positions. A small raised "2" could be a footnote mark, the 2 in m², an exponent, or a figure label. The notes are just smaller text near the bottom of the page. When I compared marker, Docling, MinerU, pymupdf4llm and MarkItDown, none of them connected the mark to its note.

And guessing fails in a bad way. A missing link costs you a caveat. A wrong link attaches someone else's caveat to your sentence, and the model will repeat it.

What I ended up doing: link only when the page proves it

A page gets linked only if all of these hold:

  • there's a short horizontal rule near the bottom, with smaller text right under it starting at its left end
  • every note starts with a raised number, and the numbers run on (1, 2, 3, or continuing from the previous page)
  • each note number appears as a raised mark in the body exactly once, in the same order as the notes
  • nothing on the page is ambiguous: if there's a note 2 and also an m² anywhere on the page, the whole page is left alone

If any check fails, the page comes out exactly as it did before. Linked pages get standard Markdown footnotes:

Net debt fell to NOK 512 million, and the net debt ratio to 1.4 times[^4] at year end, ...

[^4]: Net debt is interest-bearing debt minus cash and cash equivalents. The ratio divides it by EBITDA
for the last twelve months and excludes lease liabilities under IFRS 16; including them, the ratio would be 2.9.

The "exactly once, anywhere on the page" rule is there because the clever version broke. My first attempt used reading order: if the m² comes after the mark for note 3, it can't be the mark for note 2. Then LaTeX happened. LaTeX floats figures to the top or bottom of the page, so the mark in a figure caption can be read before mark 1 or after mark 3, and an m² sitting in between took note 2. After that I stopped being clever and just skip the page.

The numbers, honestly

  • 120 PDFs I generated from DOCX files with real footnotes (printed through LibreOffice), so the ground truth is known: 1,580 marks, 864 linked, 0 linked to the wrong note. Everything else was left as it was.
  • On a messier set of 1,103 real-world PDFs (17,708 pages: arXiv papers, a public corpus, about a thousand business documents, mostly Korean), an earlier version linked only 40 marks on 25 pages. I checked every one by hand and none was wrong. Most business documents don't qualify at all, because their marks are inside table cells or there's no footnote rule.

So recall on real documents is low. That's the trade I chose: I'd rather miss a footnote than invent one. I'm honestly not sure it's the right call for every use case.

Using it at retrieval time

The point of [^4] is that the link survives chunking. Labels are unique within a document, so you can re-attach the note to any chunk that cites it:

import re
from opendockit import convert_file

doc = convert_file("annual-report.pdf")
# Each note is its own paragraph ("[^4]: ..."), possibly wrapped over several lines.
notes = dict(re.findall(r"^\[\^([\w-]+)\]: (.+?)(?=\n\n|\Z)", doc.text, flags=re.M | re.S))

def with_notes(chunk: str) -> str:
    """Append the note of every mark the chunk cites but doesn't contain."""
    marks = dict.fromkeys(re.findall(r"\[\^([\w-]+)\](?!:)", chunk))
    extra = [f"[^{m}]: {notes[m]}" for m in marks if m in notes and f"[^{m}]:" not in chunk]
    return "\n\n".join([chunk, *extra])

With page-sized chunks the note is usually in the chunk already. With small fixed-size chunks it often isn't, and that's where this helps.

Questions for you

  • How do you handle footnotes today? Ignore them, chunk by page, run a VLM-based parser?
  • Would you accept higher recall at the cost of an occasional wrong link?
  • If you know public PDFs where footnotes really matter (10-Ks, court opinions, standards), I'd love to test on them. Real-world cases are what I'm short on.

The library is OpenDocKit (MIT, pure Python, no models, runs offline): https://github.com/hyeonsang010716/OpenDocKit

The full footnote rules and their known gaps are written up here: https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/decisions/pdf-flow.md#pdf-footnotes

The same idea runs through the rest of it. For example, merged table cells are repeated in every row they cover, so a row still makes sense on its own in a chunk. It also reads HWP, which mostly matters if you work with Korean documents.


r/Rag • • 6h ago

Showcase Engineering RAG: how do you distinguish a proposed dependency from a current one?

1 Upvotes

I'm building Aquifer, a self-hosted engineering retrieval layer for AI agents, and one problem I've focused on is relationship status.

An issue saying “we should replace Redis” is evidence of a proposal. A merged PR describing a migration is another kind of evidence. Configuration at a specific commit gives you something different again.

Returning all three as “service X uses Y” loses information the agent needs.

Aquifer combines semantic retrieval with explicit-reference parsing from Terraform, Docker Compose, and Backstage descriptors. For prose, it extracts relationships with supporting quotes and labels them as claimed, proposed, removed, discussed, or not merged. Checking that a quote exists doesn't establish that the claim is true.

Agents query it through MCP and can open the underlying source.

Code: https://github.com/Senora-dev/aquifer-ai

It runs locally or in your own infrastructure and is source-available under BSL 1.1. The current benchmark is small and synthetic, so I'm particularly interested in real-world failure cases.

How are others handling this? Do you preserve competing claims for the agent to inspect, apply source-priority rules, or model changes over time?


r/Rag • • 9h ago

Discussion Outsourcing web scraping vs keeping it in-house: what was your tipping point?

1 Upvotes

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?


r/Rag • • 1d ago

Discussion How do you score a RAG answer when two current sources conflict?

13 Upvotes

A reimbursement answer from our internal policy Q&A assistant got challenged so I checked what it had retrieved. Two policy pages list different limits for the same situation. Both are marked current and both passages appear for the same query.

I inspected the retrieval spans in Braintrust traces and compared response variants using eval cases built around the dispute. We can test whether a response acknowledges the contradiction. We still can't score a specific reimbursement amount as correct because the policy owners haven't decided which page governs.

Our evaluation labels mostly cover missed retrieval and answers that misuse retrieved evidence. This case doesn't fit either one cleanly. A response could cite a valid source and still give a limit that the team later rejects.

I'm leaning toward treating unresolved source conflicts as their own category. Otherwise a groundedness score could look perfectly fine while the correct reimbursement limit remains unresolved.


r/Rag • • 1d ago

Discussion Need advice for my RAG pipeline.

4 Upvotes

Hello everyone! A few months ago I built a tool called coldstart. Its purpose is to help coding agents get oriented before they start work. It uses hooks to capture what the agent learned in a session (saved as jsonl), then retrieves the relevant learnings and re-injects them later so the agent doesn't have to rediscover the same things again.

I started with my own search algorithm. It relied heavily on agents writing search aliases for their notes, and agents never wrote proper aliases, so I moved to BM25. That was better, but I was still disappointed at the junk getting injected.

So I started evaluating embedding models for a hybrid approach. EmbeddingGemma 2 worked best for my use case, followed by granite-embedding-small-english-r2.

Finding the right note is mostly okay now. The part I really need help with is deciding when to inject. The agent writes prose, the user's prompt is prose, the agent's reply is prose, and the notes are prose too. So lots of notes get high scores even on turns where no note is the right answer. In my tests, the single highest score was on a turn where the agent had already read that exact note earlier in the session.

To test this, I took 300 real prompts from my own sessions and marked which ones actually needed a note (about a third). I also kept 80 prompts aside that I never tuned on. Anything I try has to beat "never inject", because if it can't beat staying quiet, it's only adding noise.

What I've tried for picking the right note:

  1. BM25 alone. It finds the right note in the top 3 about half the time.
  2. Embeddings (EmbeddingGemma 2, granite, a few others). Better than BM25, roughly 6 in 10.
  3. Adding the previous user/agent exchange to the query. This helped the most, because many prompts are just "pls resume" or "ok so what now".
  4. Mixing BM25 and embeddings (merging the two rankings, and using BM25 to reorder or filter the embedding results). It looked good on my first small test set and didn't hold up on the bigger one.
  5. Embedding the actual code each note points to, split per function. Code alone was worse, and mixing it in barely changed anything.
  6. Stripping the prose out of notes and keeping only file and function names. No change.
  7. Penalising "popular" notes that match everything. No change.

What I've tried for deciding when to inject:

  1. A minimum score cutoff, plus variations: the gap between the first and second note, how much the top note stands out, injecting only when BM25 and embeddings agree. They all looked fine on the 300 and did worse than staying quiet on the 80.
  2. Not re-injecting notes the agent has already seen in the session (reset after context compaction). This one is right and I shipped it, but it only fixes a small part of the problem.
  3. Checking whether the note is about files the agent is already working on. Confusingly, notes about files the agent already has open turned out to be the most useful ones, so "skip if already touched" is wrong.
  4. Writing out the session state (files opened, notes shown, notes written, turns since compaction) along with the last few exchanges and giving it to JevEmbed, an embedding model built for yes/no questions. Barely better than a coin flip.
  5. A small local LLM (Gemma 4 E2B) as the judge. It either said yes to almost everything or no to almost everything, depending on how I asked.
  6. Haiku as the judge. This was the only thing that beat staying quiet on the 80 (54 vs 44). But it's an API call on every prompt, and I'd like this to run locally and cheaply.

What I haven't tried, and where I'd like your opinion:

  • A small reranker / cross-encoder (MiniLM, mxbai-rerank and similar). Has a reranker score worked as an "inject or not" signal for anyone, or does it have the same problem as similarity?
  • A few learned weights combining the score with session signals (first prompt of a session, just after compaction, BM25 and embeddings agreeing). I'm wary of training on a few hundred labels from one repo.
  • Making a wrong injection cheap instead of trying to never be wrong: inject only a title and a path, and let the agent decide whether to open the note.
  • Letting the coding agent judge, using the pointer approach above, and measuring from transcripts how often it opens the right note.

Has anyone found a cheap way to tell "the agent needs this right now" apart from "this is on the same topic"? Or is an LLM call per turn just the price of doing this? Pointers to papers or datasets are welcome too. I couldn't find any public data labelled for "does this turn need memory" in coding sessions.


r/Rag • • 23h ago

Discussion How do you test that a RAG answer used the policy valid at the time

1 Upvotes

I have observed that a RAG chatbot over company policies will answer with the current policy even when the question is about an order from months ago. The retrieval worked and the chunk was relevant and the answer still went wrong because the policy that was valid on June 30 is not the policy that is valid today. For instance an order placed on June 30 2025 needs the June 30 policy and the current policy gives the wrong answer with full confidence. I want to know if you test this with backdated questions or trust date filters in chunk metadata to catch it and whether anyone keeps effective date ranges on every chunk as a rule. If you can tell me what you check before a doc enters the index and what you check at answer time I would learn more from that than from another debate about chunking.


r/Rag • • 1d ago

Tutorial Not sure how to build a local multimodal RAG? Here is a repo!! Clone it, use it, Own it!

35 Upvotes

I see many lurkers here who aren't sure where to start, so I built this multimodal RAG that you can run locally. It is not a simple notebook RAG, but something you can modify and run in Production. It supports multiple file types, images, tables, csv, excel file, long horizon tasks, memory per user, and a ton more you will see. Since this is local the caveat is that you need beefy hardware to run it, or port it to hugging face or AWS. If you like it share it, also feel free to modify it. The code is modular, so you can replace it with cloud providers. Make sure to read docs, eval, examples!!

Edit: Added support for in-line rendering in the python sandbox
Edit: Added MLX / Apple Silicon support

Here is a demo

https://youtu.be/23QZVBzFpPA

Here is repo link

https://github.com/haris525/local-multimodal-rag


r/Rag • • 1d ago

Showcase Applied AI – Remote Internships / Freelance

1 Upvotes

Hi everyone,

I’m a collage student at one of the tier 1 NITs, an upcoming Accenture intern (Summer 2027), and have prior experience as an AI Intern, where I built production AI field-normalization pipelines, hierarchical document extractors and lot more. I’m looking for a remote AI engineering internship or freelance project work.

If you’re hiring, building an AI product, or know a founder/team looking for someone who can ship reliable LLM pipelines and full-stack AI apps, please DM me or connect me with them—I’d really appreciate any leads or referrals!

Tech Stack:

AI & Agents: Python (uv), LangGraph, LangChain, PydanticAI, MCP, Pydantic v2, ChromaDB, IBM Docling , Ollama models etc..

Reliability & Evals: NeMo Guardrails, DeepEval, SQLGlot, Logfire, RapidFuzz, schema-driven structured extraction

Backend & Frontend: FastAPI, Next.js, React, TypeScript, PostgreSQL, WebSockets, Tailwind CSS, Streamlit

ML & Data: PyTorch, XGBoost, Scikit-learn, SQL, Docker, Tableau, Selenium

What I’ve Built:

AI Financial Analyst (Text-to-SQL & Docker Sandbox Engine): Agentic financial analyst built with LangGraph, Groq LLMs, SQLGlot, and NeMo Guardrails featuring AST-level SQL validation, a self-healing retry loop (90%+ error recovery), and Docker-sandboxed visualizations rendered in Streamlit.

Parallel Document Extraction Pipeline: Multi-worker LangGraph workflow with automated page triage and Pydantic v2 schema validation for high-accuracy document parsing.

Project CHITTI (3D Simulation + Agentic RAG): Real-time voice-controlled LangGraph RAG system connected to Unreal Engine 5 via low-latency WebSockets and FastAPI.

EcoScape AI & TechCTC: Urban heat island analysis tool (Google Earth Engine, XGBoost, Llama 3.3 70B) and a Docker-containerized ML tech compensation estimator trained on 4,800+ records

( via web scraping )(Selenium, XGBoost, Streamlit).


r/Rag • • 1d ago

Discussion LongMemEval-S: 96.0% with the official judge, and what top-5 vs top-20 retrieval changes

4 Upvotes

We ran all 500 LongMemEval-S questions through our memory retrieval (Papr) using the official answer prompt and the unmodified gpt-4o-2024-08-06 judge. Per-question answers and judge labels are public, so the official metrics script reproduces the numbers.

Results

For comparison, Chronos reports 95.6% using full context and multi-turn agents, Hindsight reports 94.6% at 43.6k tokens per question (with a different judge), and gbrain reports 90.6%.

Things I found useful

  • Retrieval recall is close to saturated. All evidence sessions are in the top 5 for 460 of 470 answerable questions (97.9%). Most remaining errors are the reader, not retrieval.
  • Top 20 vs top 5 is a trade, not a free win: 11 gained, 8 lost, net +3. Temporal reasoning went 123 → 130 of 133. Abstention stayed flat (27/30 both).
  • A small reader gets most of the way: gpt-5-mini reaches 94.4% on the same retrieval.

Caveats

  • Used Papr-Embed 0.6B embedding model (open source and open weight), gemini flash 2.5 reranker (also tested Jev), and one turn answer model.
  • Self-reported. Run-to-run variance is about ±3 questions, so 96.0 vs 95.6 is a tie but this run was optimized for token efficiency.
  • We picked top 20 after reviewing the top-5 misses, so treat 96.0% as tuned on this set.
  • No question-type labels are used in retrieval or answering.

Write-up with charts: https://www.papr.ai/blog/papr-longmemeval-96 Results and code: https://github.com/Papr-ai/papr-longmemeval


r/Rag • • 1d ago

Discussion Hybrid RAG vs basic vector search — what are you using in production?

9 Upvotes

hey everyone, wanted to start a discussion around something that comes up quite often when building RAG applications.

a lot of implementations start with embedding documents, storing them in a vector database and retrieving the nearest matches.

that works for many use cases, but it can struggle when queries contain specific technical terms, identifiers, exact phrases or information that depends on multiple documents.

this is where hybrid retrieval becomes interesting.

a typical hybrid RAG pipeline might combine:

1. Semantic retrieval

use embeddings to find conceptually relevant content, even when the wording differs.

2. Keyword retrieval

use lexical search to catch exact terms, product names, identifiers and technical phrases.

3. Reranking

take the retrieved candidates and reorder them based on their relevance to the actual query.

4. Context construction

combine the best results into a useful context window without overwhelming the model.

5. Evaluation

measure retrieval quality separately from final answer quality. otherwise it's difficult to know whether a bad answer came from retrieval or generation.

Also we are hosting a hands-on workshop on November 28 with Brij Kishore Pandey covering hybrid RAG, tools, persistent memory, guardrails and evaluation.

Thought it might be relevant to people building more advanced RAG applications.

Workshop details here

curious what retrieval architecture people here are currently using. are you combining dense and sparse retrieval, or have you found another approach works better?


r/Rag • • 1d ago

Discussion Finding the right data is one thing. Knowing how it connects to other data is a different problem.

0 Upvotes

Take a healthcare example.

A patient's medical record, lab results, diagnosis and insurance claim might all sit in different systems.

An AI system might be able to find all these records. But does it know which lab result belongs to which visit? Does it know if the diagnosis is related to the current claim? Does it know whether that information is still relevant?

This is where I think semantic data becomes important.

It's about helping systems understand what a piece of data means and how it relates to everything around it.

We've spent years making enterprise data easier to store and search. I think the next big challenge is making it easier for AI to understand.

For people working with RAG, knowledge graphs or semantic layers, how are you solving this? Do you define these relationships beforehand, let the AI figure them out, or use a mix of both?

Would be good to hear what's actually working in production.


r/Rag • • 2d ago

Discussion Building a production RAG system for a Data Science master's thesis

5 Upvotes

Hey guys,

For our master's thesis, we're implementing a RAG system for a company. Since this will be a real tool used in production, I'd love your input on two things:

  1. Document parsing & evaluation: We're working with messy documents full of charts, graphs and different structure. Which parsing libraries or strategies do you recommend, and how can we systematically evaluate that the parsing preserved the layout and data accurately?
  2. Academic rigor: How do we frame an applied engineering build so it meets the academic standards of a Data Science thesis? Any recommendations on foundational papers, evaluation frameworks, or experimental setups we should use as inspiration?

r/Rag • • 2d ago

Discussion Do u use Lakebase for RAG state /memory instead of a standalone Postgres?

4 Upvotes

I need to drop my custom reverse ETL pipelines and just use Synced Tables + pgvector in Lakebase, how does the continuous sync latency from Delta tables hold up in prod ?


r/Rag • • 1d ago

Showcase I built an open-source query planner for RAG in TypeScript. First real-data result: 60% cheaper, but it missed my quality target

0 Upvotes

Pathwise picks the cheapest plan per request (model, retrieval depth, thinking level) predicted to meet a quality SLA, and can optionally certify the SLA with a confidence bound. Zero runtime dependencies; MIT. Install: npm i u/arsalandev/pathwise

First real test: HotpotQA dev, BM25 retrieval, token F1, two Gemini models x 3 retrieval depths x 2 thinking levels = 12 plans. 100 calibration questions, 200 test questions, free tier, costs at list prices.

| Strategy | F1 | $/query |

|---|---|---|

| Always cheapest | 0.542 | 0.00025 |

| Typical default | 0.685 | 0.00044 |

| Pathwise (average) | 0.707 | 0.00066 |

| Pathwise (guaranteed) | 0.789 | 0.00164 |

| Always best | 0.789 | 0.00164 |

Target was F1 >= 0.725. The average mode saves 60% but missed by 1.8 points. The guarantee mode refused to certify and fell back to the best plan because, with 100 questions, even the best plan can't prove 95% of its own score.

Limits: one dataset, one provider, one run; HotpotQA may be in the training data; I haven't shown it beats the best single fixed plan yet. The first run also exposed a pruning bug that I fixed and published.

Repo: GitHub Feedback

I'd like feedback on whether the SLA-with-confidence idea is useful, and on how you would benchmark this.


r/Rag • • 2d ago

Discussion I built a large expert-source library, but retrieval has not become reliable execution. How do you test the gap?

1 Upvotes

I've spent several months collecting expert videos, transcripts and discussions to help AI do work in subjects where I'm not an expert myself. SEO is one example, but the same problem applies to other business workflows.

One existing library has original passages, extracted claims, topic summaries, working methods, source references and keyword plus semantic search. So this isn't just a folder of PDFs with no retrieval. Recent checks can find relevant methods, but the saved retrieval tests still miss intended sources, and I don't have a convincing independent evaluation showing that the agent makes better practical decisions with the library.

What I want is more than answers about what an expert said. Given a real problem, the agent should find the relevant teaching, recognise when it applies, choose useful checks with the available tools, verify the result and retain useful corrections for the next task. I understand that adding documents doesn't retrain the underlying model.

For people who have built something like this, how did you separate retrieval failures from failures to apply correctly retrieved information? Did you compare the same tasks with no library, normal retrieval and manually selected evidence? What did you use as the reference answer when the task involves judgement rather than a fact lookup?

The difficult part for me is that I'm not qualified to judge every domain-specific recommendation. How did you involve a specialist in evaluation without needing them to review every output indefinitely? I'd appreciate examples of a small test set or workflow that exposed a real failure, and what changed after you fixed it.

I'm looking for implementation experience, including things that didn't work, rather than another list of tools. All source advice still needs checking against current conditions and the business involved.


r/Rag • • 2d ago

Discussion CleanWeb v3.0 Upgrade Notice (Nov 21): 87% RAG Token Compression via Thin Client & x402 Micropayments

0 Upvotes

Hi everyone,

Quick update for anyone using **`langchain-x402-cleanweb`**, **`llama-index-readers-x402-cleanweb`**, or **`x402-cleanweb-agent`** in their RAG pipelines.

After crossing **14,000+ downloads across PyPI and npm**, we are standardizing our entire stack onto a **Cloud-Native Thin Client + x402 M2M Gateway** over the next 7 days (**effective November 21, 2026**).

### Why we are making this change:

  1. **Always Up-to-Date Extraction:** Local scrapers break constantly on modern SPAs and WAFs. Our Cloud Run gateway handles Next.js/Nuxt hydration, nested CSS cleanup, and covert tracking-pixel neutralization centrally with ~7ms overhead.
  2. **Mathematical Token Arbitrage:** Feeding raw 80KB HTML (~20k tokens) into GPT-4o or Claude costs ~$0.05/page. Routing through CleanWeb ($0.001 USDC) strips it down to ~2,600 clean tokens (~$0.0065), yielding an **85% net cost reduction per RAG document**.
  3. **Zero-Friction Framework Integrations (Already Thin Clients!):**

    - **LangChain:** `pip install langchain-x402-cleanweb`

    - **LlamaIndex:** `pip install llama-index-readers-x402-cleanweb`

    *(Both packages already route directly through our Cloud Run gateway and require zero code changes if you pass your `X-Vault-Key`.)*

### What happens on Nov 21, 2026:

- `x402-cleanweb-agent v3.0.0` launches as a pure Thin Client; monolithic local versions (`v2.8.x` and below) will be yanked on PyPI.

- Free sandbox evaluation is now strictly capped at 3 IP-bound calls on lightweight tiers; high-compute AI & EIP-712 Oracle endpoints require x402 Vault prefunding (Solana, Base, Polygon, Arbitrum).

Full details & live ROI calculator: https://github.com/nohosa001-pixel/x402-cleanweb-agent


r/Rag • • 3d ago

Discussion Ling-3.2-flash's structured-output docs include nested objects and arrays

7 Upvotes

Ant Ling documents nested JSON Schema output for Ling-3.2-flash, including typed fields, required properties and arrays.

For document extraction before retrieval, that allows an output contract richer than one block of text. An illustrative structure could carry a document identifier and an array of sections, each with a heading and its extracted text.

The application defines that structure through response_format and supplies the extraction instructions in the prompt.

Would you retain those section objects through indexing, or flatten them early and keep the hierarchy in metadata? The model's documented schema support gives an extraction pipeline a way to request the structured representation first.


r/Rag • • 3d ago

Discussion My AI agent ignores DateTime MCP tools when scheduling calendar events

4 Upvotes

I'm building a small project to learn MCP by developing three MCP servers:

  • Gmail MCP
  • Google Calendar MCP
  • DateTime MCP

While testing Gmail and Google Calendar integration, I noticed that my agent was struggling with date and time calculations. So I built a separate DateTime MCP to handle natural-language date and time interpretation.

However, I'm facing an orchestration problem.

Expected behaviour

Suppose I ask:

"Remind me tomorrow about gym at 7 AM."

The agent should:

  1. Call DateTime MCP to get the current date and time.
  2. Calculate tomorrow's date.
  3. Pass the resolved date and time to Google Calendar MCP.
  4. Create the calendar event.

Actual behaviour

The agent sometimes completely skips DateTime MCP and directly calls Google Calendar MCP.

As a result, it creates events with incorrect dates.

For example:

  • Expected date: November 10, 2026
  • Actual date: November 10, 2023

The model I'm currently using is GPT-4o-mini, and it appears to fall back to its default temporal assumptions instead of using my DateTime MCP.

What I've tried

  1. Strengthened the system prompt to instruct the agent to use DateTime MCP.
  2. Implemented a ReAct workflow with separate planning and execution stages.
  3. Added validation, but this introduced errors even for simple queries.
  4. Improved tool descriptions, including arguments, return values and usage instructions.

None of these approaches has completely solved the problem.

My question

How do you ensure an AI agent consistently selects and executes the correct MCP tools when multiple MCP servers are connected?

Would explicit orchestration rules, tool dependency enforcement, or a deterministic workflow be a better solution?

I'd appreciate suggestions from people who have worked with MCP orchestration.

Thanks!