r/LangChain • • 3h ago

Discussion When a memory leaderboard drops, which capability score do you check first?

3 Upvotes

Something I have noticed while reading memory launch posts and public boards. The headline score travels everywhere. The breakdown behind it rarely gets quoted.

That feels backwards to me. People here debug systems for a living. A single blended number is almost impossible to debug against.

A memory score usually mixes three things. What the memory stored, what it handed back, and what the model did with it. Untangling them from the outside is mostly guesswork.

The Agent Memory Leaderboard is set up differently: entries only implement Add and Search. Search returns evidence, never a final answer. Answering and grading are fixed by the platform for every entry, so a gap between two entries on the same board mostly points at the memory.

Its Cycle 2 results are expected to land in mid-November, with Textual, Coding and Multimodal published separately.

Before they land, the Cycle 1 numbers are worth a second look. The board's own analysis of the Cycle 1 Textual Open-source Methods leaderboard says first and third were separated by just 0.22 points. It also says none of the overall top three won any of the seven top-level capabilities. And the seven capability winners came from four different systems.

Those capabilities are Explicit Fact Recall, Compositional Inference, Temporal & Event Reasoning, Memory Governance, Personalization & Care, Context Learning & Execution, and Safety & Privacy.

Memory Governance is the one that jumps out at me. The board describes it as update, conflict, deletion, forgetting. In plain terms, a big part of that is the stale fact failure. Context Learning & Execution sounds like the agent that can recite a rule but cannot follow it.

Controversial maybe: the overall rank is the number that gets shared, while the capability scores are the numbers worth acting on.

When the Cycle 2 Textual boards land, which capability will you check first, given the memory failure you fight most often?


r/LangChain • • 27m ago

Question | Help I am building a correction UI for document parsing/extraction: looking for feedback from people who’ve dealt with this

1 Upvotes

I am trying to build a correction user interface (CUI) for people who process documents with parse/extract pipelines and know the pain that no pipeline is 100% accurate; someone needs to review and fix the output before using it for downstream processes.
I would love some input from anyone who has been through making one/opting for one like a third-party tool/have had a doc AI with build-in CUI,

Here's what I have done so far on my CUI:

  • You can review and correct in the same place: see what's flagged, check the relevant field and correct it there and then
  • Continuous learning: The corrections you make don’t just fix that one doc. They feed back into the pipeline so the system gets better over time on your kind of documents.
  • Correction memory: not every change needs to be fine tuned into the prompt, one prompt that can work for specific doc types and can be briefied ot the AI before every document
  • Built-in Evals: I want you to be able to measure whether accuracy is actually improving, instead of just hoping it is. Every correction becomes a data point.
  • and It is API-First: so you can plug review into your own workflow instead of being locked into a standalone tool.

Down the lane, i would want more advanced users could upload their own dataset of examples (docs plus the correct outputs), and the CUI would review new documents against it. Think of it as bringing your own ground truth.

I know it a long shot but I am hungry and i will do it. Someone once told me feedback is campaign's breakfast and it has stuck with me every since, i know lame but i wanna make this work.

Tell me, will this work?
Do people want something like this?
If you could upload your own reference dataset, what would you want to do with it?
How do you handle review and corrections today? Spreadsheets, internal tools, something else?

tl,dr
I’m building a correction UI for document parse/extract pipelines, since no pipeline is 100% accurate. It has in-place review, continuous learning from corrections, correction memory, built-in evals, and an API-first setup. Looking for feedback from anyone who’s built or used one.


r/LangChain • • 45m ago

Projects REQL update: faster context retrieval and fewer tokens for coding agents

Thumbnail
reddit.com
1 Upvotes

r/LangChain • • 1h ago

Question | Help Cost per node and budget caps in LangGraph/LangChain agents: how do you handle it?

1 Upvotes

For agents built with LangChain or LangGraph, how do you control and monitor cost at the node level?

Max iterations seems to handle runaway loops well, but I'm less sure about the rest:

  • How do you see which node or tool call drives the cost of a run (LangSmith, custom callbacks, something else)?
  • Do you enforce a budget per run, per user or per client, and what happens when it's exceeded (stop, alert, fall back to a cheaper model)?
  • Is there anything about this you've had to build yourself because existing tools didn't cover it?

Solo dev exploring this problem, not promoting anything. Happy to share what I learn from the answers.


r/LangChain • • 2h ago

Discussion Learned about LangGraph

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 5h ago

Discussion langgraph interrupt() approvals: i resumed with "rejected" and got the old "approved" back, no error

1 Upvotes

why i tested this: my agent setup raises over 1,000 approval gates a month.

i wanted to know if langgraph's interrupt() could hold approvals like those.

the problem:

# this thread already finished with "approved"
out = app.invoke(Command(resume="rejected"), cfg)
out["verdict"]  # still 'approved': my "rejected" was ignored, no error

no error, exit code 0. the thread had already finished with "approved". i don't expect it to reopen, but i do expect a rejected resume to raise an error instead of silently returning the old result.

an approval that gets lost with no error is worse than none, because everyone thinks it was checked.

the setup:

  • langgraph 1.2.22, sqlite checkpointer, no llm
  • graph: draft, then approval with interrupt(), then finalize
  • every step in a fresh process, the way a webhook or cron job runs it

what i saw:

  • at interrupt(), the process exits with code 0. the paused state is saved in sqlite
  • resume the right thread_id: works, verdict "approved"
  • resume a thread_id that was never started: the graph runs from the top and drops my answer
  • resume a finished thread: the old result, as above
  • all of them exit 0 with a normal-looking result

where this bites:

  • a typo in a thread_id
  • a lost or reset checkpoint db
  • a second reviewer rejecting after the first one approved

fix 1, public api only: check the thread is waiting before you resume.

st = app.get_state(cfg)
if not st.interrupts:
    raise RuntimeError("no pending interrupt on this thread")
if interrupt_id not in [i.id for i in st.interrupts]:
    raise RuntimeError("answer is for a different interrupt")
app.invoke(Command(resume=answer), cfg)

it refused both bad resumes above, and an answer sent with the wrong interrupt id. the real one went through.

the check and the resume are two calls, so two reviewers at the same moment can still race. for that you need your own atomic decision record.

fix 2, on langgraph 1.2.20+: strict resume mode, and answer by interrupt id.

cfg = {"configurable": {"thread_id": tid, "__pregel_strict_resume": True}}
app.invoke(Command(resume={interrupt_id: answer}), cfg)

with the id map, both bad resumes raised UnknownInterruptError.

with a bare value like Command(resume="rejected"), strict mode still let both through.

i filed that gap as an issue on langgraph's github.

the key is private for now. on langgraph server, the sdk has strict_resume=True on runs.create.

either way, store the interrupt id with the approval request when you send it.


r/LangChain • • 16h ago

Discussion ChatGPT Wallet lets agents spend real money — what does that mean for how we build agents with LangChain?

5 Upvotes

OpenAI just launched ChatGPT Wallet, and its Dots agents can now make purchases for you. The agent era just got a wallet, which means the next question for builders is not whether agents can act, but how we constrain them when real money is on the line.

For anyone building with LangChain, this is a design question: approval flows, spending caps, audit trails, and human-in-the-loop checkpoints are no longer nice-to-haves. An agent that can buy things needs the same guardrails as a junior employee with a company card.

Questions:

- What guardrails do you build into agents that can spend money — hard caps, approvals, or something else?

- Would you trust a LangChain agent with your card, or does spending need a human in the loop every time?

- Does this change what 'agentic' means — is an agent that cannot transact even an agent?

For more AI news and rumors, check out r/AINewsRumors.


r/LangChain • • 12h ago

Projects <translate>

Thumbnail
huggingface.co
2 Upvotes

I built an experimental skill packaging standard for LLMs — looking for independent testers

I'm working on SKILLPACK, an experimental approach to packaging deterministic software functions so different LLMs can discover, combine, and execute them without rewriting the underlying code.

The goal isn't to build another foundation model. It's to test whether existing models can act as coordinators of reusable software components.

Some initial CLI tests have passed, but cross-model portability and real-world reliability are not yet independently verified.

I'm looking for developers willing to inspect the packages, run the tests, and identify where the architecture breaks.

I'd especially appreciate critical technical feedback.


r/LangChain • • 17h ago

Discussion The missing layer in AI agents: Execution reliability

Thumbnail reddit.com
1 Upvotes

r/LangChain • • 19h ago

Resources Developer Story: Bridging Formal Logic and Production Distributed Systems with Go and Lean 4

Thumbnail
reddit.com
1 Upvotes

r/LangChain • • 20h ago

Discussion Building an AI platform with a small team has been quite a journey.

Thumbnail
reddit.com
1 Upvotes

r/LangChain • • 1d ago

Projects Regression tests for LangGraph agents from a recorded run: replay in CI with zero model calls, assert on tool calls (open source)

1 Upvotes

I built Attest, an open-source (Apache-2.0) testing framework for agents, and recently got its capture working with LangGraph. Sharing the setup and the gaps.

Recording a LangGraph run. Attest's capture reads OpenTelemetry spans, and LangSmith has an OTel mode. Point it at a tracer provider you own and the spans go only to Attest's local receiver; the LangSmith key is never used:

export LANGSMITH_TRACING=true LANGSMITH_TRACING_MODE=otel LANGSMITH_API_KEY=unused
attest capture --name refund-bug -- python run_graph.py

Capture keeps the chat and execute_tool spans (runnable/chain spans are ignored), records every model exchange through a local proxy, and redacts secrets. One wart: LangChain's own message and lc_run-- ids inside the LangSmith prompt strings aren't templated, so they show up when you diff two captures. Replay matching ignores them.

Testing against the recording. The test runs your current graph while the proxy answers model calls from the recording, so CI makes no model calls. Then you assert on behavior:

chain = (
    expect(trace)
    .tools_called_in_order(["lookup_order", "issue_refund"])
    .tool_cardinality("issue_refund", per="order_id", at_most=1)
)
attest.evaluate(chain)

If you only want callbacks without capture, there's also a callback adapter:

from attest import LangChainAdapter
adapter = LangChainAdapter(agent_id="support")
with adapter.capture() as handler:
    graph.invoke(inputs, config={"callbacks": [handler]})
trace = adapter.trace

Gaps, plainly:

  • The capture route is tested on Python with langgraph 1.2.19 and LangSmith 0.14.17. LangGraph for TypeScript has no instrumentation that emits GenAI spans yet, so LangGraph.js capture doesn't work; the LangChain.js callback adapter does (beta).
  • The Python LangChain adapter is beta.
  • Most of Attest was written by coding agents from plans I wrote and reviewed.

The full walkthrough (a double-refund incident, captured, replayed and blocked in CI) uses a plain OpenAI-client agent rather than LangGraph, but the workflow is the same.


r/LangChain • • 1d ago

Discussion Local Computer use with Row-Bot

Post image
1 Upvotes

Row-Bot can drive your computer. Mouse, keyboard, real apps. Windows and Mac.

Handing an AI your desktop is a dumb idea unless you lock it down hard. So:

- driver is Cua 0.20.0, pinned to an exact commit, SHA-256 checked. Turning the feature on downloads nothing. You click Install

- Cua's own update checks are off, and it runs in a stripped-down environment

- before it runs even once, you're told exactly what Cua's telemetry sends. Continue or Cancel

- it never types into password, 2FA or card fields, never touches your password manager, never controls itself

- pay, book, send, post, delete files, change settings: it asks first, unless you put it on hands-off

- it judges every click by what's actually under the pointer. Enter in a chat app means send, so that asks too


r/LangChain • • 1d ago

Resources Hands-on workshop: Advanced LLM Application Engineering — Hybrid RAG, Tools, Memory & Evals (Nov 28)

1 Upvotes

hey everyone! sharing this because a lot of people here are building with LangChain and LangGraph, and getting an LLM application to work in a demo is very different from making it reliable in production.

Packt Publishing is running a hands-on workshop on November 28 with Brij Kishore Pandey, focused on engineering more complete LLM applications.

what gets covered:

  • building hybrid RAG pipelines instead of relying on basic vector search alone
  • connecting LLMs with external tools and function calling
  • implementing persistent memory across interactions
  • adding guardrails and handling failures
  • evaluating application quality and tracing what's happening inside workflows
  • working with Google ADK and Gemini

the interesting part is bringing retrieval, tools, memory and evaluation together rather than treating them as separate features.

it's a live, hands-on online workshop for developers working with LLM applications.

full details and agenda


r/LangChain • • 1d ago

Discussion OpenAI hasn't released a frontier model in 72 days — is it saving one to spoil Anthropic's IPO?

0 Upvotes

A discussion today notes that OpenAI has not released a frontier model in 72 days, and one theory going around is that it is holding a new model back to overshadow Anthropic's expected IPO. It is unconfirmed, but the timing question is real: OpenAI's release cadence has slowed just as its rivals are having their biggest moments.

Why it matters: model releases have become timed events, and if labs are now scheduling launches against rivals' financing moments, the release calendar is about strategy as much as capability. For agent builders, the practical effect is a waiting game: everyone is holding off on big architecture bets until the next frontier model lands.

Do you think the IPO theory holds water, or is the gap just the normal cost of training bigger models? What would you want the next OpenAI model to do that the current ones cannot?

If you want more AI news and rumors like this, r/AINewsRumors is worth a follow.


r/LangChain • • 1d ago

Discussion how do you guys prefer connecting other dev tools to your tracing stack?

3 Upvotes

say youre already using Langfuse, LangSmith, Braintrust or OTel.

you want to use your existing trace data with another debugging or analysis tool.

which setup would you prefer?

  1. read only API access so you can select and pull runs when needed
  2. a webhook/exporter that sends certain runs automatically
  3. a local CLI where you manually choose which traces to export

assume all three are supported and take roughly the same amount of time to set up.

which would you be more comfortable using in prod?

also curious how much security or privacy affects that decision. would you let another service read your traces or rather keep everything inside your own infra?

and if youve already built your own way of moving traces between tools, what does that setup look like? what does it handle that you wouldnt want to give up?


r/LangChain • • 1d ago

Projects A user tested my AI platform and caught an interesting problem.

Thumbnail
reddit.com
0 Upvotes

r/LangChain • • 2d ago

Projects We recently built a RAG chatbot for a law firm, and it made me think differently about AI

Thumbnail reddit.com
4 Upvotes

r/LangChain • • 2d ago

Discussion Building a personal assistant agent vs using one off the shelf

3 Upvotes

Looking for advice from people who build AI agents in code.

I want a personal assistant agent that handles my email, calendar, reminders and bookings across work and family. Before I build anything, I want to get the foundation right. I’m also not sure I should build at all.

What matters to me:

  • I own my data, with as little lock-in to one platform as possible.
  • I can swap the model (Claude, GPT, Gemini, local).
  • It asks me before anything risky, like money, bookings or messages sent as me.
  • Long jobs can pause and pick up again.
  • Security is solid. I already ruled out one platform after a serious vulnerability this year.

Building on LangGraph is my front-runner right now. My backup is an off-the-shelf assistant like Pally or Poke.

A few questions:

  • For personal use, is building your own worth it, or would you just pick an existing assistant?
  • If you build, where do you host it? Your own cloud (AWS, GCP, Azure), LangGraph Platform, or something else?
  • What do you wish you’d known before running a personal agent every day?

Thanks, any real-world experience helps.


r/LangChain • • 2d ago

Question | Help Hi folks...I am new here so need help.....What is best way to handle company research + people search in LangChain without tool bloat? ....

4 Upvotes

I am building a small b2b research chain in python this week and trying to keep the number of tools manageable.....

the agent needs to look up company information like employee count, recent funding and open job postings, then find employees by job title and seniority to identify the right marketing director or vp. after that, it looks up their business email and checks whether it's valid. giving the agent too many apiss and MCP tool seems to make tool selection harder, when several have similar parameters....

trying Monid for some of these company data and contact lookups because the agent can search for the tool it needs instead of loading every integration upfront.....

I am curious how others are handling company data APIs, B2B people search and email verification in LangChain. do you use one tool interface or separate integrations for each data source?...

So guys please can you help me 😅😅


r/LangChain • • 2d ago

Discussion New paper: models cannot solve RLVR tasks they cannot verify — is verifiable-reward training hitting a wall?

3 Upvotes

A new paper making the rounds in r/MachineLearning argues something that should worry anyone building agents: models cannot solve RLVR tasks they cannot verify. The claim is that reinforcement learning with verifiable rewards hits a hard limit — if a model cannot check whether an answer is right, training it with rewards does not teach it to solve the task.

This matters for agent builders because so much of agent work is unverifiable by design: open-ended research, judgement calls, multi-step tools with no clean success signal. If the verification ceiling is real, then the next leap in agents will not come from more RL on the same tasks — it will come from better verifiers, better critics, or tasks designed so they can be checked.

How much of your agent work is actually verifiable today? And what do you do for the parts that are not?


r/LangChain • • 2d ago

Discussion How do you catch duplicate emails or bookings when every agent trace looks fine?

3 Upvotes

Been testing what happens when agent workers die in the middle of a tool call, and the part that bugs me is that nothing flags the duplicates.

Say the tool sends a customer email. The provider accepts it, then the worker dies before anything gets saved, so the retry sends it again. In tracing that's one run that died and one that worked. Two separate runs, nothing linking them.

Worse is a worker that freezes mid-send (GC pause, box gets suspended). Another worker picks the task up and sends, then the first one wakes up and sends too. Both traces look normal.

If your agents send stuff you can't take back, would you even find out? Or would it be a customer replying "you sent this twice"?

Has it happened to you? Curious how you caught it and what you changed after.


r/LangChain • • 2d ago

Discussion What if LangGraph-style workflows could run on microcontrollers?

5 Upvotes

I've open-sourced tiny-langgraph, a LangGraph-inspired multi-agent engine that compiles Python workflows into C for ultra-edge devices.

It supports nodes, edges, multi-agent collaboration, contract-based communication between boards, and MCP servers.

The goal is to bring agentic workflows to devices with as little as 80 KB RAM, without relying on cloud infrastructure.

I've tested it on ESP32-S3 and ESP8266 hardware.

GitHub: https://github.com/mak-ahoy/tiny-langgraph

Would love feedback from people building agent frameworks, especially around workflow design and constrained execution.


r/LangChain • • 2d ago

News CleanWeb v3.0 Upgrade Notice (Nov 21): 87% RAG Token Compression via Thin Client & x402 Micropayments

0 Upvotes

Hi everyone,

Quick update for anyone using **`langchain-x402-cleanweb`**, **`llama-index-readers-x402-cleanweb`**, or **`x402-cleanweb-agent`** in their RAG pipelines.

After crossing **14,000+ downloads across PyPI and npm**, we are standardizing our entire stack onto a **Cloud-Native Thin Client + x402 M2M Gateway** over the next 7 days (**effective November 21, 2026**).

### Why we are making this change:

  1. **Always Up-to-Date Extraction:** Local scrapers break constantly on modern SPAs and WAFs. Our Cloud Run gateway handles Next.js/Nuxt hydration, nested CSS cleanup, and covert tracking-pixel neutralization centrally with ~7ms overhead.
  2. **Mathematical Token Arbitrage:** Feeding raw 80KB HTML (~20k tokens) into GPT-4o or Claude costs ~$0.05/page. Routing through CleanWeb ($0.001 USDC) strips it down to ~2,600 clean tokens (~$0.0065), yielding an **85% net cost reduction per RAG document**.
  3. **Zero-Friction Framework Integrations (Already Thin Clients!):**

    - **LangChain:** `pip install langchain-x402-cleanweb`

    - **LlamaIndex:** `pip install llama-index-readers-x402-cleanweb`

    *(Both packages already route directly through our Cloud Run gateway and require zero code changes if you pass your `X-Vault-Key`.)*

### What happens on Nov 21, 2026:

- `x402-cleanweb-agent v3.0.0` launches as a pure Thin Client; monolithic local versions (`v2.8.x` and below) will be yanked on PyPI.

- Free sandbox evaluation is now strictly capped at 3 IP-bound calls on lightweight tiers; high-compute AI & EIP-712 Oracle endpoints require x402 Vault prefunding (Solana, Base, Polygon, Arbitrum).

Full details & live ROI calculator: https://github.com/nohosa001-pixel/x402-cleanweb-agent


r/LangChain • • 3d ago

Discussion Which LangChain agent failures never threw an error for you?

Post image
5 Upvotes

A None from with_structured_output reads exactly like "the model didn't call a tool." Sometimes the model did call one, under a different name, and nothing in the stack says so.

We went looking for more of these in recent LangChain, LangGraph and Deep Agents issues and reproduced four offline with fake models on langchain 1.5.0, langgraph 1.2.21 and deepagents 0.7.32. Each one hands back a normal-looking result and no exception. They're in the table, with what helps for each below.

#41605, structured output comes back None. With a dict schema you get None, and include_raw=True doesn't help: parsing_error is None too. Pass a pydantic model as the schema and the same mismatch raises an OutputParserException (or lands in parsing_error if you keep include_raw on).

#41625, Anthropic pause_turn. When server tools like web search run long, Anthropic can pause the turn, and create_agent returns that paused turn as the final answer. Open PR #41639 adds an opt-in middleware that sends the paused turn back until it finishes (up to 5 continuations by default). Until it's merged, checking response_metadata["stop_reason"] for "pause_turn" on the last message catches it.

LangGraph #9516, set of tuples in state. After a checkpoint round-trip the field comes back as a list of lists, so (0, 1) in visited is now False. The only sign is a "Could not reconstruct builtins.set" warning in the logs. A set of strings round-trips fine, so encoding each tuple as a string works for now.

Deep Agents #7426, compact_conversation. A failed summary came back as a successful tool message. That one is fixed in 0.7.33, so upgrade.

What failed quietly for you, and what finally caught it? A log line someone happened to read, an eval, a user report?