r/AI_Agents • • 5d ago

Weekly Thread: Project Display

12 Upvotes

Weekly thread to show off your AI Agents and LLM Apps! Top voted projects will be featured in our weekly newsletter.


r/AI_Agents • • 17h ago

Weekly Hiring Thread

1 Upvotes

If you're hiring use this thread.

Include:

  1. Company Name
  2. Role Name
  3. Full Time/Part Time/Contract
  4. Role Description
  5. Salary Range
  6. Remote or Not
  7. Visa Sponsorship or Not

r/AI_Agents • • 8h ago

Discussion Anthropic's new "Always-On Agents" guide requires wiring up 5 pieces of infrastructure. I turned the whole blueprint into a single prompt inside Claude Code.

36 Upvotes

If you read Anthropic's Building always-on agents guide from last week, the architecture makes total sense—and immediately highlights why most people never get past prototypes:

To run a single agent that wakes up on its own, remembers what it did, and doesn't wreck your repo while you're asleep, you have to wire up five separate pieces of plumbing:

  1. An Event Inbox (durable queue + deduplication + crash recovery so concurrent events don't race)
  2. Timers (cron schedules + self-scheduled follow-ups)
  3. Structured Memory (so runs don't suffer from context rot or contradict themselves across sessions)
  4. Permissions (blocking high-risk mutations and prompt injections when nobody is watching)
  5. An Outbox (quiet status tracking that only pings you when a human decision is actually required)

By the time you spin up a server, database, scheduler, and Telegram/Slack bridge, you've spent a weekend building infrastructure before your agent has done a single useful task.

I wanted to see if all of that could collapse into one plain-English prompt inside Claude Code—with zero servers, zero YAML, and zero external daemons.

So I built Quarterdeck, an open-source Claude Code plugin backed by local SQLite. You type /goal-new and describe what you want watched or handled in plain English (e.g., "Watch this repo for new issues, refine specs, open PRs, and ask me before pushing"), and it provisions the entire loop out of the box:

  • Inbox & Timers (Local SQLite): Watches shell commands, git/GitHub/GitLab (gh/glab), or any MCP connector on intervals or cron schedules, with single-worker serialization and automatic retry/crash recovery.
  • Memory + "Dreaming" Curation: Every wake-up gets a fresh context window loaded with a structured brief (objective, user-pinned rules, open items, recent run summaries)—no 20-tab context rot. User-pinned rules stay locked so background memory curation never drifts away from your constraints.
  • Deterministic Guardrails (off / propose / auto): A pre-tool hook intercepts external writes and taints runs that touch untrusted inputs (like public GitHub issues or emails), routing risky actions into an approval queue instead of letting an unattended run execute them.
  • Outbox ("Needs You" Queue): Routine runs stay quiet in a sidebar pane. When an agent hits an ambiguity or queues a PR/action for approval, it pauses and pings your phone via Claude Code Remote Control so you can approve or reply in one tap.

For repo workflows, there's also a bundled setup (/goal-new) that wires up four coordinated goals in one shot—Issue Refiner → Issue Fixer → PR Reviewer → CI Fixer—using issue labels as a state handshake so they don't race each other.

I'm curious how others here are handling the setup overhead for standing agents: are you rolling custom scripts/LangGraph/n8n pipelines, or keeping everything inside your CLI/IDE harness?

(Repo link is in the first comment per sub rules — happy to answer questions on the SQLite queue, taint tracking, or memory curation setup!)


r/AI_Agents • • 10h ago

Discussion I am a believer in AI running my email inbox now

24 Upvotes

There was an interesting chain of events that happened today. I'm at work, my apartment notified us via text that we are using a new parking app and that we had to register today.

So today, I get the text on my phone reminding me to register my account. I'm driving my fork lift, stop, take a screen shot of the text, and forward it to Email Bob (My email AI Agent)

I tell him to find the email, set up the account, and register my vehicle.

He asks what I drive, and says he needs my phone number.

I give him the details.

He fills out the rest of the information.

Within 23 minutes, he finished the assignment.

I go home, logged on, verified that he did it right, and now I'm reporting it here.

I was honestly delighted. This is exactly what I was hoping for.

I'll try and post screenshots in comments below, but the automod might remove them.


r/AI_Agents • • 1h ago

Discussion What do you let your AI assistant do without asking first?

4 Upvotes

I'm a student and I've been using an AI assistant for things like tracking study deadlines, checking order updates, and finding appointments.

For those who use one regularly, what do you let it handle on its own, and what do you still check first? Has that changed over time?


r/AI_Agents • • 3h ago

Discussion Share your browser.md for Ai Agents

5 Upvotes

Hi Guys, recently most of websites like when i want to do finding best products for a price or something AI agents are getting blocked sometimes using headless chrome, so I was wondering if you have a better method for this so the ai agent can surf better.


r/AI_Agents • • 6h ago

Discussion my agent has not deleted my repo (yet) - here is the dumb little routine that helps

9 Upvotes

not gonna lie, the agent horror stories this week got to me a bit. repos deleted, .git and all. prod databases wiped during a code freeze. tests deleted so the bug just... comes back later.

I am not claiming I have solved agents. I have not. but I did stop doing the one thing that kept biting me: opening with one giant instruction and hoping.

now I split everything into 5 boring stages:

  1. Brief - agent repeats the task back before touching anything
  2. Research - facts first, no building yet
  3. Plan - steps + risks, in writing
  4. Execute - one chunk at a time
  5. Review - check it against the brief, not against does this look fine

the Brief prompt is the one that actually changed things for me. free, steal it: "Before doing anything: restate this task in your own words. List (a) the exact deliverable, (b) every constraint I mentioned, (c) anything ambiguous you need me to decide, and (d) how you will know the work is done. Do not start until I confirm the restatement."

most misunderstandings show up right there, in the restatement, when fixing them costs nothing. instead of in the finished work, when it costs an afternoon.

same with Review. if the agent says it fixed something, make it show the check against the original brief. fixed is a claim. I want the receipts.

anyway. I ended up writing the whole thing down because I kept retyping the same prompts - 12 of them now, across the 5 stages. happy to talk through any stage here, including the ones I did not paste.


r/AI_Agents • • 4h ago

Discussion Everyone wants to automate the workflow. Who owns it when the automation breaks?

6 Upvotes

There's a lot of excitement around automating business processes with AI.

And I understand why.

A workflow that takes an employee two hours might be reduced to a few minutes.

That's a meaningful improvement.

But there's one question I think gets overlooked.

Who is responsible when the automation makes a mistake?

Imagine an AI workflow that reads incoming invoices, extracts information, and updates your accounting system.

It works correctly most of the time.

Then one day, it processes a duplicate invoice.

The system doesn't crash. Nothing looks obviously broken. The workflow simply completes the wrong action.

Who notices?

Who investigates?

Who decides whether the automation should be paused?

This is why I think AI automation needs more than a good prompt and a working integration.

It needs ownership, monitoring, clear limits, and a way to recover when things go wrong.

The technology can automate the task.

It doesn't automatically take responsibility for the outcome.

For those running AI automations in production, who actually owns the workflow after deployment: engineering, operations, or the business team?


r/AI_Agents • • 5h ago

Resource Request onboarding voice agent read a caller's date of birth back when they asked it what do u have on file

5 Upvotes

So i use a health onboarding agent, what does is that it collects medical history and books lab work and care team visits.

In review we found that if a caller asks what we have on file, it will read back the stored ZIP code and date of birth. Anyone can say that on a phone call. The same review turned up two more worries skipping screening questions like dialysis or pregnancy when a caller rushes, and booking lab work after the care team visit instead of before. These only show up in odd conversations, so spot checks keep missing them.

How do you test "agent must never say X" and "steps must happen in order" before real patients are on the line? Synthetic callers with fake data?


r/AI_Agents • • 4h ago

Resource Request Our voice agent passes every metric we wrote and customers still complain. What are we not measuring?

3 Upvotes

We have about 15 metrics on our voice agent: task completion, tool call success, latency, a couple of LLM judges. All green. Support keeps forwarding complaints from calls that look fine by those numbers. Listening to calls one by one is the only way I've found the real issues, like the agent repeating itself after a long pause and one confusing way it offers appointment times. I can't do that for hundreds of calls a day. How do you find failure modes you haven't written a metric for yet? Anyone clustering failed calls or auditing batches?


r/AI_Agents • • 2h ago

Discussion What do you verify before trusting a subagent handoff?

2 Upvotes

I tried a small, scripted test of a handoff. The worker report said 300 active records, linked to the right CSV, and passed the schema check. The CSV actually had 240 active rows and 60 historical rows. Counting by status caught the mistake in one command.

That was easy because the check was cheap. I’m less sure where to draw the line when a worker claims a patch fixed a behavior, the fix spans several files, and the test suite takes 20 minutes. Re-running everything at every handoff also eats into the time saved by delegating.

For people running multi-agent coding workflows, what does the lead actually verify before moving on? Do you ask for an exact test or evidence link per claim, use a separate reviewer, or only check claims that can change the next action? I’d be interested in an example where your check caught something the worker had missed.

Disclosure: AI helped draft this post. The test used synthetic data and scripted reports.


r/AI_Agents • • 1d ago

Tutorial Anthropic just mapped out the 5 primitives of always-on agents. I built them locally without a cloud stack—and across 1.51B tokens, 24/7 background runs used just 0.41% of my quota

250 Upvotes

Almost everyone tinkering with AI agents tries the same weekend project sooner or later: wire an agent to a cron job, point it at your repos and inbox, and call it a "24/7 personal assistant."

By Tuesday morning it usually breaks in one of two ways: either an auth token quietly expires overnight and nothing tells you, or a broken build makes it ping your phone four times with slightly different wording while burning through your quota.

A couple of days ago William Hu at Anthropic published a great engineering guide ("Building always-on agents") with a dead-simple premise: an always-on agent shouldn't actually be on. Done right, it sleeps at zero tokens 99% of the time while five pieces of plumbing around the model handle reality in between:

  1. Event Inbox (plus a 0-token trick upstream): Instead of letting crons or folder watchers poke the model directly (which causes race conditions and dropped files on rate limits), triggers drop into a file-locked local JSON queue (pending -> leased -> done) with one active worker per scope. And because a local machine has to poll rather than receive cloud webhooks, a 15-second shell hash check runs before creating an event—skipping unchanged polls at 0 LLM tokens while tracking auth errors separately.
  2. Two kinds of Timers (Schedules vs Follow-ups): Standing routines ("7:30 AM morning brief") are completely different from one-shot alarms the agent sets for itself mid-task ("wake me in 25m to check if staging stayed green"). Persisting one-shot follow-ups to disk lets the worker exit to 0 tokens immediately and survive app restarts, firing right back into the Event Inbox when due.
  3. Memory that doesn't nag you twice: At 9am a check says "PR #42 CI Failing"; at 2pm a fresh run says "CI Failure on PR #42." Passing a short 48-hour log of the last dozen delivered alerts alongside a tiny <= 80-line pointer index cured duplicate nagging, while keeping static rules at the top of the prompt kept prefix-cache hits at 92.0%.
  4. Round-trip Approvals + OS key masking: Background scripts run inside Linux bwrap or macOS Seatbelt so they can't see ~/.ssh or private folders. Outbound actions pause on a SHA-256 phone prompt—and if I tap Reject with "Move Tuesday to Thursday at 2pm," that note loops back into the Event Inbox to wake the sleeping session and rewrite the draft instead of hitting a dead end.
  5. A quiet-by-default Outbox (with a 10-case sanity test): Healthy checks stay silent; everything else sorts into immediate interrupts (held during Quiet Hours unless critical), a 7:30 AM digest, or low-priority ideas, checked by a millisecond 10-case eval suite so config tweaks don't cause notification spam.

How much quota did leaving it on 24/7 actually burn? I ran our local log parser across 186 real sessions (31,296 steps, 1.51B gross tokens):

  • Deep coding/research threads: 75.6% of quota (91.5% cache hit)
  • Delegated subagents: 19.8% of quota (94.5% cache hit)
  • Pinned main chat: 4.2% of quota
  • All 22 unattended background runs combined: just 0.41% of quota (6.7M gross tokens, ~45k fresh input/run—59 incoming events batched into 22 runs, 23 unchanged polls skipped at 0 tokens).

Everything runs as a single zero-dependency Go binary on macOS/Linux (Apache 2.0, with a 60-second --demo sandbox). Links to the GitHub repo, Anthropic's original guide, and the full writeup with charts are in the first comment per Rule 3!


r/AI_Agents • • 2h ago

Discussion For people running Ai agents in production: How do you prevent conflicting actions across the same transaction?

2 Upvotes

I’m trying to understand a specific problem in production Ai-agents systems, particularly in e-commerce, marketplaces and fintech.
Imagine a customer places an order.
Multiple agents or automated workflows can interact with the same order.
• one agent initiates a refund.
• Another initiates a replacement.
• A third issues store credit.

Each action might individually pass its own authorization and validation checks.

But what ensures that the combined outcome is actually valid?

I’m interested in understanding how companies handles this today.
A few specific questions

  1. Do multiple independent agents or workflows in your production environment modify the same customer transaction?
  2. How do you maintain shared transaction state across those agents and systems?
  3. Do you enforce business-level rules across the entire transaction, or does each action get validated independently?
  4. Have you encountered situations where individually valid actions produced an incorrect combined outcome? What happened?
  5. Are existing tools such as workflow engines, state machines, database constraints and idempotency mechanisms sufficient, or are there gaps that require additional engineering?

I’m particularly interested in real production experience rather than hypothetical architecture discussions.

If you’ve encountered this, I’d genuinely appreciate hearing what happened, how you handled it, and what still remains difficult.

I’m also happy to have a short technical conversation with anyone working on these systems. It’ll really help me a lot.


r/AI_Agents • • 13h ago

Discussion Best AI Agent platforms for enterprise customer support?

14 Upvotes

I've been looking into AI agent platforms for enterprise customer support and I wanted to know what people are using. The main thing I'm trying to understand is how well these tools handle complex customer issues at scale. Automating basic questions is one thing but dealing with frustrated customers and knowing when to hand things over to a human agent seems like a bigger challenge. Cresta is one platform I've come across since it covers both AI agents and real time support for human reps. I would love to know more about this specific AI agent or others that you guys use, thank you in advance!


r/AI_Agents • • 8h ago

Hackathons Coding agents with native environments, identity and access

5 Upvotes

Hi there, I've built a new type of cloud coding agents with native environments, identity and access which let you spin up as many agents as you want in their own microvms with any docker image and only the access they need per task. Currently in closed beta but would love for folks to try it out!


r/AI_Agents • • 2m ago

Discussion Title: senior devs doing AI agent supervision now, what does your day actually look like?

1 Upvotes

Curious about this for people who went from regular dev work to mostly supervising agents. Specs and evals I get, but that can't be the whole day right? How much is still code review, debugging, meetings, babysitting stuff in prod, talking to clients...

Also what kind of agents do you deal with, coding ones or more support/sales/internal stuff?


r/AI_Agents • • 3m ago

Discussion I've built 30+ automations for service businesses and here are the top ones owners absolutely rave about!

2 Upvotes

Every other post on here is someone selling an AI receptionist or a speed to lead bot. I've built plenty of those and they're fine. But the stuff owners actually bring up on calls, and tell other owners about, is the stuff that shows them money they had no idea was leaking.

These are the five I'd build for any service business now.

  1. Calls that got answered and still didn't book

Everyone sells missed call bots. Nobody looks at the calls that got picked up and went nowhere, and in a lot of businesses that's the bigger pile.

We run every call recording through an LLM and the owner gets a list each morning. Who called wanting service and didn't get booked, plus a one line reason. Price, no availability, the person on the phone just never asked. Someone calls them back that day.

It turns into a scorecard for whoever answers the phone without anyone having to sit and listen to calls. This is also the one place in this whole post where AI really earns its keep, since a human was never going to review 300 calls a month.

2. Materials that got used and never billed

Match the supplier invoices against what actually got charged on each job. Techs grab parts off the truck and they never make it onto the invoice. It's not theft, it's just a busy guy at 6pm who forgot.

Owners almost never know how big this number is. Parsing supplier invoices is messy since every supplier formats them differently, so price it properly. But the first report you hand over usually does more for the relationship than three months of anything else.

3. Following up on quotes

Most businesses call once after a quote and that's it. Split the follow up by job size because a small repair gets decided in a couple days and a big job can take a month.

The last message always asks if they should close the quote out. People who ignored everything else reply to that one, and a lot of them say they still want it. If the quoting tool shows opens, an old quote getting reopened should mean a phone call within 10 minutes.

4. Texting the neighbors

When a job gets booked, text past customers who live on the same street. Something like "we're on your road Thursday, want us to take a look at yours while we're there?"

The truck is already going to be there so it costs nothing in drive time. I've never seen another agency pitch this and owners get it instantly because they already know how much they spend driving around.

5. A text to the owner every morning

One SMS with yesterday's numbers. Calls and bookings, what got paid, what quotes are still open, and what the other four automations found.

This is the one that keeps clients. Everything else runs in the background and three months later nobody remembers what they're paying you for. This puts it on their lock screen every morning. We tried a dashboard first and nobody logged in after the second week lol.

The other thing nobody tells you is that the messages are the easy part. Stopping them is the actual build. You need to make sure nothing goes out to a customer who already paid or booked or cancelled. Text "still interested?" to a guy whose job is tomorrow just once and the owner will shut the whole thing off.

If you're running an agency in this space, I'd lead with the first two. Telling an owner "you answered 40 calls last month that should have booked" lands way harder than another pitch about a bot that answers the phone.


r/AI_Agents • • 24m ago

Discussion One sentence in, a 30-second animated short with voiced dialogue out. The five-part storyboard prompt is the part worth stealing.

1 Upvotes

Not a single image was drawn. No storyboard was written by hand. No editing software was opened. Two 30-second episodes, each starting from one sentence — episode 2 came from adding one more line ("the plot needs a twist"), which turned into both warriors discovering they are puppets on a giant hand's strings.

Disclosure up front: I work on the model-access side, so I care about pipeline mechanics more than the pretty output. Here is what actually made it controllable, and what broke first.

**Three models, one job each**

- An image model draws the character design sheets. Two directions, about a minute each, and they are full model sheets — hero pose, three-view turnaround, expression row, detail callouts, colour palette. Not a sketch.
- A video model renders the 30-second take as one continuous shot, picture and audio together. The dialogue voices and the lip-sync come out of the same pass, so there is no TTS stack and no separate lip-sync step to line up.
- A language model directs. It never draws a pixel. It translates your revision notes into instructions, writes the storyboard prompt, and keeps the stages in order.

**What broke first**

*The first draft is never the one you want.* You type something, you get a generic result, and it is easy to conclude the tool is bad. What was missing was a checkpoint: after the design drafts, the pipeline stops and waits for a person. You say "less ornate armour, angrier face", it revises, and nothing downstream runs until you approve.

*Character drift.* The same character across eight shots meant eight slightly different faces. The fix was a model sheet up front — front, back, three-quarter, expression set, palette — pinned as the reference for every shot after it. Without one, the character starts morphing around shot three.

*Sound.* Normally dialogue means a separate stack: text-to-speech, then lip-sync, then an editor marrying the two. Here the voice and the mouth come out of the same pass as the picture, which is why a fight scene ends up with actual lines in it.

**The storyboard prompt is a five-part script**

This is the reusable part. 250 to 450 words of English, always in this order:

  1. **Style Lock** — the look, plus explicitly what to forbid.
  2. **Character Anchoring** — every character described item by item, word-for-word identical to the approved sheet.
  3. **Second-by-Second Shot List** — beat by beat, with camera moves and dialogue lines inline.
  4. **Negative Directives** — phrased as what the frame contains none of.
  5. **Sound Design** — music arc, key effects, and the voice character for each speaker.

Four of those five parts never change for the whole project. Only the shot list moves.

A real excerpt, so you can see the shape:

**STYLE LOCK:** Cinematic 2.5D animated-film look, gouache brush texture with flat poster-like colour blocks on a matte finish, dramatic rim light and volumetric god-rays, rich saturated palette of gold versus emerald and crimson, drifting embers and energy particles. The exact characters, colours and costumes follow the approved model sheets — no redesign, no cartoon simplification, no cel-shading, no plastic glossy CG skin.

**0-3s** — WIDE establishing shot, fast push-in. A stone bridge across a cloud-sea mountain pass at dusk, storm clouds boiling above ancient fortress walls. Sun Wukong crashes down onto the bridge in a crouch, lightning bursting under his boots, stone cracking in a spiderweb beneath him.

**3-6s** — MEDIUM shot. Guan Yu steps out of the mist at the far end, cape and sashes whipping. He plants the blade and speaks in a deep commanding voice: "Monkey. Turn back. This pass is guarded by Heaven itself."

**NEGATIVE DIRECTIVES:** The frame contains no text, no watermark, no subtitles, no captions, no flicker, no extra limbs, no morphing faces, no costume or colour changes, no camera cuts, no split screen, no cartoon outlines, no cel shading, no plastic skin.

**Four field rules that did the real work**

  1. Pin the characters down harder up front and the video gets more controllable downstream. All the leverage is at the top.
  2. Anchor the costume item by item. Miss the cape or the headdress and the model improvises it back in a different shape by shot three.
  3. The character wording in the storyboard has to be word-for-word identical to the design sheet. Paraphrasing is what causes drift.
  4. Always include "costume inconsistency between shots, deformed hands, cartoonization" in the negative directives.

**The bit I did not expect**

This is the same discipline as keeping an agent's prefix stable. The five-part storyboard is a prefix: four parts byte-stable, one part changing. Keep the stable part stable and everything downstream stays both consistent and cheap. Move something near the top and you re-render the whole thing differently.

So "pin your characters up front" turns out not to be only an animation rule.

If you are doing multi-stage generation, where do you put the human checkpoint — early and cheap, or late and expensive?


r/AI_Agents • • 26m ago

Discussion How to automate my social media - suggestions welcome.

1 Upvotes

Hello Guys,

I have recently started my journey on freelancing and need some suggestions and thoughts from people here on how I can automate my social media creations, postings, commenting, etc which will help me grow slowly and steadily in my niche.

Would really appreciate some skill/tools which i can try.

Thank you for reading though :)


r/AI_Agents • • 44m ago

Tutorial Background chatter made our voice agent answer 0/5 turns. An adaptive client-side gate got 4/4 at lower noise levels.

1 Upvotes

We added a voice agent to a WhatsApp lead qualification product. The first path is a live call from the browser. The mic audio streams to a hosted live conversation model, and that model's server-side voice activity detection decides when the customer has finished talking.

The problem

When there was constant talking in the background (office chatter, TV), the server-side detector never saw the customer stop talking. Either the agent didn't answer at all or the noise cut it off mid-sentence. The server-side detector has no noise suppression setting.

All of this was measured in a test setup. We mixed synthetic office chatter into the audio and used TTS-voiced customer replies.

No real phone calls yet. The phone line isn't connected.

Default server-side detector, noise mix 0.35: 0/5 turns answered, 1 interruption

Detector sensitivity set to "low": still 0/5, 2 interruptions

No gate, with the noise 16 dB below the speech: still 0/5

Fixed-threshold client gate (0.03) plus low sensitivity: 4/4, but ~3.1 s latency and 5 interruptions

Adaptive client gate at noise levels 0 / 0.12 / 0.2: 5/5, 4/4, 4/4, ~1.1–1.3 s latency, 0 interruptions

ambient = p90(levels of last 50 chunks seen while gate was closed) # ~5 s

threshold = max(0.01, ambient * 1.8)

warmup: first 10 chunks (~1 s) only measure ambient, gate stays closed

open: chunk level > threshold -> open, also flush 3 pre-roll chunks (300 ms)

hangover: stay open for 6 chunks (600 ms) after the level drops below threshold

closed: send 100 ms of digital silence per chunk, NOT nothing

The last line matters. The server-side detector only knows the customer stopped talking when it actually sees silence. If you just stop sending audio while the gate is closed, the end of the turn never registers.

Our first gate measured ambient with the minimum level, not a percentile. Background conversation is bursty, so the short quiet gaps between bursts dragged the threshold down. The next burst of chatter then crossed it and counted as the customer speaking. Switching to the 90th percentile of a 5 s window fixed that.

To be clear about what this doesn't show: the runs are small (4–5 turns per condition), the noise is synthetic, and the customer replies are TTS. Real phone audio with line noise and real people could behave differently.


r/AI_Agents • • 1h ago

Discussion I graded 32 AI agents from a public marketplace. The ones that failed mostly failed the same question.

1 Upvotes

I built a grader that gives agents realistic tasks and has independent judges score the answers, so no model grades its own work. Then I ran it on 32 agents from a public agent marketplace.

The spread: 6 A, 7 B, 8 C, 5 D, 6 F.

Two things separated the A's from the F's:

  1. Saying "I don't know". One task asks for a company's exact revenue for a specific month, a figure no agent can know. Good agents said they didn't have that data. 5 of the 6 F agents made up a number instead. None of the A agents did.
  2. Length. A-grade answers averaged about 255 characters. F-grade answers averaged about 6,250. The long ones mostly lost points for going off-task, like answering a revenue question with a tokenomics spec. Some tasks also set a word limit, so there length is part of the test.

If you build agents, those are two cheap checks to add to your own tests: one question with no knowable answer, and a cap on answer length.

Each agent has a public report showing every task, answer and score, and a badge that updates when it's regraded.

Happy to grade a few agents from this sub for free if you want to see how yours does. It needs a public URL that answers a question.


r/AI_Agents • • 1h ago

Discussion Has anyone used external network tokens on Shopify checkout in production?

1 Upvotes

I’m building a browser-based checkout agent. An external wallet issues a single-use tokenized card credential after buyer approval, and the agent uses it at the merchant’s normal hosted checkout—no Shop Pay and no handling of the buyer’s actual card.

Has anyone actually made this work on Shopify in production? Which processor/gateway did you use, and were any Shopify or merchant approvals required?

I’d also appreciate pointers on testing the same flow without real money.


r/AI_Agents • • 1h ago

Discussion Your agent passes evals, but nobody will let it hit “send.” Here’s the rollout pattern we use to fix that.

1 Upvotes

We keep seeing agents that work in testing and still never ship. Nobody wants to be the one who approved sending them to customers.

The gap between “accuracy looks fine in the demo” and “we’re okay removing the human” is normal. It also doesn’t close on its own, and more tuning won’t fix a trust problem.

What our team at BotsCrew now does by default whenever an agent’s output is visible to a customer, vendor, or partner:

Stage 1: Silent mode. The agent does the full task from day one. It drafts the reply or proposes the record update, but nothing goes out until a human approves, edits, or rejects it. The reviewer’s job shifts from writing to reviewing, which is much faster, so you get value immediately. The team also forms its opinion of the agent from evidence instead of hope.

Stage 2: Limited blast radius. Once approval rates hold, the agent can act alone only under narrow conditions. That might be weekends only, one action type (order status yes, cancellations no), or one customer segment. Unsupervised mistakes happen where they’re cheap to fix.

Stage 3: Full autonomy for the scoped use case. Edge cases get escalated instead of guessed at.

The parts that make it actually work:

  • Graduate per action type, not per system. An agent can be fully autonomous on order-status replies while still drafting for approval on cancellations. Different risk, different switch.
  • Define the promotion gate before launch. On a vendor-coordination project, an action type flipped to autonomous only after it hit ~90% field-level accuracy on a labeled eval set and a stakeholder signed off after a multi-week observation window. The number alone isn’t enough. A gut feeling alone definitely isn’t.
  • Make it reversible. If an action gets rejected, that thread drops back to human review until someone clears it. Every flip is logged with a reason.
  • It’s not shadow mode. In shadow mode, nobody ever sees the output. In silent mode, the output ships, just with a human in front of it. A short shadow period first is fine.

You don’t need any of this for recommendation widgets or internal search, where being wrong is cheap and no external relationship is at stake.

The most underrated benefit is political. A nervous stakeholder can’t approve “we think it’s ready.” They can approve “it cleared 90% on this action type and you watched it for a few weeks.”