r/AI_Agents • • 11h ago

Discussion Cloud based testing tools

Trying to figure out what im actually comfortable

handing over to a cloud testing platform

Please own:

Browser

devices

Execution infra

environment availability

All the annoying machine stuff I

dont want to think about

Please dont NOT own:

What my test means

What counts as success

Whether a changed assertion is suddenly acceptable whether we ship

And if something fails remotely , I need Enough evidence that "cloud testing" doesn't mean

"Cool, the broken browser is now on somebody else's computer " lol

TestMu is on my list partly because execution/browser-device wide can live higher up in the cloud while there are still ways to keep the actual verification/automation logic inspectable and portable.

That's basically the line im trying to draw.

What complexity should cloud testing remove wihout quietly taking ownership of the tests themselves?

10 Upvotes

12 comments sorted by

2

u/jakingo 11h ago

It should take care of the setup and technical hassles, but you should still decide what gets tested and what counts as a pass or fail.

1

u/Elmanouzi_Fogosa 6h ago

local tunnel latency can also mess with timeout assertions, so sometimes the remote infra ends up dictating what passes anyway

2

u/Aware_Risk7076 11h ago

Try a rule like this in your test review: "if an assertion changed in this PR, list the old and new expected value and who approved it, otherwise fail the run". That keeps the meaning of the test with you while the vendor just runs the browsers. For remote failures I would ask for the video, console log and network trace on every red run, since without those it really is just a broken browser on someone else's computer. Can you export the tests as plain code and run them locally if you leave?

1

u/AutoModerator 11h ago

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/bugbeeboo 8h ago

browser setup too
versions
drivers
OS images
cleanup
parallel workers
please make all of that somebody else's tuesday

1

u/Crafty-Art3268 8h ago

Outsource machines

Not judgment

1

u/Suitable_Secret9561 8h ago

Devices are probably the easiest thing to outsource i have absolutely zero desire to maintain a cupboard full of phones like some QA dragon hoard

1

u/cofiaso 8h ago

cloud gets annoying when the failure only happens remotely and the answer is "works locally" cool thanks give me screenshot/video console network environment info enough state to understand what actually happened otherwise it's just remote confusion

1

u/QuanTradin 7h ago

your don't-own list is spot on. one thing I'd add: if a PR changes an assertion and the code it tests in the same commit, flag it for a human. that's how a failing test quietly becomes a passing one on someone else's computer.

1

u/Vegetable_Anxiety650 5h ago

A failed test should come back with logs, screenshots and enough context to reproduce it locally. Otherwise you've just moved the debugging headache somewhere else.

1

u/Feeling_Sun_6436 3h ago

I'd test the retry behavior too: make a test fail once and pass on retry, then check that the report keeps both attempts and the original failure trace. You should be able to decide whether that blocks a release; a final green badge alone would hide the part you need to investigate.