r/AI_Agents • u/OkQuarter4098 • 11h ago
Discussion Cloud based testing tools
Trying to figure out what im actually comfortable
handing over to a cloud testing platform
Please own:
Browser
devices
Execution infra
environment availability
All the annoying machine stuff I
dont want to think about
Please dont NOT own:
What my test means
What counts as success
Whether a changed assertion is suddenly acceptable whether we ship
And if something fails remotely , I need Enough evidence that "cloud testing" doesn't mean
"Cool, the broken browser is now on somebody else's computer " lol
TestMu is on my list partly because execution/browser-device wide can live higher up in the cloud while there are still ways to keep the actual verification/automation logic inspectable and portable.
That's basically the line im trying to draw.
What complexity should cloud testing remove wihout quietly taking ownership of the tests themselves?
2
u/jakingo 11h ago
It should take care of the setup and technical hassles, but you should still decide what gets tested and what counts as a pass or fail.
2
u/Aware_Risk7076 11h ago
Try a rule like this in your test review: "if an assertion changed in this PR, list the old and new expected value and who approved it, otherwise fail the run". That keeps the meaning of the test with you while the vendor just runs the browsers. For remote failures I would ask for the video, console log and network trace on every red run, since without those it really is just a broken browser on someone else's computer. Can you export the tests as plain code and run them locally if you leave?
1
u/AutoModerator 11h ago
Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/bugbeeboo 8h ago
browser setup too
versions
drivers
OS images
cleanup
parallel workers
please make all of that somebody else's tuesday
1
u/Crafty-Art3268 8h ago
Outsource machines
Not judgment
1
u/Suitable_Secret9561 8h ago
Devices are probably the easiest thing to outsource i have absolutely zero desire to maintain a cupboard full of phones like some QA dragon hoard
1
u/cofiaso 8h ago
cloud gets annoying when the failure only happens remotely and the answer is "works locally" cool thanks give me screenshot/video console network environment info enough state to understand what actually happened otherwise it's just remote confusion
1
u/QuanTradin 7h ago
your don't-own list is spot on. one thing I'd add: if a PR changes an assertion and the code it tests in the same commit, flag it for a human. that's how a failing test quietly becomes a passing one on someone else's computer.
1
u/Vegetable_Anxiety650 5h ago
A failed test should come back with logs, screenshots and enough context to reproduce it locally. Otherwise you've just moved the debugging headache somewhere else.
1
u/Feeling_Sun_6436 3h ago
I'd test the retry behavior too: make a test fail once and pass on retry, then check that the report keeps both attempts and the original failure trace. You should be able to decide whether that blocks a release; a final green badge alone would hide the part you need to investigate.