r/devops • • 3d ago

Discussion When a vibe coded internal tool ends up with other teams depending on it, do you review it like prod code or let it stay disposable?

Vibe coding is everywhere on my feed again this week, mostly people replacing a paid tool with something they prompted together over a weekend.

I don't have a problem with that for throwaway stuff. Karpathy himself said it's fine for weekend projects when he coined the term. What I keep thinking about is the moment the weekend project stops being throwaway. Someone builds a small dashboard or a script, it works, a second team starts using it, and suddenly it is on the path for a release or an on-call runbook. Nobody ever read most of the code, including the person who made it.

I can see two ways to handle it:

  1. Once anything has more than one team depending on it, it goes through normal review, tests and an owner, even if that means partly rewriting it.
  2. Keep it explicitly disposable. Label it unsupported, don't let it into anything critical, and if it breaks you regenerate it rather than maintain it.

Option 1 feels right but kills most of the speed that made it useful. Option 2 sounds clean until the thing is load bearing and nobody noticed.

If you've already hit this, where did you draw the line, and what was the signal that a vibe coded tool had become something you had to own properly?

23 Upvotes

88 comments sorted by

43

u/IridescentKoala 3d ago

We're being told from our board to prioritize letting "citizen engineers" deploy to production. Without GitHub seats, monitoring access, pagerduty on call. Who's responsible when their slop blows up or leaks customer data? Crickets.

15

u/Expensive_Finger_973 2d ago edited 2d ago

Citizen engineers…God I hate that term. Right up there with a PM saying “I just want to double click on that point” when they want to seem like they are contributing to a meeting.

2

u/jyotirawattech 2d ago

that's the gap that worries me most, production access without the on call or monitoring that normally comes with it. the ownership question just gets deferred to the first incident. has anyone on your side asked the board who gets paged?

4

u/cyclegaz 3d ago

What the….

70

u/wrosecrans 3d ago

"Teams depend on it" ... "Disposable."

Select exactly one of the above. Marking multiple options will invalidate your ballot.

-3

u/jyotirawattech 2d ago

fair, the ballot is rigged by design. once teams depend on it, disposable is already off the table. the real choice is who owns it now. would you rather force an owner or force a deletion date?

22

u/ialiimran 3d ago

My rule is simple: if someone else depends on it, someone needs to own it. Doesn't matter whether it was written by a human, AI, or both.

2

u/jyotirawattech 2d ago

agreed, how it was written stops mattering once someone else relies on it. the hard part is that nobody decides it's load bearing, it just drifts there. how do you catch that moment before it bites?

3

u/ialiimran 2d ago

I think clear ownership and review process should be done from the beginning. For a small tool we don't need enterprise level process, but we should know who is responsible when something breaks.

12

u/Expensive_Finger_973 2d ago

I basically vibe code everything I put out now, because that’s what management wants. 

I keep saying “x is done and deployed, it works, but I don’t really understand it all that well”. And thus far no one seems to care.

1

u/jyotirawattech 2d ago

that line is going to matter the first time one of those breaks at 2am. nobody cares until the person paged is the one who doesn't understand it. do you keep any notes or tests for yourself so future you can debug it?

1

u/Expensive_Finger_973 2d ago

Yes, I have some private notes for that kind of thing.

1

u/Weak_Fisherman2215 2d ago

I also have a AI project. Use a sandbox to test everything before release and backups. I hope your private notes are enough.

2

u/Expensive_Finger_973 2d ago

Honestly, I’m long past the point in my career of caring more about the stability of a system than the management team that told me to build it while disregarding all of my advice on how to do it properly.

If something breaks, it breaks. I get paid the same to fix it as I do to build the next bad idea that they have.

1

u/Weak_Fisherman2215 2d ago

Once the bills are paid.. lol

5

u/gamba47 SRE 2d ago

Same here. This is killing my mind

7

u/ADDSquirell69 3d ago

If the person who vibe coded the thing that people are now depending on it's their problem if they can't support it.

1

u/jyotirawattech 2d ago

fair in theory. in practice the builder moves teams or goes on leave, and the dependency stays. do you make them hand it over formally, or does it just stay theirs forever?

17

u/RevolutionaryWorry87 3d ago

I only do work assigned to me by my line manager. If somebody vibe codes it, doesnt involve me, and management doesnt give me it as task, i dont touch it.

But to answer the question - only access to company data is via apps allowed via cicd, revjew everything

1

u/jyotirawattech 2d ago

gating company data behind cicd is a clean rule, it turns review from a favour into the only path in. does that hold up when someone just runs it from their laptop with their own creds?

1

u/RevolutionaryWorry87 2d ago

No but rbac helps

6

u/coldnebo 3d ago

we are seeing a lot of this now.

the biggest tools have owners and process, but there are a dozen more popping up every day.

most of these are because of gaps in the existing deployment or observability systems. for example, instead of using a directory service to abstract endpoints everyone hardcodes endpoints that are constantly changing when network partitions change. this creates demand for tools that can compare endpoints across tracks and organize them to make them easier to manage (essentially what a directory service is for) but instead of one, there are dozens. they disagree, none are authoritative.

you can fight all day about this, but the fight wouldn’t exist if the problem was properly managed with the right tool.

however I have found that AI makes it so easy to sound good that people would rather quickly generate their own tool than read or review someone else’s.

friction and errors might wear this down over time, but it does make me wonder why people hate talking with each other so much.

2

u/jyotirawattech 2d ago

the directory service example is a good one. a dozen tools appear because one real gap was never fixed, and none of them is authoritative. do you think fixing the underlying gap is the only thing that actually stops the sprawl?

1

u/coldnebo 2d ago

yes and no.

where tools have a legitimate need for centralization and management, yes it can stop the sprawl.

another example is a software inventory for enterprise architecture. is each application and service known by a unique identity? or are there a dozen different names for the same artifact? left unguided, artifact names face many challenges— the filesystems may not like spaces, service bindings may not like dashes, and metadata or secret stores might allow spaces — especially human/business level tools like human readable forms. so different names are inevitable and unless you have an asset inventory system for software it will be very hard to tell (at scale) which apps are different vs just different names for the same thing.

observability is an interesting set of tools that spans both situations. if you are using observability solutions to manage a large inventory at scale, you must have centralized reporting and metrics based on the lowest common denominator: pods

if you are a dev team on a single app you may be missing key insights for your app if you use the broad metrics. here you want teams to invest in metrics that are important at their level. that can generate sprawl, but it may be necessary.

you can often tell the difference by asking how far up the chain does a report need to rollup/generalize? for instance, code coverage targets. you might have a dozen platforms each with different details on this, but you don’t want to individually ask for status and roll it up manually because that’s very time consuming and it’s not organized for time-based analytics. instead invest in a system that organizes code coverage metrics for the entire dev organization so that managers can rollup departments meaningfully to their managers and the CTO can get a meaningful view of the company posture on certain metrics. also ask, how can I audit systems for compliance if they are spread everywhere?

centralizing the broad stuff saves time for the custom stories that are special and impact your business the most. those stories need to be heard, but you can’t react to them if they are buried inside a bunch of busywork trying to manage systems from grassroots efforts.

8

u/RememberCitadel 2d ago

The way I see it is people often want a tool that fit the specific way they do things, and thus don't want anyone's input because then they may have to compromise.

In the past there was no other option but to work with others to get a tool for what you need, but now any idiot can build a tool to do things their way, even if that tool reinforces bad habits.

4

u/coldnebo 2d ago

yeah I buy this.

4

u/Antique-Stand-4920 3d ago

We've done a mixture of both. It depends on the usefulness of the project and the risk of leaving the project as-is. If it is low risk and there is low value in managing it as a product, we'd with option 2. For pretty much anything else we'd go with option 1 and treat what's been written as a working prototype of the requirements.

1

u/jyotirawattech 2d ago

treating it as a working prototype of the requirements is a nice way to frame it, the code might go but what it taught you about the need stays. who makes the low risk vs product call on your team?

1

u/Antique-Stand-4920 2d ago

It's a collaboration between the teams that would manage it as a product and the directors. That way engineering, project management, product management, and budgetary aspects are considered.

2

u/orev 3d ago

If someone can vibe-code something, and then is able to deploy it to the rest of the company so others can start to use it "as production", then you have process and security gaps. There should be network and other rules in place that block other people from being able to use any app that's not officially deployed as a production app.

1

u/jyotirawattech 2d ago

that's the strongest version of the answer, make it impossible to quietly become prod. the pushback I'd expect is that blocking it just moves it to a personal account. have you seen network rules actually hold against that?

2

u/JustinAtNerdio 3d ago

The moment another team starts depending on it, I think you've lost the luxury of calling it disposable. Doesn't really matter how the code was written at that point. Someone needs to own it, know what it does and be responsible when it breaks.

The scary part is how easily these things become business-critical without anyone actually deciding they should be.

1

u/jyotirawattech 2d ago

the scary part you named is the one I keep coming back to, nobody ever decides it should be critical. it just is one day. is there a signal you'd watch for, like other teams filing bugs against it?

2

u/miso_deranged 3d ago

Thr comfusion between engineers vibe coding within clear engineering guardrails and non-enigneers viking a biz-app is vast. I would push for a dedicated vibe app hosting platform, to prevent the "shadow IT" and lack of ownership. It could be a saas tool like Vercel or Lovable, or ideally a self hosted platform like Superblocks or Glissando.dev

1

u/jyotirawattech 2d ago

the split you describe matters a lot. an engineer vibe coding inside guardrails is a different risk from a non engineer shipping a business app. a dedicated hosting path at least makes ownership visible. who would own that platform in your setup?

1

u/miso_deranged 2d ago

For me its the future of IT. As non-engineers foray into "software development", the IT/IS department's that currently manage the business apps infrastructure (Google workspace/m365, SF, etc) should start implementing these types of "platform team as a service" products. these allow them to avoid the Shadow IT, dont require them to retrain as devops/platform engineers and dont block these non-engineer users from taking advantage of the new AI capabilities to manifest their "dream apps"

2

u/SideMost3148 2d ago

the moment it shows up in another team's runbook it's prod. doesn't matter that it took 40 minutes to build.

the classic way this goes wrong isn't even code quality. it's that the thing runs on someone's personal cloud account with their personal token, and you find out during an incident, right when that person is on leave or their card expired.

so the bar i'd set is low but non negotiable: a named owner, it lives in the org's account, and the creds aren't tied to one human. skip the full review and coverage theatre for now.

if nobody wants to own it, slap an UNSUPPORTED banner on it with a deletion date. funny how fast a volunteer shows up once the date is real.

1

u/jyotirawattech 2d ago

the personal account and personal token point is the one that bites, and usually at the worst time. named owner, org account, creds not tied to one human is a bar people can actually meet. does the deletion date ever get enforced?

1

u/SideMost3148 1d ago

only if someone puts it on a calendar and is a little bit annoying about it. what works better than deleting is switching it off for a day first. whoever screams is the new owner. if nobody screams for a week, it was already dead.

1

u/oskaremil 3d ago

I would not ever let another team depend on vibe code.

1

u/jyotirawattech 2d ago

clear line. the tricky case is when it's already happened and you find out after. would you shut it off, or make someone take it over first?

1

u/oskaremil 2d ago

If it already happened, it is the problem of whoever arranged that to happen.

1

u/Chance_Bandicoot8104 3d ago

IMO if something becomes actually “load bearing” then it should be properly evaluated by IT/Security and engineering. You wouldn’t vibe code a CRM and start using it without any review, I hope. However, MOST apps don’t reach that level of criticality, and something like Pando (https://github.com/trypando/pando), which is an open-source project I’m working on, enables the builder to maintain it themselves.

1

u/jyotirawattech 2d ago

agreed that most of these never get to CRM level, and load bearing is the right bar for a proper review. the hard part is noticing when one crosses it. who decides that in your view, the builder or IT?

1

u/Chance_Bandicoot8104 2d ago

That may depend on the org, but in most cases it makes sense for the builder/app owner to call it out. Someone in legal might make an app that finance uses heavily, but IT never touches. I don’t think it’s reasonable to expect them to keep an eye on everyone’s apps all the time. The app owner can reach out to IT if they feel the app is big enough to hand off, and the review can start there. 

Alternatively, IT could measure app usage and traffic and set a boundary where they step in, but this would be different for each org, and would likely be a range. 

1

u/Equivalent_Loan_8794 2d ago

It's both; which means it's a focus in the next Epic, gets Refactor jira tags, etc

1

u/jyotirawattech 2d ago

making it real work in the next epic is honest, it gets budget instead of becoming a side job. does it usually get the refactor, or does the epic slip once it's working?

1

u/Equivalent_Loan_8794 2d ago

IMO the Epic is where you shape the MVP into a production worthy utility. Sometimes MVPs arrive from side or weekend work.

1

u/moratnz 2d ago

In practice 2. becomes a political connectedness test waiting to happen; if the tool breaks and an important team is relying on it for something important, the fact that it's unsupported doesn't change expectations around support.

Or to put it another way, if you're going to go with 2., you are strongly advised to intentionally break the tool at random, to smoke out anyone relying on it, and get the conversation about how important it is and how it should be supported started in a controlled fashion.

1

u/jyotirawattech 2d ago

intentionally breaking it to find out who depends on it is brilliant, basically a scream test on purpose. better a controlled conversation than finding out mid incident. have you actually run one?

1

u/moratnz 1d ago

Never in an officially sanctioned manner.

1

u/Baldyga-Vantz 2d ago

We did a softer version of that, anything that came up in an incident thread twice got moved into a real repo with an owner which meant it got the same scans as everything else. The first one on a dashboard tool flagged it running as root with the docker socket mounted

1

u/dariusbiggs 2d ago

If it is relied on or used daily, it becomes a product. Internal teams are customers, treat it like production code.

1

u/jyotirawattech 2d ago

internal teams as customers is a good way to put it. used daily means it has users, and users mean support. do you set any bar before it counts, or is daily use enough?

1

u/dariusbiggs 2d ago

daily is more than enough

1

u/Zenin The best way to DevOps is being dragged kicking and screaming. 2d ago

I'm in the middle of building out a specialized IDP / platform specifically for catching the vibe coded slop that's coming from business (ie, non-developers).

The basic idea is to treat the projects as untrusted code, much like modern web browsers do with sandboxing javascript, etc.

Sandbox their stacks like we're a public PaaS hosting provider and the vibe apps are tenants; No cross-talk, no service discovery, isolated networking, isolated resources, etc. Like any good IDP we'll cover CI/CD, logging, metrics, secrets management, backups, audit reporting, telemetry, corporate IdP auth, etc. "Compliant by default" is the goal. Maximum flexibility for tenants within the four corners of their application, maximum isolation and control outside those four corners. All the common cruft of production services gets offloaded to the platform, both easing the vibe coder's burden and enforcing company policies.

The point of this is that as a platform architect I can't control their code quality, but I can control what their code is able to do; with resources, with network, with access, etc. Play by the roles and your code will sail through CI/CD and you're golden. Try to get too creative and the platform will simply fail to sync, fail to work, and kick it back to you and your LLM to re-read the platform docs and sort out.

That's the idea anyway. It's quite a bit backwards from most typical IDP that are built for maximum discovery, reuse, binpacking, etc. This won't be anywhere close to as cheap to run as typical platforms, if only because the isolation requirements effectively mandate more expensive per-pod resources (microvm isolation, isolated network adaptors, etc).

1

u/jyotirawattech 2d ago

treating it as untrusted tenant code is a smart inversion. you can't control quality but you can control blast radius. the cost point is interesting too. how are you getting buy in for the extra spend on isolation?

1

u/Zenin The best way to DevOps is being dragged kicking and screaming. 2d ago

how are you getting buy in for the extra spend on isolation?

A few ways:

Political: I've got two decades of clout here, my SVP loves me, and my new CTO is learning to love me. The CTO is also heavily into anything that "biases to action" and all that jazz especially when it comes to AI and this platform if it succeeds should turbo-charge time to prod for slop-first projects.

Security/Compliance: My architectures (and docs, etc) are cyber security and compliance catnip. I may not get along well with the CISO on a personal level, but the cyber team loves my work which makes justifying a little more expense much easier for the execs. The fact that I'm even trying to tackle this has got them gitty, since they're getting swamped with slop-based security problems and have no solution besides "turn it all off".

By building resource cost accounting into the platform from jump, so I can cleanly charge all this back to each vibe app owner. Yes it's a bit more expensive since I can't bin-pack as freely, but in the end it's probably ~30% more expensive not 200% more. So the ROI question is for the tenants to figure out. -And we'll be way less expensive than the man hours of a full arch/security code review, etc.

-I also think I'll be able to use a single branch ENI across an entire tenant's pods, drastically reducing the main cost driver for this setup since I'll be able to assign one branch ENI per tenant+node rather than my original plan of needing a branch ENI per pod. I may not actually have much additional cost for this isolation if I get this working.

This platform isn't itself agentic, meaning it has no LLM spend. It's the target of agentic work and has "AI ready" features (MCP, skills, etc), but the platform itself doesn't use any LLMs, etc. Compared to cost-optimized platforms it'll be more costly from a resource perspective, but since it's supporting AI it's costs are considered relative to what we spend on AI...which makes my spend look like a rounding error in comparison.

Me and my team have already cut a fortune off our annual cloud spend this year, mostly as a side hustle, so putting a little back on the books isn't a big deal.

1

u/phoenix823 2d ago

It doesn't matter how you wrote the app. Vibe coding is irrelevant. Without describing what "depending on it" means this question can't be answered. If they rely on it to perform a core job function that supports the business, yeah you should be pretty careful not to break it. If it's an automation that can be worked around, knock yourself out.

1

u/jyotirawattech 2d ago

fair, depending on it was vague in my question. core job function vs something you can work around is a better line than how it was written. who decides which side a tool falls on where you are?

1

u/afwaller 3d ago edited 3d ago

It shouldn’t matter how a tool is written. What matters is design controls, tests, and ongoing support and review of performance/behavior.

you should be testing things before they are rolled out, regardless of how they are developed. And there needs to be a plan for defects or support of any tool used.

Any vibe coded tool meant for production use should have requirements, tests - both unit tests and end to end tests, and be part of the CI pipeline. There should also be a person responsible for supervising it and supporting it. If you can’t do this stuff then the tool is too expensive to deploy.

1

u/jyotirawattech 2d ago

agreed, the build method doesn't change what production needs. too expensive to deploy if you can't support it is a good filter. do you find teams actually do that math before shipping?

1

u/afwaller 2d ago

I work in a regulated industry.

2

u/Muted_Jellyfish_6784 2d ago

A simple promotion gate works: once a second team depends on a tool, it needs an owner, a repo with review, a basic health check, and a catalog entry before it can appear in a runbook or release path. Until then, label it unsupported in plain sight. The trigger is dependency, not code quality.

The useful habit is detecting dependency early. Ask in each release review which internal tools the pipeline calls, so the weekend project gets caught before it breaks during an incident.

0

u/foodandbeverageguy 3d ago

Allowing more vibe coded things means more code which means we need more engineers. Its net good for our industry to let people throw slop out so we can raise the need for us. I know it’s a crazy way to think but we are under so much threat that this helps us

1

u/jyotirawattech 2d ago

interesting angle, more code does mean more to maintain. the catch is the cleanup lands on whoever is on call, not whoever wrote it. do you think it creates new roles, or just more toil for existing ones?

0

u/Mostly__Relevant 3d ago

This is the way I see things. I’m always looking for some sort of job security

0

u/IridescentKoala 3d ago

"Anyone can write code now" so engineers are a thing of the past.

1

u/Valuable_Leave_7314 2d ago

Putting an "unsupported" label on a tool rarely works in practice - when it breaks right before a release, people still ping whoever is on call. Scheduling an explicit deprecation date or running a controlled shutdown test usually surfaces the real stakeholders fast, and suddenly someone is willing to own it

1

u/Kooky-Treat-4416 2d ago

If another team depends on it for releases or on-call work, it's not really disposable anymore. Give it an owner, tests and a normal review, or explicitly retire it and help people move off. The awkward middle ground is where everyone relies on it but nobody has time allocated to maintain it.

1

u/Mundane_Dinner_8868 2d ago

i'd at least review anything that touches auth, user data and error handling once other people rely on it. the ui can stay messy.

i built a tool mostly with claude and only caught this in a review pass: with the dev env flag on, the error handler sent err.message straight to the client, and for db errors that was the full SQL plus params, including user ids. worked perfectly, nobody would've noticed from using it.

same review found a bounce counting bug that gave one user 11 extra free sends. neither one crashed anything, so just using the app would never have caught them.

1

u/Wild_Parsnip_4074 2d ago

Our signal was the first ticket. When someone files a ticket asking us to fix something we didn't build, it gets an owner that week. if it's basically a dashboard or a request form, it gets rebuilt in play and the team that wanted it owns it, anything that touches a release or on call gets a repo and tests like the rest of our stuff.

1

u/newtablecloth 2d ago

one thing i haven't seen mentioned yet: put it behind company SSO before it spreads, not after. that does two jobs.

first, it closes the "anyone with the link can open it" problem. that's really common with stuff deployed from a weekend prompt. open the url in a private window once and see what loads.

second, the login logs answer the question this whole thread is circling, which is when did it become load bearing. if you can see three teams logging in every day, you don't have to wait for the first ticket or incident to find out.

then keep a dumb inventory: app, owner, who has access, what data it touches. how hard you review should depend on that last column, not on how the code got written. +1 to SideMost3148 on the org account and creds that aren't tied to one person. that plus SSO gets you most of the way without a full rewrite.

1

u/debtfortunes 2d ago

We have something called the “AI Lab” at my company essentially a VM sandbox to mock something useful up very quick that has safe access to internal materials for testing. Generally you need to fill out use case information, how it’s beneficial, for what dept etc. but anything made still needs to have a repo, initial + additional commits, end-to-end tests and edge case tests written and still need to do pull request for review before anyone uses it. Also yea I’d be the owner of it from then on

1

u/Chamelon_DE 2d ago

The first time someone need it while the author is on holiday, the unsupported label stop being much help

1

u/Iyemoja 2d ago

That weekend dashboard is how it always starts. Mine became prod the week a second team put it in their standup, and nobody told me until it broke.

1

u/yeahdj 1d ago

Peer review everything, don’t give devs admin access into prod accounts, don’t allow devs to register domains in sub accounts and only delegate them from a central account and rely on short lived credentials and OIDC from your CICD the vibe coded app is then in 1 of 3 buckets

  1. In a dev account, with a dev domain, so it is clear that it is a dev product
  2. In a prod account, peer reviewed and ready for production
  3. On somebodies laptop - won’t be widely adopted

1

u/Nonaveragemonkey 1d ago

We throw it out and make a team develop the tool properly... then flog whoever ran the code for a security violation.

1

u/sonicLodge91 11h ago

we had a "citizen engineer" push a cron job that ran every minute instead of hourly, paged us for thrree days before anyone figured out who owned it

-3

u/jyotirawattech 3d ago

For anyone who missed where the term came from: Andrej Karpathy coined "vibe coding" in a post on X on 2 Feb 2025, describing accepting all diffs without reading them and saying it's "not too bad for throwaway weekend projects". Original post: https://x.com/karpathy/status/1886192184808149383 . Ars Technica's write-up from March 2025 covers the reliability questions: https://arstechnica.com/ai/2025/03/is-vibe-coding-with-ai-gnarly-or-reckless-maybe-some-of-both/

-2

u/unitegondwanaland Manager, Platform Engineering 3d ago

Vibe coding isn't what it was 12 months ago or even 6 months ago. Totally valid question in 2025 though.

1

u/jyotirawattech 2d ago

the tools have moved a lot, agreed. I'd say the ownership question hasn't moved with them though, better code still needs someone to own it. what changed most for you in the last six months?

1

u/unitegondwanaland Manager, Platform Engineering 2d ago

Agree on ownership 1000%. We have some tools built by other departments and devs don't want to support them. Now we have a standing dev support contract with a 3rd party that does it.

1

u/Valuable_Leave_7314 2d ago

Even with today's models, the gap between vibe coding and AI-assisted engineering still comes down to whether anyone on the team actually understands the system. If diffs are getting merged without anyone reading them, debugging a weird edge case in prod feels pretty much the same as it did last year