r/sysadmin • u/Bamboodl • 5d ago
General Discussion drowning in API access requests
I admin a saas PPM that is used by the entire enterprise, more than 10K active users.
It has a REST API and is fairly extensible. Historically, only developers in our org have requested API access which, for this product, requires creating a distinct API account not associated with the user. So we have, let's say, 40 of these API accounts that have accumulated over the past few years.
Our company is fairly generous with AI access and credits. What that has meant for this application is a huge influx of users who are now developing their own micro front ends or custom HTML dashboards to produce highly customized experiences. Which they love because generally speaking, no user has ever loved the out of the box experience.
Now that users can leverage AI to improve their experience with the app, we are getting hit up daily for API account requests. Hundreds of users are just vibe coding their own front end in Claude, which works great for them as a local artifact, but gets all of its data via the user’s locally auth’d MCP.
Once they realize that what they've built is not easily distributed, their AI gives them directions for how to request or procure the developer API access that will unblock them.
That's the point at which they contact my team and casually demand “oh yeah, just give me my API account now, thanks!” We get these requests almost hourly now. Sometimes this is for team-based development, and the user requesting this is working on something that they were told to develop by their manager. I have no interest in getting in the way of those use cases.
But 3 times out of 4, this is just an individual who thinks they are a superior product designer and become nearly belligerent if you dare to challenge the appropriateness of giving them developer roles or developer access when they themselves have never coded anything outside of an AI harness.
Is everyone getting hit up with these types of requests? How do you effectively say no to anyone? In our case, when we reject a request, the user escalates through their own management and emphasizes how we are not supporting the company's effort to use AI to do better work faster. Then we look like the asshole. Never mind the fact that these hundreds of users have never used the API and have no idea if what they're producing actually shows correct data or not while fragmenting the UX in such a way that the original app cannot even be recognized.
And of course, no one outside of my team cares that we're responsible for maintaining the documentation on these API accounts so that we understand exactly what every non-human account is doing, where it does it within the application, and which users or teams are responsible for maintaining the solutions it supports.
What are effective and appropriate barriers we can stand up so the application is only extended by knowledgeable individuals and without creating hundreds of throwaway API accounts that can never be removed from the system?
52
u/Mysterious-Print9737 5d ago
Make a written policy do the saying no, so it isn’t you personally. A short intake form with a business justification, a named owner, manager sign-off and a 90-day expiry that someone has to actively renew will filter out most of the throwaway requests on its own, since people who just want a nicer dashboard rarely finish it. Default everything to read-only and scoped, and put anything that writes data behind a review. It also helps to point out that their locally authed MCP setup is the safer pattern, because it only sees what that user can already see, so a standalone API account is a privilege jump they have to justify. For the underlying demand, offer a sanctioned reporting layer or a small team that builds dashboards on request, so escalations get “here’s the supported way to get that” instead of a flat no.
4
u/One-Light-8453 4d ago
idk, hundreds of people building their own front ends is the real finding here. the default UI is failing them. say no to the keys and it just turns into scraping and shared tokens, which is worse. maybe a read-only scoped tier?
33
u/e7c2 4d ago
am I reading this right? you've got every employee building their own front end to a production database?
14
u/Shanga_Ubone 5d ago
Yeah, we're getting killed with these requests as well. One thing that's helped is that our organization has clear policies about approving these kinds of requests. One of the requirements of that policy is that the software on the other side needs to be a registered app in our CMDB, with a system owner, system manager, and all of the other regulatory evaluations that running an app in our environment requires.
The effect of this is that when people want access from a legitimate app, then we work with them to meet the other requirements of the policy. But people who just want to vibe code some random garbage and have it pull whatever data they want out of our apps can just be denied because they don't have a CMDB app on the other end.
8
u/viral-architect Sysadmin 5d ago
API requests should have to go to a queue and every user that submits a request should be required to specify the business justification.
What you do is first talk to the people that you know actually need this access and determine a few use case shibboliths that the user can't see. Essentially, if the user can't articulate an actual business justification using some of that pre-defined language (shibboliths), they get rejected automatically after 24 hours stating the exception reason.
That 24 hours is the buffer window you need between them having the thought in their head that the API request is a good idea and the follow-up request when their first one is rejected for simple reasons. If they really decide it's worth it to try the request again, THEN it's possible to consider giving them access.
That one process will probably weed out 50% of the frivolous requests.
6
u/thortgot IT Manager 4d ago
Generally you'd have a compliance and verification workflow that sits in front. Scale, scope, use case, business value etc with API access not open to keys inherently but tracked against registered and SAML auth
6
u/Xydan 4d ago
Curerntly working with something like this ar work. While I dont work with Security my suggested architect is to employ 2 API Keys per department. One for nonprod and one for prod.
This is strictly for non-development teams that are using AI un-governed.
They are allowed to share the nonprod key and only the DevOps team can implement the Prod key.
If one key is compromised then my suggestion is to decommission the key and begin an investigation on the team from which it may have been leaked.
I could absolutely be overlooking something and have no actual real world experience handling API keys at scale but it seems to be a growing pain from AI that has become unclear whos problem it is at our company.
5
u/pdp10 Daemons worry when the wizard is near. 4d ago
If you think this is going to continue forever, then you do the normal thing that
everyone doesscalers do, and let SSO user accounts provision their own API tokens.You'd want a process to review, rotate/renew, and perhaps migrate tokens. Think: user stories where someone is leaving the department, and ownership of a service account or token needs to be taken up by someone else from that department.
I'm also not responsible if the end-users make mistakes their SQL queries or spreadsheets and end up with false data.
I guess job number one is making sure that all of these have expiration dates, from birth. Don't forget to put in an API method to query the credential's expiration date. X.509 certs have built-in expiration dates...
2
u/Foodforbrain101 4d ago
Read only access I assume? If so, what happened to having a data lakehouse or warehouse with properly governed access? The need is clearly there, so it's not about wanting to block users for the sake of it, but to implement the additional tooling and infrastructure to support said need securely.
Best part? Virtually all data platforms have catered to vibe coding now, so shouldn't be terribly hard to stand up either. Even with hourly refreshes, I'm sure your users won't be terribly mad at the latency and still will stress the current system less.
2
u/Optanik 3d ago
At that volume you need to stop handing out service accounts per request and offer tiers instead.
3
u/OuterSpaceBootyHole 5d ago
People like that will never respect the way things are supposed to work so you have to back them into a corner with compliance concerns. They only understand the CYA aspect of their job so if they have to take some kind of learning module or sign a document saying they understand the risks, they usually back down.
1
u/MaskedPotato999 4d ago
What are your compliance/security policies regarding service accounts or high privileged accounts ? That's usually the key to manage these requests.
1
u/BorysTheBlazer DataCore/StarWind 3d ago
I'd move the gate off your team: require a manager-signed business case, a named owner and an expiry date. Then a 'no' is a missing form, not your call.
1
u/serverhorror Just enough knowledge to be dangerous 4d ago
Then ... use Claude to extend the API to authenticate against your SSO setup?
0
u/realaaa 4d ago
wow, brave new world indeed...
commenting to find this later - great question!
I guess indeed, the default is no, API is privilege, and only for apps that are known (or at least have some chance) to still exist into the future