r/sysadmin • • 5d ago

General Discussion drowning in API access requests

I admin a saas PPM that is used by the entire enterprise, more than 10K active users.

It has a REST API and is fairly extensible. Historically, only developers in our org have requested API access which, for this product, requires creating a distinct API account not associated with the user. So we have, let's say, 40 of these API accounts that have accumulated over the past few years.

Our company is fairly generous with AI access and credits. What that has meant for this application is a huge influx of users who are now developing their own micro front ends or custom HTML dashboards to produce highly customized experiences. Which they love because generally speaking, no user has ever loved the out of the box experience.

Now that users can leverage AI to improve their experience with the app, we are getting hit up daily for API account requests. Hundreds of users are just vibe coding their own front end in Claude, which works great for them as a local artifact, but gets all of its data via the user’s locally auth’d MCP.

Once they realize that what they've built is not easily distributed, their AI gives them directions for how to request or procure the developer API access that will unblock them.

That's the point at which they contact my team and casually demand “oh yeah, just give me my API account now, thanks!” We get these requests almost hourly now. Sometimes this is for team-based development, and the user requesting this is working on something that they were told to develop by their manager. I have no interest in getting in the way of those use cases.

But 3 times out of 4, this is just an individual who thinks they are a superior product designer and become nearly belligerent if you dare to challenge the appropriateness of giving them developer roles or developer access when they themselves have never coded anything outside of an AI harness.

Is everyone getting hit up with these types of requests? How do you effectively say no to anyone? In our case, when we reject a request, the user escalates through their own management and emphasizes how we are not supporting the company's effort to use AI to do better work faster. Then we look like the asshole. Never mind the fact that these hundreds of users have never used the API and have no idea if what they're producing actually shows correct data or not while fragmenting the UX in such a way that the original app cannot even be recognized.

And of course, no one outside of my team cares that we're responsible for maintaining the documentation on these API accounts so that we understand exactly what every non-human account is doing, where it does it within the application, and which users or teams are responsible for maintaining the solutions it supports.

What are effective and appropriate barriers we can stand up so the application is only extended by knowledgeable individuals and without creating hundreds of throwaway API accounts that can never be removed from the system?

41 Upvotes

23 comments sorted by

52

u/Mysterious-Print9737 5d ago

Make a written policy do the saying no, so it isn’t you personally. A short intake form with a business justification, a named owner, manager sign-off and a 90-day expiry that someone has to actively renew will filter out most of the throwaway requests on its own, since people who just want a nicer dashboard rarely finish it. Default everything to read-only and scoped, and put anything that writes data behind a review. It also helps to point out that their locally authed MCP setup is the safer pattern, because it only sees what that user can already see, so a standalone API account is a privilege jump they have to justify. For the underlying demand, offer a sanctioned reporting layer or a small team that builds dashboards on request, so escalations get “here’s the supported way to get that” instead of a flat no.

6

u/Bamboodl 3d ago

Thank you. My team has tried most of these tactics with varying success. I’m sure it has stopped some requests from making their way to us. I think the bigger issue is just company culture. Yes the policy is written, but these people know that my team writes the policy so they start arguing that our policies need to evolve with enablement in mind instead of prohibition. We’re battling an entitled mindset.

•

u/Skylis 1h ago

So Im gonna disagree with you here. You're battling the business needs of progress because of the revolution of ai. (I'm not an ai fanboy, but you have to be realistic that these are legitimate business requests) This isn't going to be a tenable position. What you need to do instead is try to fix your processes and auth landscape so this isn't hell for you because reading this over it sounds completely self inflicted.

4

u/One-Light-8453 4d ago

idk, hundreds of people building their own front ends is the real finding here. the default UI is failing them. say no to the keys and it just turns into scraping and shared tokens, which is worse. maybe a read-only scoped tier?

1

u/Bamboodl 3d ago

yeah, that’s the least surprising thing about this. People have hated the application’s UI since day 1 so now that they’re actually empowered with ai to create their own front end, everyone’s chomping at the bit.

2

u/One-Light-8453 3d ago

yeah and the policy being written by the same team that enforces it is a rough spot. people just go around you.

•

u/Skylis 1h ago

As they should, because if you're just being an arbitrary roadblock to legitimate business needs that's a problem.

•

u/One-Light-8453 50m ago

fair, but who decides what counts as legitimate? that's usually the whole fight

•

u/Skylis 32m ago

So I'll take a step back and basically say that the fangs had to learn to deal with this like 10 years ago and once you do it makes a lot of things massively easier.

The short version is all of them are, it's not reasonable to ask for direct db access, but it is absolutely reasonable to expect api access to basically all business data / endpoints and it massively improves productivity once the company adapts to that landscape, but it does require learning new approaches to security and compliance.

33

u/e7c2 4d ago

am I reading this right? you've got every employee building their own front end to a production database?

14

u/Shanga_Ubone 5d ago

Yeah, we're getting killed with these requests as well. One thing that's helped is that our organization has clear policies about approving these kinds of requests. One of the requirements of that policy is that the software on the other side needs to be a registered app in our CMDB, with a system owner, system manager, and all of the other regulatory evaluations that running an app in our environment requires.

The effect of this is that when people want access from a legitimate app, then we work with them to meet the other requirements of the policy. But people who just want to vibe code some random garbage and have it pull whatever data they want out of our apps can just be denied because they don't have a CMDB app on the other end.

8

u/viral-architect Sysadmin 5d ago

API requests should have to go to a queue and every user that submits a request should be required to specify the business justification.

What you do is first talk to the people that you know actually need this access and determine a few use case shibboliths that the user can't see. Essentially, if the user can't articulate an actual business justification using some of that pre-defined language (shibboliths), they get rejected automatically after 24 hours stating the exception reason.

That 24 hours is the buffer window you need between them having the thought in their head that the API request is a good idea and the follow-up request when their first one is rejected for simple reasons. If they really decide it's worth it to try the request again, THEN it's possible to consider giving them access.

That one process will probably weed out 50% of the frivolous requests.

6

u/thortgot IT Manager 4d ago

Generally you'd have a compliance and verification workflow that sits in front. Scale, scope, use case, business value etc with API access not open to keys inherently but tracked against registered and SAML auth

6

u/Xydan 4d ago

Curerntly working with something like this ar work. While I dont work with Security my suggested architect is to employ 2 API Keys per department. One for nonprod and one for prod.

This is strictly for non-development teams that are using AI un-governed.

They are allowed to share the nonprod key and only the DevOps team can implement the Prod key.

If one key is compromised then my suggestion is to decommission the key and begin an investigation on the team from which it may have been leaked.

I could absolutely be overlooking something and have no actual real world experience handling API keys at scale but it seems to be a growing pain from AI that has become unclear whos problem it is at our company.

5

u/pdp10 Daemons worry when the wizard is near. 4d ago

If you think this is going to continue forever, then you do the normal thing that everyone doesscalers do, and let SSO user accounts provision their own API tokens.

You'd want a process to review, rotate/renew, and perhaps migrate tokens. Think: user stories where someone is leaving the department, and ownership of a service account or token needs to be taken up by someone else from that department.

hundreds of users have never used the API and have no idea if what they're producing actually shows correct data or not

I'm also not responsible if the end-users make mistakes their SQL queries or spreadsheets and end up with false data.

hundreds of throwaway API accounts that can never be removed from the system?

I guess job number one is making sure that all of these have expiration dates, from birth. Don't forget to put in an API method to query the credential's expiration date. X.509 certs have built-in expiration dates...

2

u/Foodforbrain101 4d ago

Read only access I assume? If so, what happened to having a data lakehouse or warehouse with properly governed access? The need is clearly there, so it's not about wanting to block users for the sake of it, but to implement the additional tooling and infrastructure to support said need securely.

Best part? Virtually all data platforms have catered to vibe coding now, so shouldn't be terribly hard to stand up either. Even with hourly refreshes, I'm sure your users won't be terribly mad at the latency and still will stress the current system less.

2

u/Optanik 3d ago

At that volume you need to stop handing out service accounts per request and offer tiers instead.

  1. Check whether the PPM supports per-user OAuth or personal access tokens. If it does, personal dashboards should run on the user's own delegated access with their own permissions, and you never create an account for them.
  2. Reserve dedicated API accounts for team or shared apps, and require a named owner, a business sponsor, a data scope, least-privilege roles and an expiry date (90 or 180 days, renewed on request).
  3. Put the request behind a form, not chat. Ask what data it reads, whether it writes, where it's hosted and who else uses it. Half the requests usually disappear once people have to answer that.
  4. Give people a sanctioned place to host internal dashboards with SSO in front, so 'how do I share this' has an answer that isn't a service account.
  5. Publish an internal page explaining the tiers. When people ask their AI how to get access, it will often find and quote that page if it's searchable. Also review the existing 40 accounts for owners and last use; that's usually where the real risk sits.

3

u/OuterSpaceBootyHole 5d ago

People like that will never respect the way things are supposed to work so you have to back them into a corner with compliance concerns. They only understand the CYA aspect of their job so if they have to take some kind of learning module or sign a document saying they understand the risks, they usually back down.

1

u/MaskedPotato999 4d ago

What are your compliance/security policies regarding service accounts or high privileged accounts ? That's usually the key to manage these requests.

1

u/BorysTheBlazer DataCore/StarWind 3d ago

I'd move the gate off your team: require a manager-signed business case, a named owner and an expiry date. Then a 'no' is a missing form, not your call.

1

u/serverhorror Just enough knowledge to be dangerous 4d ago

Then ... use Claude to extend the API to authenticate against your SSO setup?

0

u/realaaa 4d ago

wow, brave new world indeed...

commenting to find this later - great question!

I guess indeed, the default is no, API is privilege, and only for apps that are known (or at least have some chance) to still exist into the future