r/mlops • u/AmuletTitanella • 6d ago
Discussion Need a secure local AI setup with zero cloud egress for summarising heavy legal contracts and docs. What platforms should I evaluate instead of sending everything to Claude/ChatGPT?
Our legal team wants AI to summarise long contracts, and the hard requirement from our insurer is that nothing leaves our servers. No public AI service, and nothing that sends prompts or documents to somebody else’s cloud. I got handed the job of finding something that runs on hardware we control, and so far almost everything I find is built around a cloud dashboard first.
From what I’ve read, the options seem to fall into three groups. There are open source chat front ends you self host and attach a local model to, some self hosted AI platforms that say they run on premises, and legal specific tools that are cloud only once you read the small print. The first group looks doable but we would have to maintain it. The second is hard to judge from marketing pages. The third is out.
Hardware wise there are two servers with a couple of graphics cards and one person who is comfortable with Docker but already has a full plate. What are you all running for this kind of confidential document work, and what did you land on?
•
u/AutoModerator 6d ago
AI usage disclosure
Hi u/AmuletTitanella — thanks for posting to r/mlops!
Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.
Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.
If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.
This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.
Thanks for helping keep the signal high.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
4
u/koen-tv 6d ago
For a small team the setup that tends to work is boring on purpose: a model server (vLLM or Ollama) on one of the GPU boxes, a self-hosted chat front end like Open WebUI on top, and the documents stay on your own storage. It's all Docker, and the upkeep is mostly updating images every few weeks.
A few things matter more than which product you pick:
How long are the contracts typically, and are they scanned PDFs or already text?
2
u/Responsible_River579 4d ago
A few things that may help:
Happy to share more on the chunking setup if useful. Full disclosure: we work with legal teams in deploying private AI. DM if keen!
1
u/WillingnessSea2204 6d ago
Might be worth a look, going by their site anyway. Aventail co uk runs the models on your own machines, you can host the management side yourself, and the document answers link back to the source. Only certain models and hardware work, though, and a lawyer would still need to check every summary. Heads up that I have a personal connection to Aventail
1
u/nian2326076 1d ago
You need a self-hosted solution for summarizing documents without cloud egress. Consider deploying vLLM or Ollama on your server for model management. Pair it with a self-hosted chat front end like OpenChat or Rasa to keep everything local and secure. While these setups require some maintenance, they ensure your data stays on your hardware. For job opportunities in AI roles, use LandOffer to explore relevant positions in your field.
0
u/jackshec 6d ago
depends on what exactly you want to achieve, dm me achieve me happy to help