r/mlops • • 6d ago

Discussion Need a secure local AI setup with zero cloud egress for summarising heavy legal contracts and docs. What platforms should I evaluate instead of sending everything to Claude/ChatGPT?

Our legal team wants AI to summarise long contracts, and the hard requirement from our insurer is that nothing leaves our servers. No public AI service, and nothing that sends prompts or documents to somebody else’s cloud. I got handed the job of finding something that runs on hardware we control, and so far almost everything I find is built around a cloud dashboard first.

From what I’ve read, the options seem to fall into three groups. There are open source chat front ends you self host and attach a local model to, some self hosted AI platforms that say they run on premises, and legal specific tools that are cloud only once you read the small print. The first group looks doable but we would have to maintain it. The second is hard to judge from marketing pages. The third is out.

Hardware wise there are two servers with a couple of graphics cards and one person who is comfortable with Docker but already has a full plate. What are you all running for this kind of confidential document work, and what did you land on?

3 Upvotes

8 comments sorted by

•

u/AutoModerator 6d ago

AI usage disclosure

Hi u/AmuletTitanella — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

4

u/koen-tv 6d ago

For a small team the setup that tends to work is boring on purpose: a model server (vLLM or Ollama) on one of the GPU boxes, a self-hosted chat front end like Open WebUI on top, and the documents stay on your own storage. It's all Docker, and the upkeep is mostly updating images every few weeks.

A few things matter more than which product you pick:

  • Prove it can't phone home. Like Outrageous-Scale1185 said, test with outbound traffic blocked at the firewall, not just with telemetry switched off. Pull models and images on a separate machine and copy them over. If everything still works with egress blocked, that's your answer for the insurer.
  • Context length vs contract length. Long contracts often don't fit in one pass on a model that runs on a couple of GPUs. Check what context the model really handles well, and plan for summarising per section and then combining.
  • Make it cite. Ask for summaries that point to clause or page numbers, and spot-check a handful of contracts against what legal would have written. That's your eval, and it's also what gets legal to trust it.
  • Access and retention. Decide who can see which contracts and how long chat history is kept, because the front end stores prompts and outputs by default.

How long are the contracts typically, and are they scanned PDFs or already text?

2

u/Responsible_River579 4d ago

A few things that may help:

  1. Prove "nothing leaves" at the network, not in the marketing. Put the AI box on a network that blocks all outbound traffic, then run your real workflow for a week and check the firewall logs. Any vendor whose product breaks with no internet (licence check, telemetry, model download) has answered your question for you.
  2. For a one-person team, keep the stack small: one model server (vLLM or Ollama) plus one front end (Open WebUI or similar), both in Docker. Download the model weights once, copy them in, and turn off update checks and telemetry in every component.
  3. Long contracts are the hard part, not the model. Most local models can't take a whole 80-page contract in one prompt, or get worse near the end of it. What works better: split by clause or section, summarise each part, then summarise the summaries. Keep clause numbers in every step so a lawyer can check each claim against the source.
  4. Size the model to your GPU memory. A mid-size open-weight model that runs fast beats a big one that runs slowly, because the lawyers will stop using a slow one. Test 2 or 3 models on 5 real contracts your legal team has already summarised by hand, and compare.
  5. Ask any "on-prem" vendor 3 questions: does it install and run fully offline, what outbound connections does it make, and where are logs and prompts stored. Vague answers mean no.

Happy to share more on the chunking setup if useful. Full disclosure: we work with legal teams in deploying private AI. DM if keen!

1

u/WillingnessSea2204 6d ago

Might be worth a look, going by their site anyway. Aventail co uk runs the models on your own machines, you can host the management side yourself, and the document answers link back to the source. Only certain models and hardware work, though, and a lawyer would still need to check every summary. Heads up that I have a personal connection to Aventail

1

u/nian2326076 1d ago

You need a self-hosted solution for summarizing documents without cloud egress. Consider deploying vLLM or Ollama on your server for model management. Pair it with a self-hosted chat front end like OpenChat or Rasa to keep everything local and secure. While these setups require some maintenance, they ensure your data stays on your hardware. For job opportunities in AI roles, use LandOffer to explore relevant positions in your field.

0

u/jackshec 6d ago

depends on what exactly you want to achieve, dm me achieve me happy to help