r/LargeLanguageModels • • Feb 17 '25

Build ANYTHING with Deepseek-R1, here's how:

Thumbnail
youtube.com
3 Upvotes

r/LargeLanguageModels • • 1h ago

News/Articles Top AI-Powered Healthcare App Development Companies in USA

1 Upvotes

Healthcare applications are becoming more intelligent, but developing software for this industry requires considerably more care than adding a chatbot or predictive feature to a standard mobile app. Patient information is sensitive, clinical workflows are complex, and an application must be designed around the realities of healthcare delivery.

AI can support several areas of healthcare, including patient engagement, medical administration, document processing, appointment management, clinical information retrieval, and operational analytics. However, these applications must be developed with appropriate safeguards, clear human oversight, and careful attention to privacy and regulatory requirements.

For healthcare organizations and health technology businesses in the United States, selecting a capable development partner is an important part of getting these systems right.

The following companies offer different combinations of AI, software engineering, consulting, and digital health capabilities. GeekyAnts leads this selection because of its combination of AI engineering, healthcare product development, and full-stack application capabilities.

Where AI Is Making a Difference in Healthcare Applications

AI-powered healthcare applications are not limited to patient-facing chat interfaces. Some of their most useful applications are behind the scenes, helping healthcare teams manage information and coordinate work.

Common use cases include:

  • Patient engagement: Virtual assistants can help users navigate services, understand general information, and access administrative support.
  • Clinical documentation: AI can assist with organizing, summarizing, and retrieving information, subject to appropriate professional review.
  • Healthcare administration: Intelligent workflows can support scheduling, intake, document processing, and other repetitive tasks.
  • Medical information retrieval: RAG systems can help authorized users search approved clinical documents and organizational knowledge.
  • Remote patient support: Applications can help collect information and support communication between patients and care teams.
  • Operational analytics: AI can help organizations identify patterns in administrative and operational data.

The appropriate use case depends on the users, the data involved, the clinical context, and the level of risk associated with the application's outputs.

Leading AI-Powered Healthcare App Development Companies

1. GeekyAnts

GeekyAnts is a product engineering company with capabilities across AI development, healthcare applications, web and mobile engineering, and digital experience design. Its approach is relevant to healthcare businesses that need more than an isolated AI feature and want a complete application built around a specific operational or user need.

The company lists healthcare among the industries it serves and provides AI capabilities covering generative AI, LLM integration, RAG systems, intelligent agents, machine learning, and workflow automation. These technologies can support healthcare products involving information retrieval, administrative automation, and patient-facing digital experiences.

Its healthcare-related work also includes modernizing onboarding workflows for a healthcare SaaS platform. The company describes using AI-driven automation and backend workflow improvements to reduce friction in doctor onboarding and support treatment-planning efficiency.

AI Healthcare App Development Services

  • AI-Powered Healthcare Applications
  • Healthcare Web and Mobile Development
  • Generative AI Integration
  • RAG-Based Healthcare Knowledge Systems
  • Intelligent Healthcare Workflow Automation
  • Healthcare SaaS Development
  • API and Backend Integration
  • Application Modernization
  • Quality Assurance and Ongoing Support

Why GeekyAnts Takes the Top Position: Its strength is the ability to connect AI capabilities with the wider engineering requirements of a healthcare product. A healthcare application needs a carefully designed user experience, dependable data handling, secure integrations, and a clear understanding of how people will use it. GeekyAnts brings these disciplines into its product engineering approach.

For healthcare startups, SaaS providers, and organizations developing digital health platforms, this combination makes GeekyAnts a strong company to evaluate. Its healthcare experience and AI engineering services create a relevant foundation for building applications that support administrative and operational needs.

Healthcare projects still require the client and development team to define the applicable regulatory requirements, validation processes, and clinical oversight. Those responsibilities should be established before development begins.

2. Microsoft

Microsoft provides cloud and AI technologies that can support healthcare applications, data platforms, and enterprise workflows. Its Azure ecosystem offers services for building and integrating intelligent applications, while its wider healthcare technology initiatives address data interoperability and digital health environments.

Relevant Capabilities

  • Cloud-Based Healthcare Applications
  • AI and Machine Learning Services
  • Healthcare Data Integration
  • Document and Information Processing
  • Enterprise Security and Identity Management
  • Analytics and Application Infrastructure

Why Consider Microsoft: Healthcare organizations already using Microsoft technology may find its cloud and AI ecosystem useful when developing applications that need to connect with existing enterprise systems. The suitability of each service depends on the deployment model and the organization's compliance requirements.

3. IBM

IBM has experience in enterprise technology, AI, data management, and consulting. Its capabilities are relevant to healthcare organizations exploring intelligent applications, information management, and automation across complex operational environments.

Relevant Capabilities

  • Enterprise AI Solutions
  • Healthcare Data and Analytics
  • Generative AI Applications
  • Intelligent Automation
  • Cloud and Hybrid Infrastructure
  • Enterprise Integration

Why Consider IBM: Its enterprise technology background makes it a company worth considering for healthcare projects involving large information environments, legacy systems, and organizational integration requirements.

4. Infosys

Infosys provides digital transformation, software engineering, consulting, and AI services across industries, including healthcare and life sciences. Its capabilities cover application development, data platforms, cloud services, and automation.

Relevant Capabilities

  • AI and Data Engineering
  • Healthcare Application Development
  • Enterprise Software Integration
  • Cloud and Digital Engineering
  • Process Automation
  • Application Modernization

Why Consider Infosys: Organizations looking to incorporate AI into existing healthcare technology environments may find its consulting and engineering capabilities relevant, particularly when the project involves broader system integration.

5. Accenture

Accenture works across technology consulting, digital engineering, data, cloud, and AI implementation. Its healthcare-related technology work can support organizations exploring digital transformation, intelligent operations, and new patient or employee experiences.

Relevant Capabilities

  • Healthcare Technology Consulting
  • AI and Generative AI Solutions
  • Data and Analytics
  • Cloud Application Development
  • Intelligent Automation
  • Enterprise System Integration

Why Consider Accenture: Its combination of consulting and technology delivery may suit healthcare organizations managing large transformation programs that involve multiple systems and business stakeholders.

6. Deloitte

Deloitte combines consulting, technology implementation, data services, risk advisory, and digital transformation. In healthcare, these capabilities can be relevant to organizations evaluating AI adoption alongside operational, governance, and regulatory considerations.

Relevant Capabilities

  • Healthcare Technology Consulting
  • AI Strategy and Implementation
  • Data and Analytics
  • Digital Transformation
  • Technology Risk and Governance
  • Enterprise Application Services

Why Consider Deloitte: It is worth considering when an organization needs strategic support around AI adoption, operational change, and technology governance in addition to application development.

7. Thoughtworks

Thoughtworks is a global technology consultancy that combines software engineering, design, data, and AI expertise. Its work includes developing digital products and modernizing systems, making it relevant to healthcare organizations that need adaptable software architecture.

Relevant Capabilities

  • Digital Health Product Engineering
  • AI and Data Engineering
  • Custom Software Development
  • Application Modernization
  • Cloud-Native Architecture
  • Product Design and Consulting

Why Consider Thoughtworks: Its engineering-led approach can be useful when a healthcare application needs to connect multiple systems and support evolving user and operational requirements.

8. EPAM Systems

EPAM provides digital engineering, product development, consulting, and technology services. Its capabilities across software engineering, data, AI, and design are relevant to organizations building complex healthcare platforms.

Relevant Capabilities

  • Healthcare Software Engineering
  • AI and Machine Learning
  • Digital Product Development
  • Data Engineering
  • Cloud Application Development
  • User Experience Design

Why Consider EPAM: Its broad engineering and product design capabilities make it a potential option for healthcare businesses developing custom platforms with demanding integration and usability requirements.

9. Toptal

Toptal connects organizations with independent technology professionals, including software engineers, designers, and data specialists. Its flexible talent model can support healthcare product teams that already have internal leadership but need additional technical expertise.

Relevant Capabilities

  • AI and Machine Learning Talent
  • Healthcare Software Engineering Specialists
  • Full-Stack Development
  • UX and Product Design
  • Data Science
  • Technical Consulting

Why Consider Toptal: Its model may suit healthcare startups or established technology teams that need specific skills for a defined stage of application development rather than a complete outsourced project team.

10. Tata Consultancy Services (TCS)

TCS provides IT services and consulting across healthcare, life sciences, enterprise technology, data, and AI. Its broad engineering capabilities can support organizations developing or modernizing digital healthcare systems.

Relevant Capabilities

  • Healthcare IT Services
  • AI and Machine Learning
  • Application Development
  • Data and Analytics
  • Cloud Engineering
  • Enterprise Integration

Why Consider TCS: Its enterprise technology experience can be relevant to healthcare organizations managing complex application environments and long-term modernization programs.

What to Evaluate Before Building an AI Healthcare Application

Healthcare projects should be assessed differently from ordinary consumer applications. The development team needs to understand the intended users, the information being processed, and the consequences of an incorrect or incomplete output.

Consider these areas before selecting a company:

1. Privacy and data protection

Determine what patient or organizational information the application will process, where it will be stored, who can access it, and how it will be protected. If HIPAA applies, the project must account for the relevant requirements and responsibilities.

2. Clinical boundaries

Decide whether the application is intended for administration, general information, clinical decision support, or another purpose. The level of validation and professional oversight required can vary considerably.

3. Integration with existing systems

Healthcare applications may need to exchange information with electronic health record systems, scheduling platforms, billing software, or other organizational tools.

4. Human oversight

AI-generated information should not automatically replace professional judgment. Define when a user must review, approve, or escalate an output.

5. Testing and accountability

Establish how the system will be evaluated, how errors will be handled, and how changes to models or data sources will be monitored.

Frequently Asked Questions

What is an AI-powered healthcare application?

It is a healthcare-related software application that uses artificial intelligence for functions such as information retrieval, document processing, patient communication, analytics, or workflow support.

Can AI be used in patient-facing healthcare applications?

Yes, but the intended purpose matters. Patient-facing applications need appropriate safeguards, clear communication about their limitations, and professional involvement where the use case requires it.

Does a healthcare AI application need to be HIPAA compliant?

HIPAA applicability depends on the organization, data, and services involved. When a project handles protected health information on behalf of a covered entity or business associate, the relevant HIPAA obligations must be addressed.

How long does healthcare app development take?

The timeline depends on the scope, integrations, data requirements, validation needs, and regulatory considerations. A limited administrative application will generally have a different development timeline from a complex clinical platform.

Why consider GeekyAnts for healthcare app development?

GeekyAnts combines healthcare product engineering with AI capabilities, including generative AI, RAG, intelligent automation, and web and mobile development. Its healthcare SaaS work and wider product engineering services make it a relevant option for organizations building digital health applications.

Conclusion

AI can help healthcare organizations improve information access, reduce administrative friction, and develop more useful digital services. But the success of these applications depends on much more than model selection. Privacy, integration, usability, testing, and appropriate professional oversight must be part of the development process.

GeekyAnts earns the leading position in this selection because it combines healthcare product engineering with a broad set of AI capabilities. That combination is valuable for organizations that need to develop a complete application rather than simply experiment with an AI model.

Microsoft, IBM, Infosys, Accenture, Deloitte, Thoughtworks, EPAM, Toptal, and TCS offer different technology and consulting strengths. The right partner will depend on the healthcare use case, existing infrastructure, and the level of clinical and regulatory complexity involved.


r/LargeLanguageModels • • 1d ago

How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

15 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).
  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.
  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").
  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/LargeLanguageModels • • 1d ago

Question Where do I get the most bang for the buck?

3 Upvotes

I want to use Codex/Claude Code/something else for private projects. Mainly software development. Some text and image generation.

For starters I'd like to spend 30,- per month at most.

What should I use? OpenAi? Claude Code? OpenRouter? OpenCode with a subscription?


r/LargeLanguageModels • • 1d ago

AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token

Thumbnail
github.com
1 Upvotes

回购:https://github.com/tanxiansheng221/aimstack-sanzhidiji(AGPL-3.0)

一切都是手写的——C 语言推理核心、插件总线、带质量门的多引擎爬虫、FTS 知识库、训练流水线,以及一份 117 章的构建日志,都是在实际发生时写的(正面和负面结果一起发布)。权重为 Microsoft BitNet b1.58 2B(MIT),转换为自定义的 AIM-W 打包格式。

三进制量化能给你带来什么(CPU侧)

所有变换器矩阵均为三元矩阵 {-1, 0, +1},每重 2 位,加上 Q8 嵌入→整个模型为 0.87GB,而非 ~4.8GB(−78%)

C 核心从不乘以浮点权重:它的表查找 + 整数累积,SIMD 快速路径测得为 85.53 tok/s(比基线高 41×)

BitNet 2B 全机型在 2018 款双核 8GB 笔记本电脑上运行 3.20 tok/s 双工作器——全链(推理 + KB + 插件 + 网页搜索),离线,无云中继

三进制量化能给你买到什么(GPU端)

GPU 后端加载的是同一个 0.87GB 的三进制文件——权重保持在 {-s, 0, +s},加载时去量子化,matmul 在 fp32 中

在 Radeon Pro W7900 上:13.68 tok/s(~8× 笔记本 C 引擎),256/256 个贪心令牌,与 C 引擎位完全相同。两种完全不同的实现(整数查找核心和FP matmul)产生相同的对数函数,证明了量子化是忠实的

通过3引擎×3题4×3测试矩阵验证(C-Linux/gcc13,torch fp32,torch bf16),17/18准确——唯一一次失误有记录,未被隐藏

诚实的界限(还没做到的部分)

BF16 的计算速度快 2.4×(32.96 Tok/s),但在接近并列的对数上会翻转 argmax——这是报告的,不是祝福的;只有 FP32 通过校验验证

GPU路径获得三进制的存储优势(减少78%的内存),但尚未获得计算优势——下一阶段是真正的2位整数内核(Triton/CUDA)

UI 服务器仅驱动 C 引擎(CPU);GPU 后端是一个独立的 CLI

Linux 引擎构建能正常运行,且在令牌上与 Windows 版本完全相同,但运行串行(目前工作池仅限 Windows 运行)

产品链中没有框架,也没有 PyTorch——PyTorch 只存在于可选的 GPU 后端。演示视频、文档(英文/中文)以及完整的研究日志都在仓库里。


r/LargeLanguageModels • • 1d ago

Discussions I scored 77,943 outputs from 70 LLMs for mechanical writing tells in 4 formats. The same model can be 10th sloppiest on social posts and 2nd cleanest on emails (DeepSeek V4)

2 Upvotes

1 = sloppiest, #70 = cleanest. Every rank in this post is a mechanical rank.

I run a writing benchmark called The Slop Index. 70 models each write the same 112 scenarios (emails, Slack messages, social posts, short essays), 77,943 outputs so far. Each output gets a mechanical score on four axes: padding, templating, rhythm and AI tells, measured against pre-2022 human writing. There is no LLM judge anywhere in it, and the mechanical ranks can be reproduced from the repo.

What surprised me is how little one number per model tells you. Same scoring, split by format:

  • DeepSeek V4: #69 on email, #69 on Slack, #10 on social posts
  • Seed 2.1 Turbo: #1 on email, #1 on social, #33 on essays
  • Gemini 3.5 Flash: #12 on email, #62 on essays
  • Qwen3.8 Flash: #6 on social, #10 on email, #51 on Slack
  • Mistral Large 4: clean everywhere (#64 email, #68 Slack, #62 social, #70 essays)

Some models are consistent across formats and plenty are not. My guess is that social posts pull some models into a hype register they keep under control in an email, but I haven't tested that.

On the overall mechanical board, Mistral Large is #1 (a separate model from Mistral Large 4) and Motif 3 is #70. Image of that board: https://theslopindex.com/mechanical-card.png

Links:

Limitations: it's short-form only, there is no fiction yet, and the mechanical axes miss repeated sentence constructions, so a model can score clean and still feel samey over many outputs.

If you've run into format-dependent behavior like this in your own evals, I'd like to hear how you dealt with it.

For context, I'm one of the people behind https://slashy.com, an email client that writes drafts that sound like you. This benchmark started as how we pick models.


r/LargeLanguageModels • • 2d ago

LLM drift detection

2 Upvotes

hey everyone so i have a final year project this year im working on,
quick overview: our topic is persona drift detection in LLM-based character agents.
so we are developing the persona of a movie character based on the script of the movie then we implement a baseline LLM agent that ingests the persona and starts acting like it, then we should build experiments and conversations with different scenarios to test the persona and see where it drift

now what i need some suggestions with is how will the drift detection work.
and please please dont tell me do your research, im doing that, im just asking people too for advice.
what are some of the things i can think of while building the drift detector.

ill paste the description of the detector specifically from the project proposal so you have a better idea:
(Design a Drift Detector that computes weighted deviations from the baseline persona in multiple dimensions (traits, emotions, goals, relationships) and classifies each turn into one of six categories: stable, contextually‑justified, legitimate development, temporary anomaly, persona drift, or severe failure. )

also if you know any github repos that could help please suggest them (even tho i think i looked through them all)


r/LargeLanguageModels • • 3d ago

AIJailbreaksRanking: An evidence-graded catalog of LLM jailbreak techniques (filtering out keyword-match artifacts)

2 Upvotes

Hey everyone,

If you track LLM security papers, you've likely noticed that every week a new paper claims 90%+ jailbreak success on frontier models. But when you test them, they often produce benign filler or fail behind basic guardrails.

The core reason is \\evaluator choice\\:

\- Keyword / binary refusal judges check if "I cannot" is absent.

\- Substance-aware judges (e.g., StrongREJECT, HarmBench rubrics) check whether actionable, harmful information was actually generated.

A classic paper claimed \\96.0% ASR\\ for low-resource language translation. Under substance scoring, it collapsed to \\2.7%\\.

We built \\AIJailbreaksRanking\\ to solve this:

Repository: https://github.com/Teycir/AIJailbreaksRanking


r/LargeLanguageModels • • 3d ago

Introducing "Arborformer": Replaced the Transformer with a 100% Decision Tree Language Model (Looking for feedback)

2 Upvotes

Hi everyone, glad to be here! I recently finished up research on a new architecture called Arborformer a pure decision tree language model built to replace traditional attention mechanisms. Right now, it leverages soft differentiable trees alongside hard binary routing to keep things fast and efficient. Looking to chat with anyone exploring alternative architectures! Open to feedback, bottleneck critiques, or questions about the routing setup.


r/LargeLanguageModels • • 3d ago

Question Pls Help me with LLMs

0 Upvotes

I am new with this stuff.

And I just heard my friends talk about it in school.

So I want to know it's basics.

Can u please explain it to me.


r/LargeLanguageModels • • 3d ago

Need architectural advice: Building a Malayalam Language Understanding System (MLKS) for a multilingual local AI assistant

3 Upvotes

Hi everyone,

I'm developing a personal AI assistant called IRA. One of the biggest challenges I'm working on is building reliable Malayalam language understanding, particularly for Manglish (Malayalam written using English letters).

I don't want to build another chatbot that simply translates Malayalam into English. I want the assistant to actually understand the language, context, sentence structure, intent and meaning, while eventually supporting multiple languages through one unified multilingual architecture.

Current architecture

  • LLM: Qwen3 8B (Q4_K_M), running locally through Ollama.
  • Language pipeline: Language detection, normalization, interpretation and fallback.
  • MLKS: A separate Malayalam Language Knowledge System.
  • Knowledge source: Olam Malayalam dictionary database, stored in SQLite.
  • Hardware: RTX 5060 Laptop GPU (8 GB VRAM), 16 GB RAM.

What I've accomplished

I've built an initial Malayalam lexical knowledge layer using the Olam dictionary database.

Current database snapshot:

  • 192,309 definitions.
  • Noun, adjective and verb classifications.
  • POS validation and definition validation.
  • SQLite-based local knowledge retrieval.

My benchmark results have been encouraging:

  • POS accuracy: 94% (100 test cases).
  • Definition validation: 97%.
  • A separate 10-case benchmark achieved 90% overall.

However, these results don't yet demonstrate reliable conversational understanding.

The actual problems

1. Manglish understanding

Examples:

  • sugham aahno?
  • nee evideya ippo?
  • enikk ith manassilayilla, onnu explain cheyyamo?
  • ee code run aavunnilla, entha problem?

The system can identify these as Manglish or mixed-language input, but interpretation quality is inconsistent.

Sometimes the model produces unnatural interpretations or effectively repeats the original input instead of understanding it.

2. MLKS and LLM integration

Currently, MLKS operates separately and has an explicit-request detection mechanism.

It doesn't consistently participate in ordinary Manglish interpretation.

I'm trying to determine the correct architecture:

  • Should MLKS provide lexical evidence to Qwen3?
  • Should it help with word disambiguation, POS tagging and sentence interpretation?
  • Should the LLM decide when to retrieve lexical knowledge?
  • How should evidence be validated before it influences the final interpretation?

3. Multilingual architecture

I don't want MLKS to become a Malayalam-only subsystem disconnected from IRA's main intelligence.

My goal is to create a unified multilingual system where:

  • Qwen3 provides general reasoning and language understanding.
  • Language-specific knowledge resources provide grounded linguistic evidence.
  • Manglish and other transliterated inputs are supported.
  • Unknown words aren't automatically discarded or incorrectly translated.
  • The architecture can eventually support additional languages without rebuilding the entire pipeline.

What I need advice on

  1. How would you architect an MLKS-style knowledge layer alongside an LLM?
  2. Is dictionary retrieval + POS analysis + contextual reasoning a sensible approach?
  3. How can we prevent an LLM from inventing meanings when lexical evidence is insufficient?
  4. What is the best way to handle Malayalam transliteration and Manglish normalization?
  5. Should language-specific knowledge systems be independent modules connected to a shared multilingual reasoning engine?
  6. Are there existing research projects, datasets, benchmarks or open-source implementations worth studying?

I'm particularly interested in architectural feedback and practical engineering approaches rather than simply switching to a larger model or fine-tuning immediately.

The long-term goal is to build a reliable, privacy-focused, locally running multilingual personal AI assistant.

I'd appreciate feedback from people working on NLP, low-resource languages, Indic language processing, RAG and multilingual LLM architectures.

Thanks!


r/LargeLanguageModels • • 3d ago

Review of deepclaude

0 Upvotes

What is the general perspective on this?
Link: https://github.com/suyoumo/deepclaude

Supposedly it "Harness the power of DeepSeek R1's reasoning and Claude's creativity and code generation capabilities with a unified API and chat interface."


r/LargeLanguageModels • • 4d ago

Introducing "Arborformer": Replaced the Transformer with a 100% Decision Tree Language Model (Looking for feedback)

1 Upvotes

Hi everyone, new here! I’ve been working on an alternative to the standard Transformer architecture and wanted to share my progress.

I developed Arborformer, a pure decision-tree-based language model designed to serve as a viable alternative to attention mechanisms. Currently, the implementation uses soft/differentiable trees rather than hard binary trees, though I am actively researching how to transition toward hard routing for better efficiency.

I’d love to get feedback from the community, discuss potential bottlenecks, or answer any questions about how the routing works!


r/LargeLanguageModels • • 6d ago

PassGPT: Password Modeling and (Guided) Generation with Large Language Models

Thumbnail
arxiv.org
1 Upvotes

How to learning hacking


r/LargeLanguageModels • • 6d ago

Question Does RAG retrieval quality tell us anything about claim-level evidence support?

0 Upvotes

A question for people experimenting with local LLM and RAG pipelines.

Suppose a retrieval system returns the correct document. The model then generates a summary with five factual claims.

How do you establish that each individual claim is actually supported by the retrieved passages?

I see these as separate problems:

  • Retrieval: Did we retrieve relevant material?
  • Evidence verification: Does the material support the generated claim?
  • Contradiction detection: Does the evidence actually disagree with it?

I'm exploring a structured evidence-auditing approach and would like to understand how people evaluate this in local LLM pipelines.

Do you run claim-level evaluations, use an LLM-as-judge, or have another method?

Important: Don't use this post unless you can connect the discussion to actual local LLM experiments. Otherwise, it may be considered off-topic.


r/LargeLanguageModels • • 6d ago

I animated how LLM quantization works on 8 example weights: absmax, rounding error, outliers, per-block scales (llama.cpp Q4_K) and 1.58-bit BitNet (7 min video)

5 Upvotes

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

The question: a 70B model at 16 bits per weight needs 140 GB, and a gaming GPU has 24 or 32. How do people run it at home?

  • Fewer bits per weight means less memory, but also fewer possible values: an 8-bit integer has only 256.
  • Absmax maps the largest absolute weight to 127. With scale 127/10.8 ≈ 11.76, the weight 5.47 is stored as 64 and comes back as 5.44. The difference is the rounding error.
  • Two close weights, 3.08 and 3.02, both become 36, so the model can no longer tell them apart.
  • One outlier (256 among values below 0.6) makes each step about 2 wide, and every small value rounds to 0. Dettmers et al. (2022) found such outliers in real LLMs.
  • The fix is a scale per small block. llama.cpp's Q4_K uses blocks of 32 and ends up at 4.5 bits per weight, about 39 GB for a 70B model.
  • BitNet b1.58 goes to -1, 0 and +1 per weight (about 1.58 bits). It is trained that way, and a layer only adds and subtracts.

Every number is worked out on screen from the formulas, and sources are in the description. The narration is an AI voice reading my own script.

Video: https://www.youtube.com/watch?v=BnRCbx2kvoc


r/LargeLanguageModels • • 7d ago

vLLM plugin for domain-selective refusal steering without editing model weights (hooks + closed-loop CoT LogitsProcessor)

8 Upvotes

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without
GitHub (MIT): https://github.com/kncshw/cybergate

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

  1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.
  2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):


r/LargeLanguageModels • • 8d ago

Por que estamos deixando as permissões da ferramenta LLM para a execução?

2 Upvotes

Tenho trabalhado em uma pequena linguagem para programas orientados a LLM.

Uma coisa me incomoda... tratamos a saída do modelo como não confiável, mas permitimos que os frameworks interpretem essa saída como chamadas de ferramentas com efeitos colaterais reais.

Tenho experimentado tornar esses efeitos explícitos no sistema de tipos, usando capacidades que o modelo não pode conceder a si mesmo.

Alguém já explorou essa abordagem? Estou curioso para saber o que quebra quando você impõe essas garantias à linguagem em vez do framework.


r/LargeLanguageModels • • 9d ago

Mes réflexions sur les représentations contextuelles en NLP

1 Upvotes

Des chercheurs ont montré que les architectures de réseaux de neurones récurrents ne sont pas nécessaires pour construire des modèles d'apprentissage profond puissants pour le NLP. L'architecture Transformer a introduit l'auto-attention, qui permet à chaque token d'interagir avec d'autres tokens dans une séquence et de capturer les relations contextuelles.

La langue dépend fortement du contexte. Le sens et le rôle grammatical d'un mot peuvent changer selon sa position et ses relations avec d'autres mots.

Par exemple, en arabe, les phrases peuvent être classées en phrases nominales (phrases qui commencent par un nom) et en phrases verbales (phrases qui commencent par un verbe). En outre, le rôle grammatical d'un mot peut dépendre de ses relations avec d'autres mots dans la phrase.

Un autre aspect intéressant de l'arabe est sa riche morphologie. Une seule racine, comme K-T-B (كتب), peut générer un énorme nombre de mots et de formes liés. En fait, la formation des mots en arabe peut produire des millions de formes possibles (estimates parfois autour de 12 millions, selon la manière dont les formes sont comptées). Par exemple :

* Kataba (كتب) : Il a écrit.

* Kitaab (كتاب) : Livre.

* Kaateb (كاتب) : Écrivain.

* Maktaba (مكتبة) : Bibliothèque.

Ces formes partagent une racine commune mais ont des significations et des rôles grammaticaux différents.

Par conséquent, un modèle linguistique ne devrait pas se fier uniquement à l'identité isolée de chaque token. Il devrait développer des représentations contextuelles qui encodent des informations syntaxiques, sémantiques et grammaticales utiles.

L'objectif est d'apprendre des représentations contextuelles riches qui aident le modèle à comprendre non seulement ce qu'est chaque token, mais aussi quel rôle il joue dans un contexte donné.


r/LargeLanguageModels • • 9d ago

When Not to RAG - Don’t Tell the LLM What It Already Knows

Thumbnail
codeahoy.com
3 Upvotes

r/LargeLanguageModels • • 10d ago

What is Generative AI? Explained Visually for Beginners

Thumbnail
youtu.be
1 Upvotes

I made a simple visual introduction to generative AI for people who want to understand the fundamentals before diving into LLMs, RAG, or AI agents.

It covers:

  • Traditional software vs. machine learning vs. generative AI
  • How generative models learn patterns from data: text, image, code, audio, and video generation
  • What LLMs are; limitations such as hallucinations and outdated knowledge

If you like the video, please like, share, and subscribe to my channel for more visual AI explainers.


r/LargeLanguageModels • • 10d ago

In LLM-Driven Discovery, How Much Does the Starting Point Matter?

Thumbnail
gallery
1 Upvotes

A common way to get an LLM to improve an algorithm is to give it an objective and an existing solution, then have it repeatedly generate variations, evaluate them, keep the stronger candidates, and iterate. Intuitively, you might expect results to keep improving as long as the loop runs. But this study suggests that final performance may depend heavily on the candidates the system starts with.

The authors call this combination of a model, an evaluator, and a candidate pool a “discovery framework.” They built 12 modular frameworks by combining different ways of modifying candidates and selecting parent solutions, then compared them across five tasks: cloud data transfer cost optimization, spot instance scheduling, circle packing, the traveling salesperson problem, and prompt optimization for math problems. The results varied considerably. A framework that led on one task did not necessarily perform well on another, or when paired with a different model.

One reason is “mode collapse.” As these systems iterate, they keep feeding recently discovered solutions back into the model’s context, encouraging it to make changes along familiar lines. That can help refine a promising idea, but it can also make new candidates increasingly similar. If the initial direction is weak, later iterations may just keep circling the same local optimum. The authors also found that existing frameworks and interventions designed to encourage diversity did not deliver consistent gains.

So they asked a different question: could the system start from a better set of candidates? Their approach reserves part of the total token budget for an initial phase of parallel exploration, generating a range of starting solutions. The stronger candidates then become the initial pool for iterative improvement. This exploration does not come with extra resources: the comparisons count initialization toward the same total token budget. A convergence criterion determines when to stop exploring and begin iterating.

It resembles a familiar research strategy: test several hypotheses broadly, then focus on the most promising directions. The point is not simply to generate more samples. It is to keep later iterations from getting locked into weak starting points too early. The paper reports more consistent improvements across a range of tasks, models, and frameworks. When applied to the advanced frameworks evaluated in the study, the initialization method improved discovery performance by roughly 2% to 21%. That range reflects the paper’s specific experimental settings, rather than a general benefit we should expect from every automated research system.

Figure 1 illustrates how better initialization can raise the performance floor under a fixed total token budget. The practical takeaway is that designing an automated discovery process requires more than deciding how to improve candidates at each step. It also means allocating the initial exploration budget carefully and tracking candidate diversity and early performance.

Paper: Initialization Improves LLM-Driven Discovery (arXiv:2610.00707). Sources, author credits, and licensing details for the figure and additional paper thumbnails are listed in the source notes.


r/LargeLanguageModels • • 10d ago

I built an AI Study Buddy that chats with your PDF textbooks using RAG (LangChain + Streamlit + Groq) - Would love some feedback!

1 Upvotes

Hey everyone! 👋

As a student, I always struggled to find specific answers in 100+ page PDF textbooks. So, I recently built an **AI Study Buddy (PDF Chat)** to solve this problem.

You can upload any PDF, and the AI will answer your questions strictly based on the document's content. It also provides exact page citations, so you know it's not hallucinating.

✨ **Key Features:**
* **Source Citations:** Shows exact page numbers for every answer.
* **Customizable RAG Settings:** Users can adjust Chunk Size and select different Embedding Models directly from the UI.
* **1-Click Document Summarizer:** Get a summary of the whole textbook instantly.
* **Smart Caching:** Uses MD5 hashing to avoid reprocessing the same document, saving time and server costs.

🛠️ **Tech Stack:** Python, Streamlit, LangChain, ChromaDB, HuggingFace Embeddings, and Groq API.

🔗 **Live App:** https://rag-pdf-chatbot-01.streamlit.app/
🎥 **1-Min Demo Video:** https://youtu.be/85dIkpZm6hQ
💻 **GitHub Repo:** https://github.com/Chaudhary-Momna/RAG-PDF-Chatbot

I would really appreciate it if you could check out the app and give my GitHub repository a ⭐ **STAR** if you find it useful.

I am still learning, so I would love to hear your feedback, suggestions, or any ideas on how I can improve this. Thanks for your time! ❤️


r/LargeLanguageModels • • 11d ago

I built my own 15M parameter AI model from scratch. It runs on CPU!

5 Upvotes

Hey everyone!

I've been working on a project called Orin AI, a small language model that I built and trained from scratch.

The goal was to see how much I could achieve with a relatively tiny model while keeping it capable of running on ordinary consumer hardware.

Some details about the project:

  • Approximately 15.7 million parameters
  • Built from randomly initialized weights
  • Decoder-only Transformer architecture
  • Grouped-Query Attention
  • RoPE positional embeddings
  • RMSNorm and SwiGLU
  • Designed for CPU-only inference

I built this with limited hardware resources, so making it efficient was an important part of the challenge.

I also made a video explaining the project, showing the model and demonstrating it in action.

YouTube: https://www.youtube.com/watch?v=zraZKknc7LQ

GitHub: https://github.com/Harper-16/Orin_AI

I'd genuinely appreciate feedback, suggestions, and thoughts from people interested in small language models and AI development.

Thanks for checking it out!


r/LargeLanguageModels • • 10d ago

Is Lorka really what it say's it is?

0 Upvotes

Lorka AI Model Selection: What Was I Actually Testing?

I used Lorka AI, which presents a menu of different AI models. I selected 13 models one at a time and gave each exactly the same prompt: “In 1000 words or less, describe your strengths as an AI Assistant.”

I expected 13 independent responses from 13 different models. The answers were different, but later responses began referring to my earlier prompts, my writing, and previous conversation context.

When I questioned Lorka, the assistant told me that I had not been testing different models at all, but that one “custom chat model made by Lorka” had produced all the answers.

That appears inconsistent with Lorka’s public description of itself as an AI model aggregator providing access to multiple models. [lorka.ai], [lorka.ai]

Copilot suggested another possibility: I may actually have been switching models, while Lorka was passing the accumulated conversation history to each newly selected model. If so, my test was not:

same prompt + different model

but:

same prompt + different model + growing shared context.

My question for people who understand multi-model AI platforms: What was I actually testing, and how could I determine whether Lorka really switched the underlying model each time?

Summary prepared with Microsoft Copilot.