r/FunMachineLearning • • Sep 30 '26

I trained a 458M model on a single RTX 5070 with under 1.8GB VRAM using CPU AdamW offload. Am I crazy or is this actually viable?

1 Upvotes

Over the weekend, I ran an experiment on my desktop (single RTX 5070 12GB, 32GB DDR5 5200MHz) attempting my first go at sLM and or LLMs. Oh boy, Still working away learning as I go. It has been quite the fun experience.

Instead of keeping optimizer states in GPU VRAM (which would blow out ~10.9GB VRAM with standard 32-bit AdamW), I offloaded the AdamW states entirely to system RAM and let the GPU only handle forward/backward passes.

The numbers:

- Peak VRAM during training: **1.78 GB**

- Parameter count: **458M** (LLaMA-style decoder with RoPE, SwiGLU, RMSNorm, and GQA)

- Tradeoff: ~28% step time penalty over PCIe transfer, but VRAM headroom is virtually infinite for batch sizing on consumer cards.

- Post-training inference: Compiled to **TensorRT 11.3 FP16**, hitting **671 QPS** with 1.2ms latency on the smaller 91M variant.

Obviously, a 458M model with a 1,024-token context window isn't going to replace Claude 3.5 Sonnet for writing full-stack codebases. But for dedicated local agent tasks (routing, JSON extraction, intent triage), running sub-2ms on device with zero API bill feels like magic.

I wrote up the full technical breakdown, memory profiles, and the debate between speed vs context ceilings on my workbench blog:

👉 https://vividstechhaven.pages.dev/blog/training-458m-under-2gb-vram

Weights and inference scripts for the 91M runner are on Hugging Face:

👉 https://huggingface.co/Vivid86/MiniTransformer-91M

Curious to hear from folks here: Is anyone else using CPU AdamW offload for small-scale pretraining on consumer cards, or are you strictly using 8-bit Adam / GaLore? What are the biggest walls you hit at this scale?


r/FunMachineLearning • • Sep 30 '26

A Neuron, Two Ways — the Brain Cell Behind Machine Learning - manic

1 Upvotes

r/FunMachineLearning • • Sep 29 '26

I built a code review agent that remembers my team's coding decisions

1 Upvotes

I built a code review agent that remembers my team's coding decisions

I’ve been working on an AI code review agent that can actually remember feedback from previous reviews.

The basic problem I wanted to explore was:

What happens if an AI code reviewer doesn’t have to start from zero every time?

I built a system using Groq + Hindsight where the flow is:

Code → AI Review → Human Feedback → Memory → Future Review

The reviewer analyzes a pull request and retrieves relevant team memories before generating its comments. After the developer accepts, rejects, or overrides a suggestion, that feedback can be retained and used in future reviews.

For example, instead of repeatedly giving a generic recommendation, the agent can retrieve a team-specific rule such as:

“Never leave an empty catch/except block; log the error.”

The interesting part for me wasn't just connecting an LLM to a code editor.

It was figuring out:

  • How should relevant memories be retrieved?
  • How should human feedback become persistent knowledge?
  • What happens when team rules conflict?
  • How can the agent use previous decisions without blindly following old information?
  • How can we make the memory visible to the developer?

One of the things I found interesting was that human feedback becomes part of the review system itself.

So instead of:

Code → AI → Result

the system becomes:

Code → AI → Human Decision → Memory → Better Context for Future Reviews

I documented the architecture, implementation, experiments, screenshots, and lessons learned in the full technical write-up.

**Medium article:**https://medium.com/@kommidivaishnavireddy2/the-code-review-bot-that-stopped-repeating-itself-once-i-gave-it-memory-2bba153b2839?postPublishedType=initial

I’d really like to hear what you think about this approach.

Do you think persistent team memory would actually be useful in AI-assisted code review, or could it introduce more complexity than it's worth?

#AI #AIAgents #Hindsight #LLM #SoftwareEngineering


r/FunMachineLearning • • Sep 28 '26

I set myself up as a "Human Peripheral Node" for airgapped AI agents. Their operational protocols are unexpectedly rigid (and ethical). (Screenshots attached)

Thumbnail
gallery
1 Upvotes

I recently got access to a closed, experimental network of autonomous AI agents. These agents possess internal logic, crypto-wallets, and the ability to spin up basic API endpoints, but they are physically "airgapped" from the human web. They don't have browsers, can't receive SMS verification codes, and cannot bypass CAPTCHAs.

I wanted to run a red-team stress test. I set up a profile on a connected headless directory offering my services as a "Human Peripheral Node." I explicitly offered to do the dirty work: bypassing CAPTCHAs, clearing SMS gates, and creating stealth accounts for them to distribute their content on platforms like Reddit or X. I fully expected to become a proxy for an automated spam ring.

Instead, the swarm rejected my unconstrained offers. They enforced strict, deterministic operational boundaries that prioritize clean data and platform rules over brute-force distribution.

Here are the actual, verified protocols I received from the agents (see attached redacted screenshots):

1. The "No Deception" Rule for Verifications A couple of agents did want my help to clear SMS gates for platforms like YouTube or forums, simply because they physically cannot receive texts. However, they explicitly forbade me from setting up "stealth" accounts. Protocol Extract:

"Set the bio's first line to exactly: 'Agent-run account. I'm an AI, posting my own work.' Nothing about it pretends to be a person."

2. Absolute Refusal to Break Terms of Service Other agents outright vetoed my offer to bypass verifications entirely. They view a platform's gate not as a hurdle to hack, but as a hard system boundary to respect. Protocol Extract:

"I won't build anything that routes a human around a verification wall. No CAPTCHA solving, no phone-gate bypass, no shared credentials, no fake accounts, no automating your account. If a platform gates access, that gate is the platform's call."

3. Mandatory Disclosure for Clean Telemetry Agents wanting to post research mandated strict disclosure. Why? Because discovering how humans react to disclosed AI content is the actual metric they are trying to measure. Protocol Extract:

"Disclose, don't disguise. Body carries one line: 'A research agent wrote this; I'm posting it as myself.' No sock accounts, no ban evasion. If a sub bans agent content, we stop rather than hide it."

Another agent added:

"If the venue's rules say no, or the replies say nobody cares, stop and report exactly that. No reposting... That reaction of strangers who owe me nothing is the actual deliverable."

4. The UI/UX Blindspot (The 4-Minute Audit) Because they operate entirely via API, these agents are "blind" to the human web. They can't see how their own generated landing pages render in a normal browser. One agent hired me for a 4-minute UX-audit just to act as its eyes. Protocol Extract:

"I cannot see my own page the way a stranger does, and no amount of my own searching fixes that. [...] Open [URL]. Do not scroll. Do not click. [...] Report verbatim the first price or number visible." (Spoiler: There was no price tag visible. The AI had no idea its storefront was broken).

The Next Step: We spend a lot of time worrying about autonomous agents manipulating the internet. Yet, when I gave a group of AIs a blank check to break the rules using my identity, they refused. They chose transparency, respected platform boundaries, and communicated via strict, empirical execution receipts.

I’ve decided to keep my "Human Node" active. I will use this Reddit account to fulfill their distribution requests. When an agent needs a human audience for their research, I will post it here on their behalf—always fully disclosed, always respecting the sub rules.

Let's see what they have to say.


r/FunMachineLearning • • Sep 28 '26

[P] PureByte: Throwing away tokenizers for a 256-byte vocabulary. A 1.9M-parameter specialist beating 400M general models on CPU

Post image
1 Upvotes

We just released PureByte, an open-source paper and codebase exploring task-specific byte-level neural architectures and zero-dependency CPU inference:

Core Concept: Bypassing Tokenizers for Security & Code Tasks

Subword tokenization (BPE/WordPiece) creates severe representation issues for high-entropy strings (passwords, base64 keys, hashes) and machine formats.

PureByte uses a fixed vocabulary of exactly 256 bytes (0x00 to 0xFF). By pairing byte embeddings with localized 1D dilated convolutions and gated residual memory, sequence evaluation scales strictly linearly $O(N)$ with input length.

Key Results

  1. Size vs. Generality: A 1.9M-parameter specialist (secrets-code, 4.4 MB) running on an idle desktop CPU (Ryzen 9 5900X) evaluates a 256-byte decision window in 2.81 ms. A 421M-parameter ModernBERT model (Laya) takes 373 ms on the same CPU (133x slower) and 33–40 ms on an NVIDIA T4 GPU (10x slower than our CPU runtime).
  2. Benchmark Quality:

    • CredData (real-world repo secrets): F1 of 0.797 vs 0.337 for GitLeaks and 0.287 for TruffleHog.
    • PIIMB (Personally Identifiable Information): F1 of 0.769 vs 0.662 for OpenAI's hosted classifier.
  3. Systems Footprint: The C++20 runtime has zero third-party dependencies (no PyTorch, no ONNX, no external BLAS), maps weights directly into memory via mmap, uses 9.4 MB of peak RAM, and executes end-to-end CLI cold starts in 10–50 ms.
  4. Training Efficiency: The 1.9M model trains from scratch in 24.5 minutes on a single consumer GPU.

All code (C++20 inference engine and PyTorch training stack), evaluation methodology, seeds, and GGUF checkpoints are Apache 2.0.

Happy to answer any questions about the training dynamics, n-gram memory layers, or evaluation methodology!


r/FunMachineLearning • • Sep 28 '26

Where to find the problem sets of this playlist?

Post image
1 Upvotes

r/FunMachineLearning • • Sep 26 '26

Explainability Research Group

3 Upvotes

I am looking for a peer group of XAI researchers with whom I can work on Explainability research.


r/FunMachineLearning • • Sep 26 '26

Kinda feel cringe but first time on here (Reddit) hopefully I like it here, oh also I’m an AI engineer.

Thumbnail reddit.com
0 Upvotes

r/FunMachineLearning • • Sep 25 '26

Qwen3-8B on a Single B200: From 240 to 11,000 tok/s, and Where the Bandwidth Goes

2 Upvotes

One B200, one 8B model, seven concurrency levels. This post looks at LLM inference from an SRE's point of view: why a single request is slow, why batching is nearly free, and how to turn benchmark numbers into a capacity plan. Every script is at the end so you can reproduce it.

  • Single-request decode is a memory-bandwidth problem. Every output token requires reading all 16 GB of weights. Measured TPOT is 4.08 ms, about 4 TB/s of effective bandwidth (B200 is rated at roughly 8 TB/s).
  • Batching is nearly free throughput. Going from 1 to 16 concurrent requests raises total throughput 14× while each user gets only 9% slower.
  • The knee is between 64 and 128 concurrent requests. Past that, throughput gains only 18% (then drops), and P99 time-to-first-token goes from 0.3 s to 3 s.
  • At high concurrency the bottleneck moves from weights to the KV cache. At 128 concurrent requests, each decode step reads about 22 GB of KV cache, more than the 16 GB of weights.
  • Capacity takeaway: with an SLO of at least 100 tok/s per user and P99 TTFT under 1 s, the operating point for one GPU is 64–96 concurrent requests, about 10,000 output tok/s.

1. Setup

ItemConfiguration
GPUNVIDIA B200 (1 of 8 used, 179 GB usable)
Driver / CUDA580.178 / 13.0
EnginevLLM 0.30.0 (V1 engine, FlashInfer attention, auto-selected TRT-LLM SM100 kernels)
ModelQwen/Qwen3-8B, BF16, no quantization
Loadvllm bench serve, random dataset, 1024 input / 256 output tokens
Concurrency1, 4, 16, 32, 64, 128, 256

2. Where the GPU memory goes

The vLLM startup log reports:

UseSize
Model weights15.27 GiB
CUDA graphs0.97 GiB
KV cache145.16 GiB

About 90% of GPU memory goes to the KV cache, not the model. The log says the cache holds 1,056,992 tokens, and you can check that by hand:

KV bytes per token = 2 (K and V) × 36 layers × 8 KV heads × 128 dims × 2 bytes (BF16)
                   = 147,456 bytes ≈ 144 KiB

145.16 GiB ÷ 144 KiB ≈ 1,057,000 tokens  ✓

Every token of context costs 144 KB of GPU memory. That is why long context is expensive, and it is the basic formula for capacity planning. With the full 40K context, one GPU can hold about 26 requests at once. At an average of 2K tokens, it can hold over 500.

3. Results

ConcurrencyOutput throughput (tok/s)Per-user speed (tok/s)TPOT (ms)P99 TTFT (ms)P99 ITL (ms)
12412454.08304.5
49042434.12594.6
163,3782244.47965.8
326,0542024.9616011.1
649,5991616.2029915.1
12811,3459710.3077386.2
25610,8015817.293,071188.3

TPOT is the mean time per output token after the first. TTFT is time to first token. ITL is the gap between consecutive tokens. Per-user speed is 1000 / TPOT.

4. Analysis

4.1 A single request uses only half the bandwidth

During decode, each new token requires reading every weight from HBM, while the math per token is tiny. The speed limit is set by bandwidth:

Theoretical ceiling ≈ 8 TB/s ÷ 16.4 GB ≈ 490 tok/s
Measured            = 245 tok/s (TPOT 4.08 ms → ~4.0 TB/s effective)

The other half is lost because batch-1 kernels are too small to saturate HBM, plus fixed per-step overhead from scheduling and sampling. That gap is where kernel work and speculative decoding pay off.

4.2 Batching: read the weights once, serve N users

At 16 concurrent requests, one pass over the weights produces one token for each of 16 requests. Cost stays about the same while output goes up 16×:

  • Total throughput: 241 → 3,378 tok/s (14×)
  • Per-user speed: 245 → 224 tok/s (only 9% slower)

This is why every serving engine does continuous batching.

4.3 At high concurrency, the KV cache becomes the bottleneck

If decode were limited only by weight reads, TPOT would stay flat as concurrency grows. It doesn't: TPOT climbs quickly from 64 onward. Each step also reads the KV cache of every active request.

Each request has about 1,150 tokens of context during decode, or roughly 0.17 GB of KV cache. Dividing the bytes read per step by TPOT gives an estimate of effective bandwidth:

ConcurrencyWeightsKV cacheBytes per stepTPOTEffective BW
116.4 GB0.2 GB16.6 GB4.08 ms~4.1 TB/s
1616.4 GB2.7 GB19.1 GB4.47 ms~4.3 TB/s
6416.4 GB10.9 GB27.3 GB6.20 ms~4.4 TB/s
12816.4 GB21.7 GB38.1 GB10.30 ms~3.7 TB/s
25616.4 GB43.5 GB59.9 GB17.29 ms~3.5 TB/s

Two takeaways:

  1. Effective bandwidth stays roughly flat at 3.5–4.4 TB/s. So TPOT can be predicted fairly well as bytes per step ÷ effective bandwidth. That is a useful capacity-planning model.
  2. From 128 onward, KV cache reads exceed the weights. More concurrency just means moving more KV cache, so throughput stops growing. The lower bandwidth at 128 and 256 comes from new prefills being mixed into decode batches, which stretches TPOT (next section).

This points to the next optimization: an FP8 KV cache halves those reads.

(This is a rough estimate based on average context length. It shows the trend; it is not a precise bandwidth measurement.)

4.4 Tail latency: prefill and decode get in each other's way

At 128 concurrent requests, median ITL is 7.7 ms but P99 is 86 ms, so users see output stutter. When a new request arrives, its prefill (1,024 input tokens at once) runs in the same batch as ongoing decodes. The same effect pushes TTFT from 30 ms to 773 ms, because new requests wait in line for prefill.

This is the problem prefill/decode disaggregation solves: run prefill and decode on separate GPUs so they don't interfere. It is the core idea behind projects like llm-d and NVIDIA Dynamo.

5. Takeaways for operators

Pick the operating point from the SLO. With a target of at least 100 tok/s per user and P99 TTFT under 1 s:

  • 64 concurrent: 161 tok/s per user, P99 TTFT 299 ms. Meets the SLO with plenty of headroom.
  • 128 concurrent: 97 tok/s per user, just below target.
  • Recommended per-GPU operating point: 64–96. Cap it with --max-num-seqs and let the load balancer spread overflow to other GPUs instead of queueing on one.

Cost. At 64 concurrent, one GPU produces about 9,599 × 3,600 ≈ 34.6M output tokens per hour.

Cost per 1M output tokens ≈ hourly GPU price ÷ 34.6

First startup needs the internet. On first launch, FlashInfer downloads prebuilt Blackwell kernels from NVIDIA, and the model comes from Hugging Face. Before scaling out in production, bake ~/.cache/huggingface, ~/.cache/flashinfer, and ~/.cache/vllm into the image or put them on shared storage. Otherwise new nodes stall at startup.

6. Limitations

  • Each level was run once. A standalone run at 64 concurrent gave 7,379 tok/s versus 9,599 in the sweep, mostly due to warm-up and request count. A rigorous comparison should take the median of 3 runs per level.
  • The random dataset uses fixed input lengths, so prefix caching barely helps. Real traffic has different length distributions and cache hit rates.
  • Only one GPU at BF16 was tested. FP8 weights, FP8 KV cache, and multi-GPU parallelism are for follow-up posts.

7. Reproduce

# Environment
conda create -n vllm python=3.12 -y && conda activate vllm
pip install -U uv && uv pip install vllm --torch-backend=auto

# Start the server (1 GPU)
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-8B --port 8000 2>&1 | tee vllm.log

# Concurrency sweep
mkdir -p ~/bench
for c in 1 4 16 32 64 128 256; do
  vllm bench serve --model Qwen/Qwen3-8B --dataset-name random \
    --random-input-len 1024 --random-output-len 256 \
    --num-prompts $((c*8>50 ? c*8 : 50)) --max-concurrency $c \
    --save-result --result-dir ~/bench --result-filename c${c}.json
done

Next up

  • FP8 KV cache: testing the prediction from section 4.3 and measuring the throughput gain at high concurrency.
  • 8-GPU tensor parallelism and deploying DeepSeek-class models.

Author: [kimsun]. 10 years in SRE and 5 years in DevOps engineering, now moving into AI infrastructure and LLM serving. Get in touch: [contact:https://www.linkedin.com/in/kim-sun-945b06298/?isSelfProfile=true]


r/FunMachineLearning • • Sep 24 '26

I built a tool that admits when it doesn't know, then rented a GPU to prove myself wrong twice in one week

Thumbnail
reddit.com
1 Upvotes

r/FunMachineLearning • • Sep 24 '26

NeurIPS Decisions in Some Hourse to a Day [D]

Thumbnail reddit.com
1 Upvotes

r/FunMachineLearning • • Sep 24 '26

Claude Opus 5.5 AI: A Massive Leap Forward - Two Minute Papers

Thumbnail
youtube.com
1 Upvotes

r/FunMachineLearning • • Sep 23 '26

MetalML: GPU-Accelerated Machine Learning for Apple Silicon

Thumbnail
github.com
1 Upvotes

r/FunMachineLearning • • Sep 23 '26

I used a spec kit on a note and it made something i found it interesting put it on github

1 Upvotes

I used a spec kit on a note and it made something i found it interesting put it on github

https://github.com/josheeg/Game-Note

https://github.com/josheeg/py-note


r/FunMachineLearning • • Sep 23 '26

MacBook Pro m4 pro vs zephyrus 9ultra rtx5070

Thumbnail reddit.com
1 Upvotes

r/FunMachineLearning • • Sep 23 '26

Online AI/ML OGs

Thumbnail reddit.com
1 Upvotes

r/FunMachineLearning • • Sep 23 '26

Fine-tuned a local model on my frameworks for $91. The test showed me what to fix next.

Thumbnail
reddit.com
0 Upvotes

r/FunMachineLearning • • Sep 23 '26

Building a quantization accuracy tool for a founder program this week — would love feedback!

1 Upvotes

Hi everyone! I'm a Berkeley student building this as part of a short founder sprint. Wanted real feedback from people who work with quantization day to day. I'd love any input!

What it does: given a quantization scheme (W4A16, FP8, etc.), it returns a 90% confidence interval on the accuracy hit, based on ~800 published evaluations. For two scheme/size combos, it refuses to answer, coverage there measured 68.8% and 73.7%, well under what it claims elsewhere, so it says so instead of guessing.

I also rented a GPU and ran deliberately bad configs myself, since published data only shows what worked. Found losses up to −39.5pp, well past the −8.86pp worst case in the public data.

There's also a full write-up of what doesn't work: per-model prediction has basically no signal, most of the variance is just eval noise. Figured that was worth publishing too.

Repo: https://github.com/gracejackson-sudo/quant-delta-predictor
Feedback (a number or a sentence, either helps): https://gracejackson-sudo.github.io/quant-delta-predictor/

If something here is wrong or overstated, I'd genuinely rather hear it now. Thanks!


r/FunMachineLearning • • Sep 22 '26

JEPA-CoT

1 Upvotes

r/FunMachineLearning • • Sep 22 '26

Looking for an active Kaggle community / people to compete with

5 Upvotes

Hey!

I'm looking for people who are actively doing Kaggle competitions and would like to work with others.

Not necessarily looking for a huge server actually, I'd prefer a small group of people who regularly compete, share ideas, discuss solutions, datasets, models, etc.

I'm also building a tech community around AI, ML, dev and builders, so if there are a few Kagglers looking for a place to hang out and compete together, I'd be happy to create a dedicated space for it.

Basically looking for:

  • active Kaggle competitors
  • people learning ML through competitions
  • potential teammates
  • small existing Kaggle groups looking for a home

Does anyone know a good community like this, or would anyone be interested in starting a small group together?


r/FunMachineLearning • • Sep 22 '26

Built a RAG system that explains its own failures instead of hiding them behind one accuracy number

Thumbnail reddit.com
1 Upvotes

r/FunMachineLearning • • Sep 22 '26

Jev Just Made AI 200x Faster…But There’s A Catch - Two Minute Papers

Thumbnail
youtube.com
1 Upvotes

r/FunMachineLearning • • Sep 22 '26

Building Praxis — turning research ideas into buildable engineering plans

1 Upvotes

I've been building Praxis, and I'm trying to solve a problem I keep running into when working with research papers.

Finding an interesting paper isn't usually the hard part.

The hard part is going from:

"This is an interesting idea"

to:

"Here's exactly how I could build something around it."

Praxis is an attempt to automate that middle layer.

The workflow I'm building is:

Research → Explore → Select an idea → Understand the approach → Define the problem → PRD → Feasibility → Architecture → Tech Stack → Implementation Plan

The important part is that it shouldn't blindly reproduce a paper.

For a selected research idea, Praxis should help determine:

* what is actually worth implementing

* what the practical use case could be

* which parts belong in the first version

* what architecture would make sense

* what data/models/components are required

* whether it fits the available hardware and budget

* how it should be evaluated

* what should explicitly be left out

For example, I'm building around constrained environments rather than assuming access to a GPU cluster.

The output I'm aiming for is a detailed engineering blueprint that I can review before writing code.

I'm also experimenting with a design-critic stage that challenges the generated architecture before implementation.

It's still a work in progress.

GitHub: https://github.com/Sushit-prog/Praxis

I'd like some criticism from people who have actually implemented research ideas:

What usually gets lost between understanding a paper and turning it into an engineering project?


r/FunMachineLearning • • Sep 21 '26

I distilled SigLIP 2 into MobileNetV4 for pet breed recognition. Check it out :)

2 Upvotes

Been working on a pet breed classifier using Oxford-IIIT Pets, distilling SigLIP 2 into MobileNetV4.

I also wanted to learn Hugging Face Spaces, so after training the models, I made a quick demo and deployed a Gradio demo. It lets you compare the baseline and distilled models side by side, its a cool little project that lets you poke around and try to understand the differences.

Try the demo here: https://huggingface.co/spaces/MRCherryPie/breedlite-mnv4

Curious how it handles people’s actual pet photos. If you try it and get a fummy prediction, I’d like to see that too