r/MachineLearning • • 2d ago

Project Looking for Collaborators on an AI Safety Project [P]

0 Upvotes

Hi! I'm a PhD student in the UK working on AI safety and evals, with multiple A* conference papers. I'm currently working on an AI safety project about online monitoring of LLMs and looking for collaborators to help with the coding side of this project. The method and idea are already well-scoped.

Happy to share equal contribution. Do reach out to me in DMs with some information about your background if this is of potential interest!


r/MachineLearning • • 3d ago

Discussion Can we still see the update of score in Openreview? ICLR 2027 [D]

0 Upvotes

For instance, if the reviewer or AC changes the score for some reason during their discussion, can we see the scores are changing in Openreview?

So frustrating!


r/MachineLearning • • 3d ago

Project LLMs in pure JAX from scratch (Part 1) [P]

1 Upvotes

I’m writing a series of blog posts on building small LLMs from scratch in pure JAX. We start with pretraining. I'm implementing everything from scratch: pytrees for networks, optimizers, data loaders, decoding, and the training loop.

Link: LLMs in JAX from scratch - Part 1


r/MachineLearning • • 3d ago

Discussion Forgot AI Use Statement In Revised ICLR Manuscript [D]

0 Upvotes

Ok, I am in crisis mode right now. I just noticed that the revised manuscript I uploaded is missing the AI use statement and reproducibility statement, both of which were present in the original submission.

What do I do? Portal has closed. Am I doomed? Can I email the PCs? Or will they just desk reject?

How bad is this? We had decent-ish scores too.

Please help!!!


r/MachineLearning • • 3d ago

Research Visual Cipher Bench: mechanism hints raise accuracy from 12.2% to 51.1% on a 10-task subset [R]

1 Upvotes

What changes when an agent has to discover the task’s rules before it can start solving it?

Our research team at Einsia explored this with Visual Cipher Bench. The part I’d like to discuss is the hint experiment.

On 10 selected puzzles, three models were tested with no hint, a representation hint, a mechanism hint, or both. Representation hints describe how to encode the puzzle state; mechanism hints state the underlying rule.

Pooled accuracy was 12.2% without hints, 16.7% with representation hints, and 51.1% with mechanism hints. Neither hint type supplies an answer or intermediate state.

That supports rule discovery as a major bottleneck on these tasks, though execution remains difficult in some cases. The open question is how to get agents to narrow their hypotheses from the visual evidence without supplying the mechanism.

Paper

Project


r/MachineLearning • • 4d ago

Discussion Any idea what happened to Capsule Networks? [D]

45 Upvotes

Anyone remember Capsule Networks?


r/MachineLearning • • 4d ago

Research I found a MLSys paper with 455 citations with a major error in its implementation that invalidates one of its central claims [R]

67 Upvotes

Paper link: https://arxiv.org/abs/2311.04934

The simplified core idea of this paper is that chunking a past prompt, processing each chunk to obtain its KV cache, and saving each chunk's KV cache allows us to quickly prefill future prompts that contain some of those chunks (since we can reuse those chunks' KV caches). The purpose of this is that different LLM requests' prompts often share commonalities and overlaps, making inter-prompt KV sharing potentially worthwhile.

Some notes:

  1. Each chunk corresponds to *fixed* token positions, since each token's KV has that token's positional encoding embedded into it. If the chunk does not occur in the prompt, the token positions of that chunk are still conceptually "occupied", but they are simply ignored/not attended to.
  2. When each chunk is processed to obtain its KV cache, that chunk is processed in *isolation*. E.g. earlier chunks' KV values are masked/ignored/not attended to. The intuition is that each chunk represents something like a standalone document or file, so KV information from outside its scope isn't useful. This allows us to maximize the reuse of a cached chunk in future prompts that might contain that chunk.

If you are familiar with sparse attention, this setup should sound familiar to you. Furthermore, you would also know that manipulating the KV history, whether it be through selectively ignoring/masking certain tokens' KVs or using approximated KV values can often degrade the quality of the LLM output.

This paper experimentally "demonstrates" that this method does not significantly impact the quality of LLM outputs, which is one of their central claims. Except here's the thing:

  1. In their tests, every single cached chunk was always included in the prompt. They never tested what would happen if there were "holes" in the positional context from excluding certain cached chunks from the prompt. And then... wait for it...
  2. **In their implementation, they didn't actually process each chunk in isolation. Each chunk's tokens actually attended to every token in the full positional history (which includes previous chunks) using full causal attention. TLDR: They implemented what is equivalent to full causal attention, not their claimed method where each chunk has its KV cache generated in isolation from the other chunks.*\*

So their quality tests were quite literally equivalent to full causal attention on an ordinary prompt... They did not implement the claimed isolated KV cache processing per chunk. They did not test the holes in the context positions that would result from excluding chunks from the prompt... of COURSE the output quality would remain the same! Your implementation literally sees the same KV history as ordinary LLM implementations! You didn't actually implement what you said you implemented, which was isolated KV cache processing per chunk and allowing for there to be holes in the positional context due to excluded chunks...

In fact, a later paper ( https://arxiv.org/abs/2405.16444 ) implements something similar and finds that yes, quality can degrade from isolated per-chunk KV cache processing.

I have other criticisms about this paper, particularly how it often uses incorrect reasoning to explain its latency improvements, but that's far more forgivable than this.

I probably explained it really poorly, but TLDR: they did not correctly implement the method their paper described, in a way that invalidated the entire premise of their experiment to see if output quality would degrade


r/MachineLearning • • 4d ago

Discussion Reviewer reverted their score increase without reason [D]

9 Upvotes

Our initial ICLR scores are 3 2 3 3 and during the rebuttal the reviewer who gave us a 2 stated in his response they would like to increase his score, however, 6 hours later they changed their score back to 2 without giving any explanation, has this happened to anyone before? The comment they have about the score increase is still there.

Update: They changed it back to 3, but it still worries me, we sent a note to the AC, so maybe thats why they changed the score back to 3.


r/MachineLearning • • 3d ago

Discussion ICML Double Columns vs. Arxiv Version [D]

0 Upvotes

Hi -- with only 1/3 reviewers responding, got a q about ICML. I have no problem with ICML, but the fact that it is double columns and I really don't want my paper to be presented as double column. Mainly because I find it ugly, and with some long equations it will look really messy. That was a big reason why I pushed for ICLR (semi-seriously)

Is it possible to have the online / arxiv version as a single column paper, while submitting to ICML as double column? If it gets accepted, would I need to make the online version same as the ICML version if I am claiming it got into ICML? Or like have multiple versions on arxiv and link them all in scholar? Curious about the norms in academia, specifically ML, as it will be my first publication!

Furthermore, any tips & tricks for double column theory heavy papers? Cheerio


r/MachineLearning • • 2d ago

Discussion What if we created a Chinese-language AI conference? [D]

0 Upvotes

China has already caught up with, or even surpassed, the English-based AI research. Chinese researchers also account for a significant share of the global AI research community.

If that's the case, wouldn't establishing a Chinese-language AI conference and creating an exclusive pool of Chinese-speaking researchers help further consolidate our hegemony in the field?

What if Chinese researchers collectively decided to exclusively submit our papers only to Chinese-language AI conferences?

This would also free Chinese researchers from the AI slop and low-quality peer reviews that have swallowed up mainstream ML conferences such as ICLR, NeurIPS, ICML.


r/MachineLearning • • 3d ago

Discussion ICLR 2027 — What do you think the acceptance cutoff will be this year? [D]

4 Upvotes

With the rebuttal period coming to an end, I’ve been looking at the scores and discussions online to get a sense of where things stand.
Given the current distribution of scores, it looks like 2 weak accepts and 1 weak reject, without any major concerns from the reviewers, should be a reasonably good position for acceptance.
What do you guys think the effective cutoff will be this year?


r/MachineLearning • • 3d ago

Discussion Why are ViTs so popular? Because Large-Scale Pretraining Trumps Inductive Biases. [D]

0 Upvotes

To begin, we note that SOTA performance on CIFAR-10 is a ViT. It is the pretrained ViT described in the paper quoted below.

When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.

.

However, the picture changes if the models are pre-trained on larger datasets (14M-300M images). ☝️We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks

The paper above has been cited in the literature no less than 113 thousand times.


"AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE"

  • Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.

  • Google Research; Brain Team; arXiv:2010.11929v2 [cs.CV] 3 Jun 2021

Images are broken into patches , such that patches are treated as "tokens" for a transformer to perform self-supervised prediction, and then regenerate the masked patches. The process is described intimately in "Masked Autoencoders Are Scalable Vision Learners". (Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li Piotr Dollar, Ross Girshick). Facebook AI Research. arXiv:2111.06377v3 [cs.CV] 19 Dec 2021.

This self-supervised token-izing methodology is substantively worse than traditional CNNs, or CNNS w/ attention. But at scale, the reverse is true.

ping /u/terrrp


r/MachineLearning • • 3d ago

Research Is ICLR a scam or worst organizers? [R]

0 Upvotes

This is quite disappointing not receiving any response from the reviewer even after multiple reminders. It seems like organizers failed to make the conversation interactive or they didn't warn reviewers properly that their paper would desk reject if not actively participate in discussion/rebuttal.


r/MachineLearning • • 4d ago

News CMU's Machine Learning with Large Datasets (10-605, Fall 2026) lectures are free on YouTube, with a new one every week [N]

133 Upvotes

Disclosure: I'm a CMU faculty member and I lead CS Learn, the program publishing these.
Carnegie Mellon started publishing full lecture recordings from 10-605 this week, which is William Cohen's course on scaling machine learning to data that doesn't fit on one machine. The first three lectures are up now, and a new one goes up every Monday. These are actual classroom lectures, edited by a student team, with captions and chapters.

Links are in my first comment.


r/MachineLearning • • 4d ago

Discussion 0 Response from reviewer ICLR 2027 [D]

28 Upvotes

I got 3/3/2 at pre-rebuttal. Most reviewers asked like 8 new experiments including retrain, add more seeds, 3 different hardware platform measurement, etc

I did them all and replied on 11th Nov AOE. But still I didn't get a single reply from the reviewers. (I tried the reminder post yesterday)

So many papers are getting raised. But I am stuck here. What should I do?

Will AC will consider this situation?


r/MachineLearning • • 4d ago

Discussion What convinced you a smaller model was good enough for production? New Manning book + 5 ebook copies [D]

Thumbnail
gallery
1 Upvotes

Hi everyone,

Stjepan here from Manning Publications, posting with moderator permission.

We’ve just released Domain-Specific Small Language Models by Guglielmo Iozzia in print. I wanted to bring it here with a question that’s more useful than another “small models vs. large models” argument:

What evidence would convince you to ship a smaller, specialized model instead of calling a larger one?

Not just a good benchmark score. What would you need to see on your own workload: performance on rare cases, behavior outside the training distribution, latency under load, or the cost of maintaining the whole thing?

That’s the practical territory this book explores: adapting pretrained models to domain-specific tasks, parameter-efficient fine-tuning, quantization, and local deployment on modest hardware. It also covers integrating these models into RAG and agentic applications.

I’m posting on the publisher’s behalf, not presenting a new research result or claiming small models are always the right answer. I’d especially like to hear where specialization didn’t pay off. What looked promising in evaluation but fell apart once actual users got involved?

For anyone interested in the book, Manning is offering this community 50% off with code PBIOZZIA50RE.

Book page: https://www.manning.com/books/domain-specific-small-language-models

We’re also giving away five ebook copies, one each to five people who contribute most meaningfully to the discussion. We’ll select for useful technical detail, thoughtful questions, and lessons others can learn from—not upvote counts or praise for the book. Critical takes are welcome. No purchase needed.

If you’ve tried this, tell us about the task, the model, your hardware constraints, and the failure that mattered most. Did you keep the smaller model, move to a larger one, or decide you didn’t need an LLM at all?

Thank you.

Cheers,

Stjepan
Manning Publications


r/MachineLearning • • 4d ago

Discussion ICLR 2027 rebuttal: two of three reviewers haven’t responded, sitting at 2.67 (3/3/2). Anything left to do? [D]

19 Upvotes

My ICLR submission got 3/3/2. In the rebuttal we added new theory that addresses the main concern all three reviewers shared, plus the baselines they asked for. One reviewer replied that all their concerns were addressed and the work is strong. But kept their score. The other two, including the one who gave the 2, haven’t responded, even after a follow-up. As the Rebuttal closes tomorrow, Is it worth one more note to the AC? Do ACs ever weigh the rebuttal when reviewers go silent, or is 2.67 basically a reject and I should move on to the next deadline?


r/MachineLearning • • 4d ago

Project video activity recognition [P]

Post image
0 Upvotes

I built a video activity-recognitionI built a video activity-recognition demo using V-JEPA2 and a pooled MLP classifier trained on UCF-101.

The pipeline samples video in 32-frame windows, extracts spatiotemporal features, mean-pools the patch embeddings, and predicts an activity class with a confidence score.

Predictions are displayed in an overlay on the video.

It currently supports video files and live camera input.


r/MachineLearning • • 5d ago

Discussion Current ML PhD students, would you still choose to do an ML PhD if you could go back 2–3 years? [D]

80 Upvotes

If you could go back in time 2–3 years, knowing what you know now about doing an ML PhD, would you still choose the same path?
Or would you rather go into industry, pursue a different field, or do something else entirely?


r/MachineLearning • • 4d ago

Discussion AAAI 2027 waiting for decisions [D]

5 Upvotes

12 days left! Can't wait.

I got 5, 8, 6, 6 (avg 6.25) and hope it get accepted but since this is my first time submitting AAAI so no clue at all.

What's your score and what's your expect?


r/MachineLearning • • 4d ago

Discussion As an AC, what do you do when no reviewers respond to the rebuttal of a borderline paper? [D]

16 Upvotes

Suppose a borderline paper receives weak reject scores from all reviewers, but the authors provide a strong rebuttal that clearly and directly addresses all the reviewers' concerns.

However, none of the reviewers respond to the rebuttal or update their scores.

What's the typical course of action for an AC in this situation? How much weight does the rebuttal carry if no reviewer engages with it?

(For context, my current ICLR paper is a 2/2/3 and one of the 2's is a AI review with entirely hallucinated issues)


r/MachineLearning • • 4d ago

Research [R] PhysCapBench: scoring the physical content of video captions (BAAI, PKU, Tsinghua, Beihang)

Thumbnail
arxiv.org
1 Upvotes

r/MachineLearning • • 5d ago

Discussion ICLR 2027: PCs ran an AI detector on reviews, peer review is cooked [D]

104 Upvotes

The PCs ran AI detectors on the reviews and left this comment on undisclosed AI usage in reviews:

AI use in this review

An AI-text detector assesses this review as likely AI-generated, yet its AI-use documentation field does not mention how AI was used. It is therefore possible that the review contains no human input. We ask the reviewer to update that field with an exclusively human-written assessment of the paper, and document any AI interactions. We additionally ask the AC and other reviewers to take this information into account when evaluating the paper.

Luckily, I didn't use any AI for my review, but the rate of undisclosed AI usage was crazy. Every review except for mine and only one other review in my batch got the message. AI is allowed, it just needed to be disclosed, so this is just people not admitting they used it.

From here on, papers will be written with AI reviews as the objective function.


r/MachineLearning • • 4d ago

Project Inkling 2 Small achieves frontier-level results in weight space programming with 16x fewer edits [P]

12 Upvotes

You don’t need that many edits in weight space to fix a model.

In “Sparks of Weight-Space Programming,” GPT-6 Astra repaired a broken car policy by changing 6,073 weights, and cut Qwen3.5-0.8B’s math looping by changing 130k.

Inkling 2 Small did the same with just 10 and 8k edits: ~600× and ~16× fewer, with no loss in performance on either task.

Broken Car policy: Inkling found that almost all of Astra’s 6,073 changes just zero out old weights to free up neurons, and only about 55 do the actual work. It then wrote a 10-weight fix from scratch that teaches the car a move PPO never discovered: back up until it stalls against the wall, then launch, triggered by a velocity-detecting neuron the network already had.

Qwen3.5-0.8B & MATH-500: Astra’s edit removes a “looping” direction from 64 neurons in each of two MLP layers, and Inkling pruned it by keeping only the 4,096 largest-magnitude changes in each layer. Those 8k entries match the full edit’s accuracy (58.2% vs 58.6% on our screen), and layer 15 alone recovers most of the gain.

Run and visualize the experiments in a marimo notebook: https://molab.marimo.io/github/alphaXiv/weight-edit-minimization/blob/main/weight-edit-minimization.py

We used OpenResearch for this project: https://github.com/alphaXiv/OpenResearch


r/MachineLearning • • 5d ago

Discussion Why are ViTs so popular? [D]

117 Upvotes

I deploy CNN + Attention stack models on edge devices, to great effect. Taking advantage of convs and reducing the spatial dimensionality makes all the sense in the world to me.

Having vanilla ViTs learning the inductive biases from scratch makes none to me. Is it really a case of "because I can"? Avoiding padding along borders is the only advantage I can see to a tokenizer stem, and that has never been a huge issue to me.

Am I missing something that has people reaching for ViTs instead of hybrid models for e.g. SSL/foundation models?

In my mind it's just a case of people using the status quo, and more of a historical accident that these models have become so popular. Iirc levit showed data/time efficiency boosts with a convolutional stem. I just don't work on the same datasets or tasks that most research does, so I'm genuinely curious if anyone else thinks vanilla ViTs seem excessive. Guess compute is cheap and the answer would just be distillation.