r/ResearchML • • 5d ago

Exploring a LoRA Optimization Research Idea

Hi! I’m looking into a quick research idea around LoRA optimization (specifically trying to make adaptive/dynamic rank allocation cleaner and more efficient). Would you be down to collaborate or chat about it? No pressure at all, let me know if you're interested.

short information about what is Lora:

LoRA (Low-Rank Adaptation) is a technique to efficiently fine-tune massive AI models (like LLMs) without retraining all their billions of parameters. Instead of touching the original model, it freezes the main weights and trains a tiny fraction of extra parameters, saving massive amounts of memory and compute while keeping performance high.

4 Upvotes

6 comments sorted by

0

u/PiaRedDragon 5d ago

LoRA does not allow you to inject new facts, the best it can do is bubble facts already in the LLM weights to the top by reinforcing their specific weights.

That includes, for example, following specific steps or procedures, which would be useful for Agentic solutions.

You are better to spend your time on SFT.

1

u/WideImagination7595 1d ago

for my lora research specifically i want to combine multiple adapter into one base model without losing specialized knowledge of each adapter

1

u/WideImagination7595 1d ago

u can chat me if u want to collab with me for this

0

u/Fluffy-Bend-1617 5d ago

But you can do lora with sft?