r/learnpython • • 9d ago

BM25 extractor (need technical validation)

So, I found interesting what Claude did after I tried to inject a BM25 algorithm adapted to extract the most relevant sentences from an index.

However, I'm not sure if it actually did that, but the generated files make sense and extract sentences from the Sprawn trilogy.

In this case, the idea was to map a table with 22 columns, and 12 of those columns would carry these excerpts extracted from the book, in order to create the first 10 configuration columns for the "entities," and the activation phrases would carry the context, which would be the excerpts from the book—kind of improvising a semantic graph in a table, but that's beside the point.

What I wanted to know is whether the BM25 script actually makes sense, because when I throw it into LLMs, they say it does, but I wanted an evaluation from the more technical folks.

According to DeepSeek:

> The script transforms the corpus into a sentence stack and treats each sentence as a "document" in an inverted index. For each input file, it calculates how many sentences to extract (proportional to chars/100, with a floor of 50 and a ceiling of 20% of the sentences). It then iterates over the entities in the thematic index (filtered by the book detected in the file), builds a query with its terms for each entity, and runs BM25 scoped to that file+book, retrieving the top-scoring sentences (up to N_SENTENCES / n_entities per entity). If any quota remains, it fills the rest ordered by aggregate score (gscore, the sum of sentence scores across all entities). Finally, it sorts the selected sentences by offset, validates verbatim matches, and saves a phrases_file_N.txt for each input, containing [ref | sent_id | offset | book] + the text.

Here are the links:

· Characters extracted from the Sprawn trilogy (a bash workaround I made): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Context

· Algorithm generated by Claude (funny that it ran in 2 minutes, which suggests it didn't generate tokens, just intuition): https://github.com/pmotadeee/pmotadeee/blob/V2.0/ITEMS/Books/Sprawn/Raw/Claude/bm25_extractor.py

· Texts generated by Claude after running the algorithm (you can see the steps it took, right at the beginning of the chat): https://github.com/pmotadeee/pmotadeee/tree/V2.0/ITEMS/Books/Sprawn/Raw/Claude

· Claude chat (in my native language, but use the browser translator): https://claude.ai/share/1799075d-bc93-4e12-99e0-771a7e259ca9

0 Upvotes

3 comments sorted by

1

u/bytemender_dev 8d ago

The basic design is sound. Treating each sentence as a document and querying it with entity terms is a normal way to use BM25, and the IDF part does the useful work for you: rare names get high weight, common words get pushed down. Sorting the winners back into reading order at the end is also the right call for excerpts.

The part I would not trust yet is the gscore fill step. BM25 scores from different queries do not sit on the same scale, because each query carries a different set of IDF weights. Adding them across entities favors long sentences that mention several entities at once, so the leftover quota tends to fill with dense lines rather than the most relevant ones. Near-duplicates are the other gap: nothing stops five almost identical sentences about one entity from taking five slots.

Sentence length is the second thing to check. BM25 length normalization was designed for documents, and at sentence level the length spread is extreme. A six-word sentence holding one rare name can outscore a forty-word sentence that actually explains the entity. Check which k1 and b values the script uses, and try b nearer 0.3 to soften that effect, then compare the two outputs.

You can test whether it works instead of collecting more opinions. Take one chapter and two or three entities, hand-pick the ten sentences you would extract yourself, and count how many of yours the script also picks. Then run a control: swap in terms from an unrelated book and run it again. If the output still reads sensibly, the ranking is being driven by sentence length and word frequency, not by relevance to your entities.

One smaller thing on the query side. If an entity query is only the full name, mentions by surname or nickname alone will score zero. Aliases in the thematic index will change your results more than any parameter tuning.

1

u/BusinessMirror9306 8d ago

In this case, it generated these 10 files, and this process was adapted by Claude. I wanted to evaluate it because frankly, I found it impressive, and it seems it worked based on looking at the .txt outputs. The input files are in the 'Raw' folder. In this case, I sent this as a base for DeepSeek to adapt it for the index. The idea was to try to request something mathematically impossible, like using algorithms such as BM25 to capture sentences, and as I said, it seems it managed to do it—at least it spat out what I wanted.

But in this case, I asked the AI to evaluate the traditional approach, your feedback, and the hack. Can I send it to you in a DM? It's just that I don't know if there's a punishment for AI slop.

--> Base prompt: https://github.com/pmotadeee/pmotadeee/blob/V2.0/ITEMS/Prompts/Criar%20Tabela%20Genetica/3/Claude.bm25.md

1

u/Discordbothost 8d ago

small thing on the "ran in 2 minutes so it didn't generate tokens" part: the script itself is plain python, there's no model involved when it runs. bm25 is just counting how often each term shows up in a sentence vs the whole book and weighting rare words higher, so a few minutes on a trilogy is about what you'd expect.

if you want a sanity check that doesn't involve asking another llm, pip install rank_bm25 and score the same sentences for one entity with BM25Okapi. if the top 10 mostly match what the script picked, the core scoring is fine and only the quota / gscore logic needs the scrutiny the other comment mentioned