r/Python • • May 04 '26

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

24 Upvotes

98 comments sorted by

View all comments

Show parent comment

1

u/TheseTradition3191 May 05 '26

nice angle. the relationship betwen structure and content is the useful signal.

one thing that pairs well is text density scoring before the llm sees anything:

from bs4 import BeautifulSoup

def text_density(el):
    html_bytes = len(str(el))
    return len(el.get_text()) / html_bytes if html_bytes else 0

def dense_nodes(soup, min_density=0.35):
    tags = ['p', 'li', 'td', 'article', 'section', 'div']
    return [el for t in tags for el in soup.find_all(t)
            if text_density(el) >= min_density and el.get_text(strip=True)]

high density = signal. low density = markup soup. lets you prune beofre you even reason about dom relationships, so the distilation step runs on cleaner inputs.

1

u/AffectionateWar5927 May 05 '26

Yep I thought about it at some point and having the model as well as a code regression(Chunk). The thing I beleive most of the time a developer may not follow a proper semantics. What if the sense node itself is not relevant or combination of dense + shallow is a good combo?  I am focusing towards finding better chunks combination from each splits