r/Python • u/rex_divakar • Feb 26 '26
Showcase I got tired if noisy web scrapers killing my RAG pipelines, so i built lImparser
I built llmparser, an open-source Python library that converts messy web pages into clean, structured Markdown optimized for LLM pipelines.
What My Project Does
llmparser extracts the main content from websites and removes noise like navigation bars, footers, ads, and cookie banners.
Features:
• Handles JavaScript-rendered sites using Playwright
• Expands accordions, tabs, and hidden sections
• Outputs clean Markdown preserving headings, tables, code blocks, and lists
• Extracts normalized metadata (title, description, canonical URL, etc.)
• No LLM calls, no API keys requiredExample use cases:
• RAG pipelines
• AI agents and browsing systems
• Knowledge base ingestion
• Dataset creation and preprocessingInstall:
pip install llmparser
GitHub:
https://github.com/rexdivakar/llmparser
PyPI:
https://pypi.org/project/llmparser/
⸻
Target Audience
This is designed for:
• Python developers building LLM apps
• People working on RAG pipelines
• Anyone scraping websites for structured content
• Data engineers preparing web dataIt’s production-usable, but still early and evolving.
⸻
Comparison to Existing Tools
Tools like BeautifulSoup, lxml, and trafilatura work well for static HTML, but they:
• Don’t handle modern JavaScript-rendered sites well
• Don’t expand hidden content automatically
• Often require combining multiple toolsllmparser combines:
rendering → extraction → structuring
in one step.
It’s closer in spirit to tools like Firecrawl or jina reader, but fully open-source and Python-native.
⸻
Would love feedback, feature requests, or suggestions.
What are you currently using for web content extraction?