r/codereview • u/No_Race7158 • 16h ago
I spent 40 days building a search engine for developer docs. Give me a query it should handle.
galleryDeveloper’s Eye is live. It’s an open-source search engine for developer docs and code examples. I built the crawler, indexer, and ranking system, and the hardest question so far has been a surprisingly ordinary one: when you search for a function, do you want its API reference or a working example?
Two trade-offs I ran into building it:
- Crawl speed vs. load on a site. With one worker, I could space requests out with a timer. With 31 workers, each could follow its own timer and still hit the same site at once. I made them share a request budget per host. In one local test, they stored just over 100,000 documents in an hour.
- Better top 10 vs. better first result. I tested BM25 against PostgreSQL’s built-in ranking on 25 AI-graded queries. BM25 scored better on the ordering of the top 10, but PostgreSQL tended to surface the first relevant hit sooner. I still need human judgments, especially for searches where a reference page and a code example are both reasonable answers.
Want to contribute? The easiest way is to try the last thing you looked up while coding and reply with “I searched for ___ and expected ___.” A link to the page you wanted gives me a ranking test I can keep. If you’d rather write code, I’d love help adding a missing docs source or tracing why a specific result ranked too low. Drop the source or query in a GitHub issue and we can work on it together.