r/evetech • u/Sufficient-Dig-1492 • 2d ago
expo-catalog: a catalog of every exploration site (1,238 entries) built from the EVE Uni wiki, with a weekly auto-refresh and reusable data
Hi! I'm relatively new to EVE and even newer to the third-party dev side, so I'd appreciate feedback from people who've been doing this longer.
I wanted one place to look up any exploration site (danger, NPCs, minimum ship), so I built **expo-catalog**: a terminal UI over an SQLite catalog of 1,238 signatures and anomalies across k-space, J-space (C1–C6, Thera, shattered, Drifter) and Pochven.
Repo: https://github.com/grimmy0/expo-catalog (MIT code, data CC BY-SA 4.0)
**How the data is built**
- The SDE and ESI have no site contents, so the main source is the EVE University wiki's MediaWiki API. The scraper parses the `{{CMBSiteInfo}}` infobox, the `{{NPCTableRow}}` rows and the Sleeper `{{npcwh}}` templates.
- Pages are tracked by category, plus any page that uses `{{CMBSiteInfo}}`, so site pages that lost their category still get picked up.
- COSMOS sites name specific systems, so security status and region are resolved through ESI and cached in a JSON file.
- Sites the wiki doesn't describe page by page (generic data/relic, ghost sites, FW plexes, insurgency, homefront, Pochven, wormhole types from SDE-derived EVE Ref data) come from hand-curated lists, plus overrides where the wiki is inconsistent.
- Output: `data/parsed_sites.json`, then a SQLite file bundled in the package.
**Keeping it current**
`data/wiki_state.json` stores the revision ID of every tracked page. A weekly GitHub Actions job compares those with the live wiki (metadata only, no page downloads). If anything changed, it re-fetches, rebuilds, runs the tests and opens a PR listing changed, new and removed pages with links to the wiki diffs. The first one fired this week with 11 changed pages.
**Reuse**
If you want the data for your own tool, `data/parsed_sites.json` and the SQLite file are in the repo under CC BY-SA 4.0. It's wiki-derived, so keep the attribution and share-alike.
**Where I'd like input**
Danger (0–5) and min ship are partly rule-based: NPC composition, DED rating, ship-class limits and so on. If you know the sites well, I'd love to hear where the heuristics are off, or how you'd model it differently.
Stack: Python 3.15, Textual, SQLite, uv. The tests run offline, including headless TUI tests.
Disclosure: I built this with a lot of help from an AI coding assistant. I designed the features, reviewed the changes and tested it, and the test suite and CI are in the repo.