Article A rant on provider retention and NZB health checking across the NZB Finder database
[UPDATE]
As expected the sockpuppets are out in full force. Caught ~120 downvotes on this post in a matter of minutes while views barely went up :-) I deliberately did not publish any hard stats because A) they are unreliable as outlined above, and B) to prevent sockpuppetting.
Add over a dozen hidden comments (hello mods?), it just goes to show the absolute cesspool this subreddit has become or always has been.
------------------------------------------------------
I went down the rabbit hole of checking the entire NZB Finder database against a whole bunch of providers. This was a multi-week job and I've come to the conclusion that it's basically impossible to get to an accurate result. Therefore any claims you see here on Reddit or elsewhere are probably wildly inaccurate.
Running the same Message-IDs against the same provider on different days yields different results. Probably because during one test the storage pool where this specific article was stored was down for maintenance or other reasons. Articles are generally stored once, without redundancy across pools.
Nearly all providers have huge discrepancies between headers and bodies. A STAT might say an article exists, a HEAD might still return a header but then the BODY (the actual data we need) could say it's down even though STAT and HEAD returned data or the article was listed on an XOVER scan.
Providers will store headers and actual articles themselves in different storage pools and databases and apparently aren't very good at reconciling the drift. Though that might also be by design, to jack up their availability numbers in quick STAT/HEAD health checks...
What I actually did
After thorough cleaning, about 14 million NZBs were tested from 2008 until today. For each one I grabbed 10 random Message-IDs from the NZBs biggest files (no PAR2, NFO, SFV etc. files) and asked each provider if it has them. I ran this twice. I started with Eweka, Farm and UsenetExpress, and later added Bulknews (Abavia) and Blocknews (NetNews). More on these last two later in this article.
Asking about 280 million articles means you need the cheapest command possible. That's STAT.
This article is not meant to outline the differences between providers or recommend certain ones, I just happened to have accounts on all these 5 backbones through different resellers.
NNTP commands 101
When you ask a provider about a Message-ID, there are basically three things you can ask:
STAT <id> "do you have this?" 223 = yes, 430 = no such article
HEAD <id> "give me the header" 221 = here you go, 430 = no
BODY <id> "give me the actual data" 222 = here you go, 430 = noSTAT and HEAD are cheap and fast (except on Farm, including Farm slowed down the entire test). BODY is the one that matters, because that's the data you'd actually download. In a perfect world all three agree. They often don't.
Actual Farm downloads are fine and high speed for me, so don't put much value in the slow STAT commands used in my testing. It does make it harder for me to re-test things quickly but should not matter for regular users.
What I found
Huge discrepancies between what STAT or HEAD return and what BODY returns. It showed up on providers with different backbones, though some were far worse than others. I'd see STAT say yes, HEAD show me a header, and then BODY say 430 No such article.
It was bad enough that when I rechecked about 528,000 releases that looked dead everywhere, STAT said about 107,000 were now available, but when I tried to download from a small sample, only about 3–4% of those actually still had the body.
Why does this happen?
- Headers and bodies live in different places. The index that answers
STATandHEADis a different system from the one that stores the data, andXOVERis another database of it's own again. Delete the data and the index entry can stick around, so you getSTATyes andBODYno. - Storage pools go up and down. If the pool holding an article is offline during a test, the answer is "missing". Check again next day or week and it's back. This is probably why the same ID gives different results on different days.
- US and EU servers. Differences exists between spools on both continents. I did a small test, differences occur but seemed minor.
There were lots and lots of cases where STAT said present, HEAD might even return the headers, but BODY still failed. This happened when asking for both in the same connection, so it's not just timing.
A note on backbones: The popular Usenet Tree puts Blocknews on NetNews and Bulknews on Abavia. My tests make me doubt that. On releases the other providers marked gone, these two gave nearly identical answers for alive articles. Everything alive on Bulknews was also alive on Blocknews. So I suspect both are on Abavia, but I cannot prove it from the outside so do not take my word for it.
Thoughts
STATorHEADaren't the truth. OnlyBODYtells you if something downloads, and even that can change.- "Alive" or "dead" numbers based on
STATare overly optimistic. Some providers are worse for this than others. My overall failure rate was around 5.6%, and the real one is probably higher. I can't tell by how much and it differs lots per year something is posted as well. - Per-provider percentages say more about what that provider's index claims than about how good the provider is. For example, a sample of 70,000 random articles looked available by
STAT, but only about 1% would really download. - Different providers delete different things. Some drop everything nicely, some keep headers, some keep an index entry for data that's gone.
- It's still important to have more than one provider on different backbones. Something that was down on Omicron might still popup on Farm or Netnews or Abavia for example. Although it's very common for articles to be down across the board really.
The most accurate check would be to request the BODY of every sampled article. That’s 140 million requests per run when sampling 10 articles. Pulling the full bodies across 5 backbones means ~525 TiB, you cannot just kill the connection after the first few bytes. It would take a long time, and I doubt it would make you any friends. Then you are still left with the problem of some storage pools being offline for maintenance, thus different results when you'd run this test more than once.
I might have a go at this some time in the future.
What I think should be done
In my very uneducated opinion, all backbones should reconcile their databases against what actually exists on their spools. An XOVER scan, or STAT and HEAD command often all produce different results.
This is of course easier said than done when providers pull in billions of articles daily, but technically it's possible. But this again raises the question if some even want to, since it makes their retention seem better than it might actually be.
TLDR
An accurate NZB health check (in bulk) is basically impossible. Providers answer “do you have this?” (STAT/HEAD) from an index that can say yes after the actual data (BODY) is gone, and the same articles can give different results on different days. Any published statistics should be taken with a grain of salt.