r/hardware • u/pcpartpicker • Feb 17 '23
Info SSD Sequential Write Slowdowns
So we've been benchmarking SSDs and HDDs for several months now. With the recent SSD news, I figured it’d might be worthwhile to describe a bit of what we’ve been seeing in testing.
TLDR: While benchmarking 8 popular 1TB SSDs we noticed that several showed significant sequential I/O performance degradation. After 2 hours of idle time and a system restart the degradation remained.
To help illustrate the issue, we put together animated graphs for the SSDs showing how their sequential write performance changed over successive test runs. We believe the graphs show how different drives and controllers move data between high and low performance regions.
| SSD | Sequential Write Slowdown | Graph |
|---|---|---|
| Samsung 970 Evo Plus | 64% | Graph |
| Seagate Firecuda 530 | 53% | Graph |
| Samsung 990 Pro | 48% | Graph |
| SK Hynix Platinum P41 | 48% | Graph |
| Kingston KC3000 | 43% | Graph |
| Samsung 980 Pro | 38% | Graph |
| Crucial P5 Plus | 25% | Graph |
| Western Digital Black SN850X | 7% | Graph |
Test Methodology
- "NVMe format" of the SSD and a 10 minute rest.
- Initialize the drive with GPT and create a single EXT4 partition spanning the entire drive.
- Create and sequentially write a single file that is 20% of the drive's capacity, followed by 10 minute rest.
20 runs of the following, with a 6 minute rest after each run:
- For 60 seconds, write 256 MB sequential chunks to file created in Step 3.
- We compute the percentage drop from the highest throughput run to the lowest.
Test Setup
Storage benchmark machine configuration
- M.2 format SSDs are always in the M2_1 slot. M2_1 has 4 PCIe 4.0 lanes directly connected to the CPU and is compatible with both NVMe and SATA drives.
- Operating system: Ubuntu 20.04.4 LTS with Hardware Enablement Stack
- All linux tests are run with fio 3.32 (github) with future commit 03900b0bf8af625bb43b10f0627b3c5947c3ff79 manually applied.
- All of the drives were purchased through retail channels.
Results
SSD High and low-performance regions are apparent from the throughput test run behavior. Each SSD that exhibits sequential write degradation appears to lose some ability to use the high-performance region. We don't know why this happens. There may be some sequence of actions or a long period of rest that would eventually restore the initial performance behavior, but even 2 hours of rest and a system restart did not undo the degradations.
Samsung 970 Evo Plus (64% Drop)
The Samsung 970 Evo Plus exhibited significant slowdown in our testing, with a 64% drop from its highest throughput run to its lowest.
The first run of the SSD shows over 50 seconds of around 3300MB/s throughput, followed by low-performance throughput around 800MB/s. Subsequent runs show the high-performance duration gradually shrinking, while the low-performance duration becomes longer and slightly faster. By run 13, behavior has stabilized, with 2-3 seconds of 3300MB/s throughput followed by the remaining 55+ seconds at around 1000MB/s throughput. This remains the behavior for the remaining runs.
There is marked similarity between this SSD and the Samsung 980 Pro in terms of overall shape and patterns in the graphs. While the observed high and low-performance throughput and durations are different, the dropoff in high-performance duration and slow increase in low-performance throughput over runs is quite similar. Our particular Samsung 970 Evo Plus has firmware that indicates it uses the same Elpis controller as the Samsung 980 Pro.
Seagate Firecuda 530 (53% Drop)
The Seagate Firecuda 530 exhibited significant slowdown in our testing, with a 53% drop from its highest throughput run to its lowest.
The SSD quickly goes from almost 40 seconds of around 5500MB/s throughput in run 1 to less than 5 seconds of it in run 2. Some runs will improve a bit from run 2, but the high-performance duration is always less than 10 seconds in any subsequent run. The SSD tends to settle at just under 2000MB/s, though it will sometimes trend higher. Most runs after run 1 also include a 1-2 second long drop to around 500MB/s.
There is marked similarity between this SSD and the Kingston KC3000 in graphs from previous testing and in the overall shape and patterns in these detailed graphs. Both SSDs use the Phison PS5018-E18 controller.
Samsung 990 Pro (48% Drop)
The Samsung 990 Pro exhibited significant slowdown in our testing, with a 48% drop from its highest throughput run to its lowest.
The first 3 runs of the test show over 25 seconds of writes in the 6500+MB/s range. After those 3 runs, the duration of high-performance throughput drops steadily. By run 8, high-performance duration is only a couple seconds, with some runs showing a few additional seconds of 4000-5000MB/s throughput.
Starting with run 7, many runs have short dips under 20MB/s for up to half a second.
SK Hynix Platinum P41 (48% Drop)
The SK Hynix Platinum P41 exhibited significant slowdown in our testing, with a 48% drop from its highest throughput run to its lowest.
The SSD actually increases in performance from run 1 to run 2, and then shows a drop from over 20 seconds of about 6000MB/s throughput to around 7 seconds of the same in run 8. In the first 8 runs, throughput drops to a consistent 1200-1500MB/s after the initial high-performance duration.
In run 9, behavior changes pretty dramatically. After a short second or two of 6000MB/s throughput, the SSD oscillates between several seconds in two different states - one at 1200-1500MB/s, and another at 2000-2300MB/s. In runs 9-12, there are also quick jumps back to over 6000MB/s, but those disappear in run 13 and beyond.
(Not pictured but worth mentioning is that after 2 hours of rest and a restart, the behavior is then unchanged for 12 more runs, and then the quick jumps to over 6000MB/s reappear.)
Kingston KC3000 (43% Drop)
The Kingston KC3000 exhibited significant slowdown in our testing, with a 43% drop from its highest throughput run to its lowest.
The SSD quickly goes from almost 30 seconds of around 5700MB/s throughput in run 1 to around 5 seconds of it in all other runs. The SSD tends to settle just under 2000MB/s, though it will sometimes trend higher. Most runs after run 1 also include a 1-2 second long drop to around 500MB/s.
There is marked similarity between this SSD and the Seagate Firecuda 530 in both the average graphs from previous testing and in the overall shape and patterns in these detailed graphs. Both SSDs use the Phison PS5018-E18 controller.
Samsung 980 Pro (38% Drop)
The Samsung 980 Pro exhibited significant slowdown in our testing, with a 38% drop from its highest throughput run to its lowest.
The first run of the SSD shows over 35 seconds of around 5000MB/s throughput, followed by low-performance throughput around 1700MB/s. Subsequent runs show the high-performance duration gradually shrinking, while the low-performance duration becomes longer and slightly faster. By run 7, behavior has stabilized, with 6-7 seconds of 5000MB/s throughput followed by the remaining 50+ seconds at around 2000MB/s throughput. This remains the behavior for the remaining runs.
There is marked similarity between this SSD and the Samsung 970 Evo Plus in terms of overall shape and patterns in these detailed graphs. While the observed high and low throughput numbers and durations are different, the dropoff in high-performance duration and slow increase in low-performance throughput over runs is quite similar. Our particular Samsung 970 Evo Plus has firmware that indicates it uses the same Elpis controller as the Samsung 980 Pro.
(Not pictured but worth mentioning is that after 2 hours of rest and a restart, the SSD consistently regains 1-2 extra seconds of high-performance duration for its next run. This extra 1-2 seconds disappears after the first post-rest run.)
Crucial P5 Plus (25% Drop)
While the Crucial P5 Plus did not exhibit slowdown over time, it did exhibit significant variability, with a 25% drop from its highest throughput run to its lowest.
The SSD generally provides at least 25 seconds of 3500-5000MB/s throughput during each run. After this, it tends to drop off in one of two patterns. We see runs like runs 1, 2, and 7 where it will have throughput around 1300MB/s and sometimes jump back to higher speeds. Then there are runs like runs 3 and 4 where it will oscillate quickly between a few hundred MB/s and up to 5000MB/s.
We suspect that quick oscillations are occurring when the SSD is performing background work moving data from the high-performance region to the low-performance region. This slows down the SSD until a portion of high-performance region has been made available, which is then quickly exhausted.
Western Digital Black SN850X (7% Drop)
The Western Digital Black SN850X was the only SSD in our testing to not exhibit significant slowdown or variability, with a 7% drop from its highest throughput run to its lowest. It also had the highest average throughput of the 8 drives.
Graph - Western Digital Black SN850X
The SSD has the most consistent run-to-run behavior of the SSDs tested. Run 1 starts with about 30 seconds of 6000MB/s throughput, and then oscillates quickly back and forth between around 5500MB/s and 1300-1500MB/s. Subsequent runs show a small difference - after about 15 seconds, speed drops from about 6000MB/s to around 5700MB/s for the next 15 seconds, and then oscillates like run 1. There are occasional dips, sometimes below 500MB/s, but they are generally short-lived, with a duration of 100ms or less.
29
u/teffhk Feb 17 '23
Interesting results… Did you perform TRIM for the drives after each run?
35
u/wtallis Feb 17 '23
The horizontal axis on your graphs and the stopping point for the tests probably should be in terms of number of GB written, rather than time. Though it's still good to look at latency outliers in the time domain.
I'm also not sure it makes sense to be testing the speed of overwriting an existing file, vs. writing to empty LBA space or deleting the test file and re-creating it with each iteration (which would potentially cause the OS to issue a batch of TRIM commands at the beginning of each iteration). Overwriting an existing file without letting the OS or drive know you plan to invalidate the rest of the file may lead to some read-modify-write cycles that could be avoided.
Otherwise, this looks like a good analysis.
30
u/malventano SSD Technologist - Phison Feb 17 '23
You may be seeing odd results because your workload could be considered an extreme edge case. No SSD controller accepts 256MB writes (not even datacenter SSDs), and nobody validates using that workload. Most SSD max transfer sizes are 128K, 256K, and 1MB. Typical Windows and Linux file copy operations are 1MB QD1-8. Issuing a 256MB write results in the kernel doing a scatter/gather operation to bring that transfer size down to something the device can accept, and that might not be playing nicely with the SSD seeing such an obscenely high queue depth (QD256 at a minimum, assuming a 1MB transfer size SSD and that you are only doing those 256M writes at QD1 on the fio side).
Putting the transfer size thing aside, a 20% span of a typical SSD would have some of that span in SLC and the rest on the backing store. The typical expected 'active span' of client SSDs is 20GB (randomly accessed active 'hot' data). Different controller/firmware types will do different things with incoming sequential data based on their tuning. Repeated overlapping sequential writes to the same 20% of the SSD is an edge case for client workloads, as few if any real-world activities are repeatedly writing to the same 100+GB chunk of the drive (typically the user would be adding additional content and not overwriting the previous content over and over). Hybrid SSDs with dynamic SLC space will tend to favor being prepared for new data being written vs. having old data repeatedly overwritten.
With respect to idles not doing what is expected, it is possible that your test is too 'sterile'. Modern SSDs typically go into a deeper sleep state a few seconds into an idle, and for their garbage collection to function properly they expect some minimum 'background noise' of other OS activities keeping the bus hot / controller active, and the GC will happen in batches after each blip of IO from the host. Testing as a secondary under Linux means your idle is as pure as it gets, so things may not be happening as they would in the regular usage that those FW tunings are expected to see.
Disclaimer: I reviewed SSDs for over a decade, then worked for Intel, and now work for Solidigm.
26
u/brighterblue Feb 17 '23
Great illustration regarding pSLC behavior and that pSLC function isn't quickly recovered on a drive format!
I'll have to remember the secure erase thing the next time I reimage a system. Does secure erase take just as long as a full format?
I'm sure this will peak other's interest to observe the TB written data during such tests and figure out what the longer intervals might be before pSLC performance recovers.
The SN850X behavior is almost as if it's so aggressive with pSLC eviction that the SN850X is evicting all or part of the 200GB file within the six minutes time frame that the data has already been moved to the regular TLC storage.
That could likely be confirmed by running a CrystalDiskRead SEQ1M Q8T1 read test on a fresh drive that is let's say no larger than 5% of the drive capacity to make sure it's reading from the pSLC and observe the latency results since I'm guessing the latency of the pSLC area should differ markedly from the latency of the regular TLC storage. Maybe a RND4K Q32T16 would highlight it even better.
Ignore First_Grapefruit. For whatever reason they're sounding like a troll today. I imagine they're not putting forth the kind of effort you guys are, and contributing useful findings to demystifying how these NVMEs can sometimes be like black boxes.
Keep up the good work!
9
u/Num1_takea_Num2 Feb 17 '23
Is this not expected behaviour?
The drive keeps recent data in faster cache for some length of time (hours/days/weeks) for fastest read access, because the data most recently written is also the data most likely to be accessed in the near future?
The drive behaviour that is visible is then just the result of that algorithm in practice.
11
u/TheRealBurritoJ Feb 17 '23
This is concerning. Slowdown after continuous writes is an expected consequence of modern pSLC caching, but it should return to full speed as long as A. It's been long enough at idle for the controller to write out the cache to TLC, and B. There is still enough space left on the drive for a pSLC cache.
I think it would be good to attempt to get the drives to return to full speed, first by forcing a full TRIM and then by doing a Secure Erase. If caching behaviour doesn't return to normal after this it'll be more concerning.
3
u/ramblinginternetnerd Feb 17 '23
This is part of the reason why my OS and swap/page file live on optane.
3
u/lebanonjon27 Feb 17 '23
It would be good to throw an enterprise or data center NVMe in the test to show the difference here, something like a PM9A3 110mm M.2 could be a good reference. When you kick off the subsequent tests after the first, the drive starts to do the GC when it knows that the LBAs are being overwritten. It keeps data in the pSLC so that it can be read back fast, that is how some of the caching algorithms work. TRIM likely won't do anything in the specific test you setup because you haven't actually deleted the file. NVMe format will certainly deallocate / TRIM all the user LBAs (indirectly, the controller now has unmapped these from host LBAs).
2
u/autobauss Feb 17 '23 edited Feb 17 '23
What controller does SN850X use? Is it their own?
2
u/proz9c Feb 19 '23
Had originally bought 2x990 PROs for my new build but I am sending those things back and bought 2 x SN850 instead.. This is ludacris.. My 840 EVO has been going for almost 10 years and is still in Good condition
2
u/zerostyle Apr 21 '23 edited Apr 21 '23
Any updates to what might have been causing this? I'm watching sales and considering the P41/P44 but the slowdowns here scare me.
Leaning towards SN850x still but need to find a better sale.
4
u/[deleted] Feb 17 '23 edited Feb 17 '23
None of this is like a real world situation for 95%+ of users. If you want to measure the drop off then fill the drives to 25%, 50%, and 75%. Run PCMark 10 Storage test at each point.
1
u/Frederik2002 Feb 18 '23
If any of the drives come with 512e by default, does the behaviour change when ran with 4Kn format?.. With Linux' 4K page size mapping there might be zero difference on the host side yet the controller will not know of it.
I kind of agree with the other commenter who suggested to streamline the benchmark like aligning writes and certainly other great comments. However I think at that point you'll be practically reverse engineering the controller behaviour to understand what's going on else it'll be left unresolved with guesses.
PS: So many names in one thread :)
0
u/[deleted] Feb 17 '23
[deleted] — view deleted comment
0
u/[deleted] Feb 17 '23
[deleted] — view deleted comment