"A2 promises even better performance with 4000/2000 read/write IOPS minimum but there's a problem since as outlined by the SD Association A2 cards show "much higher performance than A1 performance by using functions of Command Queuing and Cache".
Cache and Command Queuing require host (driver) support since the host needs to activate those new features first. The cache feature on A2 rated cards makes use of volatile RAM on the card requiring the host to learn new commands to issue flushing the cache (involving the risk of data losses -- for details see especially chapter 4.17 in Physical Layer Simplified Specification 6.0)"
So is there any kind of device able to use A2 cards at the rated performance yet? I can't get anything better on my Mac with multiple readers (USB 3.0 and 3.1), and I'm guessing if I tried on the reader in my 1 year old Dell XPS, I'd get the same.
What good is a performance designation (A2) if no device in the real world supports it?
I imagine Android would benefit the most (and phones also don't suffer from sudden power outages, which is the biggest risk with enabling this), so someone may try looking if mmc core in androind kernel has patches to support this.
Flagships have been skyrocketing in price and are losing marketshare to the midrange segment, which almost always has removable storage, and often has a swappable battery.
But in all those cases (I have used most of them), there was _some_ way a consumer could acquire hardware (and drivers) to use the tech. It might've been expensive, but I could go out and buy an SATA card, or SCSI 2, or whatever.
With A2, so far I have not seen any way any consumer (or anyone, really) could achieve 2000/4000 IOPS.
That would be like selling a SATA 2400 drive for 2x the cost of an SATA 600 drive... but not selling any kind of controller that could use more than SATA 600 bandwidth.
It should be possible to patch the kernel. If I had an A2 card, I'd try. It's probably just a question of detecting card features on card plug-in and sending a command to enable the cache.
Spec/code is open. I'm just sort of surprised noone tried it yet.
You will probably need a true SD/MMC controller to use any "new" commands, such as a smartphone or embedded target (such as the RPi). USB or other bridges used on notebooks are unlikely to have support for it (especially as they convert MMC/SD to a USB-storage class device)
Assuming the cards with the A2 designation do actually support the claimed speeds, support will (eventually) be added, and the faster performance will be observed, on the Pi and other host devices.
I appreciate you doing these tests, this (and your last article) instantly answers my question about "how will this perform in one of my rpis?". Please keep doing them as new cards(with their marketing claims) are introduced!
How about lifespan? I fail to understand why SD card have such a short lifespan, and it's the sole reason I won't acquire an RPi.
It seems that the only think that is lacking on RPi is flash memory storage. Most smartphones have a lot of it, and it seems that 1GB of storage would be enough for the RPi and would not necessarily be so expensive.
I can't help you with the why some SD cards are short lived, but if you want a longer lasting SD card, try the "endurance" versions. Most of the big brands have a version. They are a bit more expensive, but promise 25x the write life of a standard SD.
They are typically marketed toward video surveillance systems and dash cams. I'm not sure how the rest of a Raspberry Pi's performance would be effective.
FWIW, only noname and Sandisk cards have given up the ghost on me. Samsung Pro & Evo cards have been doing just fine. Other than that, you just need to worry about getting proper power. Just use the official power supplies, and you're fine.
Also eMMC chips are no magic. There are crappy ones and good ones. Guess which ones you're going to get for little money?
Proper power is essential. A phone charger is not a power supply even though it works. I suspect a lot of SD cards are getting undeserved blame from a PSU issue.
This is my experience. The only time I had an issue with corruption (using over 60 different microSD cards through 7 years and maybe 50 or so different Raspberry Pis) was when I used a little .5A microSD AC adapter that came with some cheap battery powered device that broke.
I have not had any problems when using a reliable power supply. I'm guessing the 'endurance' or 'industrial' cards some people claim are lifesavers are just doing something to guard against flaky power input when doing reads and writes, so it masks the issue.
Not having a proper power supply is my only pi related regret. Thing would freeze every once in a while. That since stopped happening, and the only change in-between (beside updates) was the initial used charger dying.
I have had SanDisk, Kingston, and Sony cards die on me in a variety of devices from Pis to GoPros. Sonys were the fastest to die but retained read capability, where as the SanDisk just stopped functioning completely.
My most recent additions are a Samsung Evo card and a Samsung Endurance. I am curious to see which dies first.
The Pi doesn't necessarily need an SD card. I'm running it on NFS, booted over the network. Works just fine. No need to worry about burning out the SD card.
I believe the RPi can boot directly from the network, but it does require a one-time config change that requires booting from the SD card once. After that, the SD card isn’t required for future reboots.
> ... and it seems that 1GB of storage would be enough for the RPi and would not necessarily be so expensive.
Enough for what? A typical Raspbian install not takes a bit more than 4GB. I don't think that the Lite version (no GUI or applications) would fit either. I'd suggest 8GB, if that's a convenient size. As flash density increases, more and more goes on a single chip and that's where production is going. Consider the price of SD cards or USB sticks. Below about 16 or 32GB they don't get much cheaper and I suspect is is for this reason.
And IAC, the RPi foundation is working hard to keep the price of the board at $35US so adding something like this probably means something else would have to be eliminated.
In my experience this isn't caused by exhausting the endurance of the flash. It seems like a lot of SD cards have really buggy firmware. And that Linux running on a RPi is particularly prone to triggering these bugs. Industrial SD cards seem to hold up better.
Early iterations of the RPi boards apparently had some bad design choices in the power circuitry, which makes the problem worse.
I have three RPis in abusive conditions, and they haven't had a single SD card issue. Two RPi 3b's running as a car entertainment system, with no form of battery backup/safe shutdown. They are on the ignition power, and once the car turns off, they hard shut down. These have been in place for nearly 2 years with the same Samsung Evo 16GB cards.
The other is attached to a digital signage with less frequent hard shut downs, but still has at least one per week, and been in place since late 2017. Also running a Samsung Evo card (I think it's a 32GB since that's what I had on hand).
Both have high quality power supplies, which may help. The in-car systems have Anker car USB chargers that have been de-cased and soldered in place, while the signage one has an Anker wall charger and a standard micro-b cable.
What's the write cycle like? Seems like those would mostly be read only.
The cards that die quickest for me are either written excessively like a GoPro or Pi based data loggers written constantly. The GoPros are just burning out the cards but the data loggers corrupt the disk images during unexpected power loss.
FWIW I have several Pis running 24x7 and have not had a single SD card failure. I usually buy high end cards but I also buy cheap cards (4GB cards)and a few Micro-Center house brand, for which I don't know the quality.
I've had one that has been running years, I think. Drives a monitor, looping a video where my wife works.
The only SD card failure I have experiences was in a camera.
just a guess but s/he may have been referring to just putting root filesystem on a USB drive, but still requiring the bootloader (and possibly kernel?) to be on sd card. It's a one-time "hit" that is completely negligible.
That's the present situation and I've used it before. Some recent Pis can boot directly from a USB drive without the need for a bootloader on the SD card. That's still in the works for the Pi 4 as I understand, but network boot (PXE?) will come before that.
The major problem is not disk write speed, which we can all agree sucks, but longevity and lifespan when used as a system boot disk and also containing /var and similar. The write wear leveling on a u3 microsd card is nowhere near as good as even a $50 consumer class ssd.
Small single board computers should move towards m.2 2280 nvme drive support, or at least one sata 6 gbps interface.
The Pi steals the lion’s share of the attention the in the cheap SBC space, but there are plenty of alternatives that have better storage solutions out there. I too grew fed up of SD card issues.
I always see the argument “but the Pi has the best community support!” trotted out, and I don’t disagree. However, it’s all normally just sane Linux distros on the Pi at the end of the day. Whatever instructions/advice someone has written for a given Pi project will likely be applicable on a non-Pi board anyway.
I think it depends a lot on your use case. If you just want a dirt cheap HTPC, then yes, there are lots of SBCs that will fit the bill, but the RPi also has a hardware add-on ecosystem around its particular camera and GPIO/SPI/I2C/I2S availability, and even though lots of other boards have those things available and broken out too, you're going to have to be willing to do a lot more leg work to get it all hooked up properly and working for a different SoC.
SD card performance on the Pi <4 is so poor (and the bandwidth is shared with the USB controller anyway, I think) that you might as well use an attached USB hard disk; even for booting.
Jeff's benchmarks are accurate in my experience. If you try saving single images from a camera at even 5-10fps, the card quickly bottleneck and you start dropping frames. If you buffer and store (in my case 16-bit raw), you can essentially saturate USB bandwidth if you're also using a USB camera.
We've got a few Pi's in office "production" (used as dashboards) and the best solution we've found it to use an SSD drive via USB. Something like a 128GB SSD drive is cheap enough, although it's about as expensive as the Pi itself.
Yeah I'm always fighting both write speed and network speed limitations when I do timelapse projects with my Pi Zeros. Hopefully the I/O improvements from the 4 trickle down to the Zero form factor sometime soon!
There are SD cards rated for "high endurance" for example https://www.amazon.com/gp/aw/d/B07B984HJ5 is designed for continuous 24/7 writes such as surveillance cameras. These should have great longevity when used in a RPi.
What I would really like to see is RPi configurations optimized to reduce writes. For example mount the whole SD card read-only, mount a few essential filesystems as ramfs (/var, /tmp, /run), and have, say only one partition like /srv mounted read-write (+ noatime). If you run a webserver, and serve files off /srv you can still update them whenever with no hassle, but when the webserver is otherwise just serving the files, there should be zero writes.
That's essentially what the Nerves Project does [1]. You can boot Phoenix webserver serving static files with zero writes. The root filesystem and server code is written to one of two read-only partitions updated using an A/B scheme. Then a separate ext4 partition is RW and mounted to `/root` by default.
The trick is dealing with stateful apps (e.g. sensors) that write continuously. Power outages are painful on SD cards. Lately I've moved to a separate USB drive for writing bulk sensor data with Pi's. Then the application data partition on the SD card can be used for backup sensor readings -- probably summarized data written rarely to reduce chance of SD card corruption. Mostly when I've had SD card issues its with just the RW partition, specifically the database I'm using gets checksum errors. Never had one not boot due to a bad SD.
Though I'm contemplating trying out BTRFS mirror'ing on a single USB drive, though bcachefs looks promising once mainlined. It's cheaper to buy two 32GB USB Thumbdrives (mirrored) or even a full usb based ssd than buying an SD card with a large amount of SLC based storage. The new Pi's will be a significant upgrade in speed with external USB SSD/Thumbdrives.
Given the market segment, theoretical longevity and lifespan are not a concern - both on the producer side, who has to put more expensive chips, and on the consumer side, who has to pay for the chips and the media (SD cards are extremely cheap).
Moving up in the market segnment there are in fact higher-end solutions:
- the Odroid XU4 has an eMMC port (for those who want a compact solution but don't care
about long-term durability)
- again, the XU4 supports a SATA bridge
- the Odroid has both eMMC and SATA ports
(I mention Odroid purely because I'm familiar with the products)
Or at least support USB boot out-of-the-box. At lease someone could buy a USB 3.1 M.2 enclosure and boot from that if they need the stability and speed without modifying the form-factor or increasing the price.
I'm surprised we haven't seen a smaller than M.2 form factor for SATA on embedded devices. It seems like the big limiter on devices like the Raspberry Pi and others are the sd cards/emmc modules, since the amount processing power and ram have been continually going up.
If you need disk performance on the Pi 4, wouldn't it make more sense to only keep the kernel,initrd on the sdcard and mount the primary filesystem off a USB3 device (like an mSATA-2-USB enclosure), that is if you have the space to spare for it in your project/enclosure?
I did something similar with one of the SolidRun devices, but I had to scuttle it due to other hardware problems:
General practice in embedded is to avoid writing to disk as much as possible. Data collection is mostly done by sending it to the cloud for storage there. However 'edge processing' and 'edge storage' are hot topics, so there will probably be more work in this area.
Some of the surveillance camera vendors are showing interest in this, since it would be beneficial to reduce data needed to be sent. Especially since vast majority of the data is never looked at, or only looked at once (human/machine event detection).
But one has the (fundamental?) problem that the sensor is often installed in an adversarial environment. Would be sad to try to pull data after a crime, only to realize the camera storage was destroyed.
Many embedded applications are also not particularly form-factor constrained. A good security camera lens is often easily bigger than a couple of 2.5" hard-drives and a medium sized single-board computation.
Heck, just wiring boxes for an industrial sensor system, are can often bigger than 10s of 3.5" disks.
Good chance that doesn't trickle down to the microSD interface; it seems like the best performance improvements are reserved for the full-size SD cards, likely because they can fit more thermal mass and connection pins in the larger case.
The term you want is “microSD express”. That is the standard for PCIe and NVME on microSD cards. It specs out at about 1GB/second transfer rates.
Fun engineering decisions: It can use RAM from the hosting computer to store its logical block mapping tables so it doesn’t have to have DRAM onboard for that. That’s a clever way to use what the host has in abundance to avoid putting it on the price and volume constrained microSD card.
Now… how you are going to get the heat out of it when it is running at 1GB/sec. I wonder if we will see new microSD sockets with integrated heat sinks.
Solder the microSD and use the solder-points for thermal connection to the board, like for most other medium-power ICs. It is also cheaper than a socket.
Unfortunately I have not found microSDs suitable for reflow soldering. Typically one uses eMMCs instead, which are very similar, but has a much higher price (and reliability).
I'm surprised that eMMC hasn't seen more of a price drop off like SSDs, mSD and other flash storage. The OrangePi PC+ does include 8GB of eMMC, but I think that was the largest they could get for a cheap price...
Soon (hopefully) the Pi 4 will be able to completely boot off USB 3.0 devices; right now you can partially boot off an external volume, but /boot still has to be on a microSD card.
Though in my testing, it's still not a night-and-day performance difference, if judged by the Pi 3B+. I am not sure if the Pi 4's USB 3.0 port bandwidth will give that much of an improvement.
It would be quite nice to have any form of drive supported more directly, NVMe, m.2, whatever.
It doesn't, but in my informal tests (I haven't had time to set up a thorough benchmark) real-world usage of an external USB 3.0 SSD (testing with a cheap Kingston SSD, though, and a cheap USB SATA interface) is only marginally faster than a Samsung Evo+ microSD card.
I wonder if this is also due to the fact that many desktop SSDs do not focus on 4k read/write performance. This is one big area where nvme SSDs seem to have an advantage, their 4k read/write is usually significantly higher than that of other SSDs (at least from benchmarks I've seen).
Maybe even a lower end nvme ssd + usb enclosure would be the way to go to get the best performance.
Unless there's something about the USB interface itself that kills random 4k r/w performance?
True; some of the cheapest SSDs I've tested seem to just be the same package that would be inside a USB flash drive or SD card, hardwired to an SATA plug in a giant 2.5" enclosure!
There's also the UFS Card format, which is a card version of the internal storage that most smartphones use these days. It's still nascent, but in theory should trickle down to single board computers as low cost SoCs get UFS interfaces. It's substantially higher perf than eMMC/SD at read, write, and IOPS.
they are "at most" some speed. What even is the point of writting such spec?
the car analogy would be if you measure some car going down a hill (doesn't even matter if tumbling and killing everyone inside), measuring that it was 300mph and then advertising it as "speeds of 300mph!!!"
It’s how everything is advertised. Fuel efficiency/mileage, serving suggestions, battery life, investment returns, ‘savings of up to x%’ etc. Unless these things get regulated (and sometimes even if they are) the advert doesn’t really match reality.
The UK has to do that because the fastest widely-deployed internet connections available are VDSL2. DSL connections in general are relatively unreliable, and speeds vary wildly depending on a variety of factors, most importantly the length of the cable between you and your cabinet/exchange. Australian folks may chime in here with tales of their DSL connections dropping out during heavy rain, when their telephone junction boxes fill with water and short out.
If the UK had deployed fit-for-purpose tech like Fibre-To-The-Home (FTTH), there would be much less need for the UK's advertising regulations. (Obviously speeds are never guaranteed on residential connections, but advertised fibre speeds are a much more accurate predictor of internet experience than DSL or cable.)
"But", you might say, "that BT page is advertising 'Superfast Fibre'!". Well, yes, in the UK it is legal for telcos to lie to their customers. When they say fibre, they mean Fibre-To-The-Cabinet - and VDSL the rest of the way. Would anyone call their 3G mobile service "Superfast Fibre" because the tower is fibre-fed? No. For whatever reason, though, this is totally okay in the UK.
On the flip side, in the early 2000s Bellsouth (now AT&T) deployed FTTP in much of the Southeast and would convert that to ADSL to plug into existing house POTS wiring. It was sold as ADSL at 12mb/s. There was literally no other internet option available from them at the time either.
The averages are basically pointless. They might as well publish average average book lengths for categories on Amazon. Want a young adult novel? Average price 4¢ per page! Oh you want a specific novel? Sorry we advertise averages only.
What you'll actually get is determined by factors entirely out of their control. Whether that's an install engineer using a reel of Cat5e because that's what was in the truck even though the work order said crappy Category 3 phone cable would do, or even the neighbours are all using the most expensive package and that causes electrical interference that reduces everybody's bandwidth.
You can measure your actual characteristics, and the ISP will offer to do a half decent estimate of that, that's useful to at least decide which service to buy but the average is essentially worthless.
Testing microSD card speed on a Mac is rather pointless because performance is (severely) limited on the Raspberry Pi through it's terribly slow interface, not necessarily the actual card.
Remember RPi 1-3 use a plain SPI interface, and even the new 4 only has UHS-1.
I have no idea what your point is. Testing on a mac is pointless because RPi performance is limited?
Except the article showed the exact same performance in both places.
It started with: Why don't these cards designed for things like RPi actually show any improvement? Oh, they don't show improvement on any device at all, they are full of crap.
The point is if the cards actually meet spec and if they behave differently on the RPi.
So the first step is to show how the cards behave in an environment with no artificial performance limitations. This is the baseline "peak performance".
The next step is to show whether the RPi's performance ceiling is below that baseline, or if any specific card behave particularly poorly on the RPi. 2 cards may be equal on a high performing machine but for some reason have big differences when running in the RPi.
Even at 25MBps, it's very easy to see if you're bottlenecked to less than a thousand 4K writes per second (4MBps), and pretty easy to see if you're bottlenecked to less than three thousand 4K reads per second (12MBps).
A2 needs support in the Linux kernel. It's not there, yet.
https://github.com/ThomasKaiser/Knowledge/blob/master/articl...
"A2 promises even better performance with 4000/2000 read/write IOPS minimum but there's a problem since as outlined by the SD Association A2 cards show "much higher performance than A1 performance by using functions of Command Queuing and Cache".
Cache and Command Queuing require host (driver) support since the host needs to activate those new features first. The cache feature on A2 rated cards makes use of volatile RAM on the card requiring the host to learn new commands to issue flushing the cache (involving the risk of data losses -- for details see especially chapter 4.17 in Physical Layer Simplified Specification 6.0)"
So is there any kind of device able to use A2 cards at the rated performance yet? I can't get anything better on my Mac with multiple readers (USB 3.0 and 3.1), and I'm guessing if I tried on the reader in my 1 year old Dell XPS, I'd get the same.
What good is a performance designation (A2) if no device in the real world supports it?
I don't know. It's a bit of a bummer.
I imagine Android would benefit the most (and phones also don't suffer from sudden power outages, which is the biggest risk with enabling this), so someone may try looking if mmc core in androind kernel has patches to support this.
My phone will reboot randomly; depending on what layer the reboot happens at, there may be no flush command sent to the card.
No worries, I checked it a few minutes ago, and android doesn't support this either.
> and phones also don't suffer from sudden power outages,
Phone batteries die, and the battery remaining percentages are only an estimate so writing code to react to that condition can be fickle.
> and phones also don't suffer from sudden power outages
Perhaps you have never dropped your phone, but in my experience, on a dropped phone the battery cover and the battery often go flying.
I will say that at least most flagship phones these days have internal batteries.
Flagships have been skyrocketing in price and are losing marketshare to the midrange segment, which almost always has removable storage, and often has a swappable battery.
> What good is a performance designation (A2) if no device in the real world supports it?
this is often the case for new hardware specs/protocols..
see also: any SATA/SCSI rev N+1, TCQ/NCQ, UASP, PCI n+1, etc.
But in all those cases (I have used most of them), there was _some_ way a consumer could acquire hardware (and drivers) to use the tech. It might've been expensive, but I could go out and buy an SATA card, or SCSI 2, or whatever.
With A2, so far I have not seen any way any consumer (or anyone, really) could achieve 2000/4000 IOPS.
That would be like selling a SATA 2400 drive for 2x the cost of an SATA 600 drive... but not selling any kind of controller that could use more than SATA 600 bandwidth.
It should be possible to patch the kernel. If I had an A2 card, I'd try. It's probably just a question of detecting card features on card plug-in and sending a command to enable the cache.
Spec/code is open. I'm just sort of surprised noone tried it yet.
You will probably need a true SD/MMC controller to use any "new" commands, such as a smartphone or embedded target (such as the RPi). USB or other bridges used on notebooks are unlikely to have support for it (especially as they convert MMC/SD to a USB-storage class device)
Assuming the cards with the A2 designation do actually support the claimed speeds, support will (eventually) be added, and the faster performance will be observed, on the Pi and other host devices.
Does anyone know if an improved Linux driver is in the works?
I appreciate you doing these tests, this (and your last article) instantly answers my question about "how will this perform in one of my rpis?". Please keep doing them as new cards(with their marketing claims) are introduced!
How about lifespan? I fail to understand why SD card have such a short lifespan, and it's the sole reason I won't acquire an RPi.
It seems that the only think that is lacking on RPi is flash memory storage. Most smartphones have a lot of it, and it seems that 1GB of storage would be enough for the RPi and would not necessarily be so expensive.
I can't help you with the why some SD cards are short lived, but if you want a longer lasting SD card, try the "endurance" versions. Most of the big brands have a version. They are a bit more expensive, but promise 25x the write life of a standard SD.
They are typically marketed toward video surveillance systems and dash cams. I'm not sure how the rest of a Raspberry Pi's performance would be effective.
FWIW, only noname and Sandisk cards have given up the ghost on me. Samsung Pro & Evo cards have been doing just fine. Other than that, you just need to worry about getting proper power. Just use the official power supplies, and you're fine.
Also eMMC chips are no magic. There are crappy ones and good ones. Guess which ones you're going to get for little money?
Proper power is essential. A phone charger is not a power supply even though it works. I suspect a lot of SD cards are getting undeserved blame from a PSU issue.
This is my experience. The only time I had an issue with corruption (using over 60 different microSD cards through 7 years and maybe 50 or so different Raspberry Pis) was when I used a little .5A microSD AC adapter that came with some cheap battery powered device that broke.
I have not had any problems when using a reliable power supply. I'm guessing the 'endurance' or 'industrial' cards some people claim are lifesavers are just doing something to guard against flaky power input when doing reads and writes, so it masks the issue.
Not having a proper power supply is my only pi related regret. Thing would freeze every once in a while. That since stopped happening, and the only change in-between (beside updates) was the initial used charger dying.
I have had SanDisk, Kingston, and Sony cards die on me in a variety of devices from Pis to GoPros. Sonys were the fastest to die but retained read capability, where as the SanDisk just stopped functioning completely.
My most recent additions are a Samsung Evo card and a Samsung Endurance. I am curious to see which dies first.
The Pi doesn't necessarily need an SD card. I'm running it on NFS, booted over the network. Works just fine. No need to worry about burning out the SD card.
What component supports NFS booting? The bootloader is on the SD card, right?
I believe the RPi can boot directly from the network, but it does require a one-time config change that requires booting from the SD card once. After that, the SD card isn’t required for future reboots.
(Or actually even supported, IIRC)
I don't think the Pi 4 supports netboot yet[1].
[1] https://www.raspberrypi.org/documentation/hardware/raspberry...
I should have mentioned that this was currently only for the 3.
Hopefully that will get fixed soon.
> ... and it seems that 1GB of storage would be enough for the RPi and would not necessarily be so expensive.
Enough for what? A typical Raspbian install not takes a bit more than 4GB. I don't think that the Lite version (no GUI or applications) would fit either. I'd suggest 8GB, if that's a convenient size. As flash density increases, more and more goes on a single chip and that's where production is going. Consider the price of SD cards or USB sticks. Below about 16 or 32GB they don't get much cheaper and I suspect is is for this reason.
And IAC, the RPi foundation is working hard to keep the price of the board at $35US so adding something like this probably means something else would have to be eliminated.
It don't know about Raspbian, but Armbian (headless) fits in 1.2GB
I'd look into industrial SLC uSD cards like the ones from Swissbit: https://www.digikey.com/en/supplier-centers/s/swissbit
Most consumer cards are designed around media storage so they don't last as long.
smartphone batteries will go out much, much sooner than a RPi SD card write-life. And nowadays they are much harder to replace.
That is just not true. I have burned through many a RPi SD card in 3-6 months, while phone batteries last 2 years at minimum.
In my experience this isn't caused by exhausting the endurance of the flash. It seems like a lot of SD cards have really buggy firmware. And that Linux running on a RPi is particularly prone to triggering these bugs. Industrial SD cards seem to hold up better.
Early iterations of the RPi boards apparently had some bad design choices in the power circuitry, which makes the problem worse.
What cards are you using?
I have three RPis in abusive conditions, and they haven't had a single SD card issue. Two RPi 3b's running as a car entertainment system, with no form of battery backup/safe shutdown. They are on the ignition power, and once the car turns off, they hard shut down. These have been in place for nearly 2 years with the same Samsung Evo 16GB cards.
The other is attached to a digital signage with less frequent hard shut downs, but still has at least one per week, and been in place since late 2017. Also running a Samsung Evo card (I think it's a 32GB since that's what I had on hand).
Both have high quality power supplies, which may help. The in-car systems have Anker car USB chargers that have been de-cased and soldered in place, while the signage one has an Anker wall charger and a standard micro-b cable.
Interesting. The ones that failed were mostly Kingston. I recently switched to Samsung Evo and have not had any failures yet.
The failed ones were all running media servers with actual media storage on external drives, but indexing done on the SD card.
I ended up aggressively moving write loads to external storage and most of my RPis are running with read-only mounted SD cards these days.
What's the write cycle like? Seems like those would mostly be read only.
The cards that die quickest for me are either written excessively like a GoPro or Pi based data loggers written constantly. The GoPros are just burning out the cards but the data loggers corrupt the disk images during unexpected power loss.
And what brand are you buying?
FWIW I have several Pis running 24x7 and have not had a single SD card failure. I usually buy high end cards but I also buy cheap cards (4GB cards)and a few Micro-Center house brand, for which I don't know the quality.
I've had one that has been running years, I think. Drives a monitor, looping a video where my wife works.
The only SD card failure I have experiences was in a camera.
What is your use case for the Pi? That’s a very short lifespan.
Looks like I'll keep booting my pi's from usb attached storage. At least with the Pi4 USB speed improvements should be nice.
I didn’t even know you could do that! Any idea if it works on the pi4 or not? The docs don’t mention it.
https://www.raspberrypi.org/documentation/hardware/raspberry...
just a guess but s/he may have been referring to just putting root filesystem on a USB drive, but still requiring the bootloader (and possibly kernel?) to be on sd card. It's a one-time "hit" that is completely negligible.
That's the present situation and I've used it before. Some recent Pis can boot directly from a USB drive without the need for a bootloader on the SD card. That's still in the works for the Pi 4 as I understand, but network boot (PXE?) will come before that.
The major problem is not disk write speed, which we can all agree sucks, but longevity and lifespan when used as a system boot disk and also containing /var and similar. The write wear leveling on a u3 microsd card is nowhere near as good as even a $50 consumer class ssd.
Small single board computers should move towards m.2 2280 nvme drive support, or at least one sata 6 gbps interface.
The Pi steals the lion’s share of the attention the in the cheap SBC space, but there are plenty of alternatives that have better storage solutions out there. I too grew fed up of SD card issues.
I always see the argument “but the Pi has the best community support!” trotted out, and I don’t disagree. However, it’s all normally just sane Linux distros on the Pi at the end of the day. Whatever instructions/advice someone has written for a given Pi project will likely be applicable on a non-Pi board anyway.
I think it depends a lot on your use case. If you just want a dirt cheap HTPC, then yes, there are lots of SBCs that will fit the bill, but the RPi also has a hardware add-on ecosystem around its particular camera and GPIO/SPI/I2C/I2S availability, and even though lots of other boards have those things available and broken out too, you're going to have to be willing to do a lot more leg work to get it all hooked up properly and working for a different SoC.
SD card performance on the Pi <4 is so poor (and the bandwidth is shared with the USB controller anyway, I think) that you might as well use an attached USB hard disk; even for booting.
Jeff's benchmarks are accurate in my experience. If you try saving single images from a camera at even 5-10fps, the card quickly bottleneck and you start dropping frames. If you buffer and store (in my case 16-bit raw), you can essentially saturate USB bandwidth if you're also using a USB camera.
We've got a few Pi's in office "production" (used as dashboards) and the best solution we've found it to use an SSD drive via USB. Something like a 128GB SSD drive is cheap enough, although it's about as expensive as the Pi itself.
No, the bandwidth isn't shared as both sd controllers (sdhci and sdhost) are directly accessible using MMIO (and DMA).
Thanks for clarifying that, I was never sure.
Yeah I'm always fighting both write speed and network speed limitations when I do timelapse projects with my Pi Zeros. Hopefully the I/O improvements from the 4 trickle down to the Zero form factor sometime soon!
There are SD cards rated for "high endurance" for example https://www.amazon.com/gp/aw/d/B07B984HJ5 is designed for continuous 24/7 writes such as surveillance cameras. These should have great longevity when used in a RPi.
What I would really like to see is RPi configurations optimized to reduce writes. For example mount the whole SD card read-only, mount a few essential filesystems as ramfs (/var, /tmp, /run), and have, say only one partition like /srv mounted read-write (+ noatime). If you run a webserver, and serve files off /srv you can still update them whenever with no hassle, but when the webserver is otherwise just serving the files, there should be zero writes.
That's essentially what the Nerves Project does [1]. You can boot Phoenix webserver serving static files with zero writes. The root filesystem and server code is written to one of two read-only partitions updated using an A/B scheme. Then a separate ext4 partition is RW and mounted to `/root` by default.
The trick is dealing with stateful apps (e.g. sensors) that write continuously. Power outages are painful on SD cards. Lately I've moved to a separate USB drive for writing bulk sensor data with Pi's. Then the application data partition on the SD card can be used for backup sensor readings -- probably summarized data written rarely to reduce chance of SD card corruption. Mostly when I've had SD card issues its with just the RW partition, specifically the database I'm using gets checksum errors. Never had one not boot due to a bad SD.
Though I'm contemplating trying out BTRFS mirror'ing on a single USB drive, though bcachefs looks promising once mainlined. It's cheaper to buy two 32GB USB Thumbdrives (mirrored) or even a full usb based ssd than buying an SD card with a large amount of SLC based storage. The new Pi's will be a significant upgrade in speed with external USB SSD/Thumbdrives.
1: https://github.com/nerves-project/nerves/blob/master/docs/Ad...
Given the market segment, theoretical longevity and lifespan are not a concern - both on the producer side, who has to put more expensive chips, and on the consumer side, who has to pay for the chips and the media (SD cards are extremely cheap).
Moving up in the market segnment there are in fact higher-end solutions:
(I mention Odroid purely because I'm familiar with the products)Corrections:
Would be nice to see an NVMe slot on one of these things. I’m guessing there aren’t even enough PCI Express lanes on RPi 4 to do that today.
Or at least support USB boot out-of-the-box. At lease someone could buy a USB 3.1 M.2 enclosure and boot from that if they need the stability and speed without modifying the form-factor or increasing the price.
I expected they would have at least added a EMMC connection by now. With the improvements from the new faster EMMC options it isn't even that slow.
Exactly, who cares about speed when all the microSD cards I try eventually become corrupt due to poor controller logic.
Try an ‘industrial’ flash card. I outfitted my pi with a SwissBit microsd and haven’t had any disk problems since.
That'd be an interesting feature to see but I'm sure it'd also really run up the board cost...
rockpi 4A is $39 and has an m.2 slot
https://shop.allnetchina.cn/collections/frontpage/products/r...
If you want longevity, but an SLC card.
Support for booting from SSD should solve the SD issues before too long.
why things on the RPi take so long? is the SoC still undocumented because of the NDA as before?
Very glad I took your recommendation from your previous article on the Evo cards for my new RPi4s! kudos and thank you sir!
SD is a garbage spec. Arasan IP for SD host (in Raspberry SoC) is trash. Driver support is minimal. What'd you expect?
I'm surprised we haven't seen a smaller than M.2 form factor for SATA on embedded devices. It seems like the big limiter on devices like the Raspberry Pi and others are the sd cards/emmc modules, since the amount processing power and ram have been continually going up.
If you need disk performance on the Pi 4, wouldn't it make more sense to only keep the kernel,initrd on the sdcard and mount the primary filesystem off a USB3 device (like an mSATA-2-USB enclosure), that is if you have the space to spare for it in your project/enclosure?
I did something similar with one of the SolidRun devices, but I had to scuttle it due to other hardware problems:
https://penguindreams.org/blog/review-clearfog-pro/
General practice in embedded is to avoid writing to disk as much as possible. Data collection is mostly done by sending it to the cloud for storage there. However 'edge processing' and 'edge storage' are hot topics, so there will probably be more work in this area. Some of the surveillance camera vendors are showing interest in this, since it would be beneficial to reduce data needed to be sent. Especially since vast majority of the data is never looked at, or only looked at once (human/machine event detection). But one has the (fundamental?) problem that the sensor is often installed in an adversarial environment. Would be sad to try to pull data after a crime, only to realize the camera storage was destroyed.
General practice in embedded is to avoid writing to disk as much as possible.
Yes.
Data collection is mostly done by sending it to the cloud for storage there.
Nononono, what has the world come to?
However 'edge processing' and 'edge storage' are hot topics
Meaning "how it used to be done when hardware was a hundred times slower"?
Many embedded applications are also not particularly form-factor constrained. A good security camera lens is often easily bigger than a couple of 2.5" hard-drives and a medium sized single-board computation. Heck, just wiring boxes for an industrial sensor system, are can often bigger than 10s of 3.5" disks.
> Many embedded applications are also not particularly form-factor constrained.
just because the overall package is large doesn't mean the embedded designers are allocated a large portion of that.
and the upstream suppliers are always going to be pushing to be a bit smaller, so they can fit into more things, and get more sales.
Heat and power (think PoE class) are other constraints to take into consideration.
Agreed, but Raspbian is not an embedded operating system. It spams the syslog and uses a journalling filesystem.
The SD 7.0 standard includes support for PCIe and NVMe.
https://www.sdcard.org/press/SD_EXPRESS_A_REVOLUTIONARY_INNO...
Good chance that doesn't trickle down to the microSD interface; it seems like the best performance improvements are reserved for the full-size SD cards, likely because they can fit more thermal mass and connection pins in the larger case.
The term you want is “microSD express”. That is the standard for PCIe and NVME on microSD cards. It specs out at about 1GB/second transfer rates.
Fun engineering decisions: It can use RAM from the hosting computer to store its logical block mapping tables so it doesn’t have to have DRAM onboard for that. That’s a clever way to use what the host has in abundance to avoid putting it on the price and volume constrained microSD card.
Now… how you are going to get the heat out of it when it is running at 1GB/sec. I wonder if we will see new microSD sockets with integrated heat sinks.
Solder the microSD and use the solder-points for thermal connection to the board, like for most other medium-power ICs. It is also cheaper than a socket.
Can standard microSD's survive standard reflow soldering? I wouldn't think they survive, but would be nice.
Unfortunately I have not found microSDs suitable for reflow soldering. Typically one uses eMMCs instead, which are very similar, but has a much higher price (and reliability).
I'm surprised that eMMC hasn't seen more of a price drop off like SSDs, mSD and other flash storage. The OrangePi PC+ does include 8GB of eMMC, but I think that was the largest they could get for a cheap price...
Actually the microSD Express spec was announced recently.
https://www.theverge.com/circuitbreaker/2019/2/25/18239558/m...
Yep. USB disks also are generally much more robust to errors than SD cards. My mail spool is on USB, the OS is on the SD card.
Soon (hopefully) the Pi 4 will be able to completely boot off USB 3.0 devices; right now you can partially boot off an external volume, but /boot still has to be on a microSD card.
Though in my testing, it's still not a night-and-day performance difference, if judged by the Pi 3B+. I am not sure if the Pi 4's USB 3.0 port bandwidth will give that much of an improvement.
It would be quite nice to have any form of drive supported more directly, NVMe, m.2, whatever.
Does /boot get read/written to enough where this is not "good enough"
It doesn't, but in my informal tests (I haven't had time to set up a thorough benchmark) real-world usage of an external USB 3.0 SSD (testing with a cheap Kingston SSD, though, and a cheap USB SATA interface) is only marginally faster than a Samsung Evo+ microSD card.
I wonder if this is also due to the fact that many desktop SSDs do not focus on 4k read/write performance. This is one big area where nvme SSDs seem to have an advantage, their 4k read/write is usually significantly higher than that of other SSDs (at least from benchmarks I've seen).
Maybe even a lower end nvme ssd + usb enclosure would be the way to go to get the best performance.
Unless there's something about the USB interface itself that kills random 4k r/w performance?
True; some of the cheapest SSDs I've tested seem to just be the same package that would be inside a USB flash drive or SD card, hardwired to an SATA plug in a giant 2.5" enclosure!
SATA DOM can get pretty tiny, https://www.neweggbusiness.com/Product/Product.aspx?Item=9SI...
Raspberry Pi has supported direct USB boot on most boards for a while now (Pi 4 awaiting a boot firmware update), no need to also use a uSD.
Glad to hear I'm not the only one who had a nightmare experience trying to use the ClearFog Pro as a router :).
There's also the UFS Card format, which is a card version of the internal storage that most smartphones use these days. It's still nascent, but in theory should trickle down to single board computers as low cost SoCs get UFS interfaces. It's substantially higher perf than eMMC/SD at read, write, and IOPS.
sd card performance specs are all a joke.
they are "at most" some speed. What even is the point of writting such spec?
the car analogy would be if you measure some car going down a hill (doesn't even matter if tumbling and killing everyone inside), measuring that it was 300mph and then advertising it as "speeds of 300mph!!!"
Also how ISP speed is measured ;)
It’s how everything is advertised. Fuel efficiency/mileage, serving suggestions, battery life, investment returns, ‘savings of up to x%’ etc. Unless these things get regulated (and sometimes even if they are) the advert doesn’t really match reality.
Advertised, not measured.
And it's not like that everywhere, the UK has started what I hope develops into a trend: https://www.bt.com/broadband/deals/
The UK has to do that because the fastest widely-deployed internet connections available are VDSL2. DSL connections in general are relatively unreliable, and speeds vary wildly depending on a variety of factors, most importantly the length of the cable between you and your cabinet/exchange. Australian folks may chime in here with tales of their DSL connections dropping out during heavy rain, when their telephone junction boxes fill with water and short out.
If the UK had deployed fit-for-purpose tech like Fibre-To-The-Home (FTTH), there would be much less need for the UK's advertising regulations. (Obviously speeds are never guaranteed on residential connections, but advertised fibre speeds are a much more accurate predictor of internet experience than DSL or cable.)
"But", you might say, "that BT page is advertising 'Superfast Fibre'!". Well, yes, in the UK it is legal for telcos to lie to their customers. When they say fibre, they mean Fibre-To-The-Cabinet - and VDSL the rest of the way. Would anyone call their 3G mobile service "Superfast Fibre" because the tower is fibre-fed? No. For whatever reason, though, this is totally okay in the UK.
On the flip side, in the early 2000s Bellsouth (now AT&T) deployed FTTP in much of the Southeast and would convert that to ADSL to plug into existing house POTS wiring. It was sold as ADSL at 12mb/s. There was literally no other internet option available from them at the time either.
And here I am, an hour ouside the city and would gladly pay 150/mo for half that. But noone can, or cares to, deliver.
The averages are basically pointless. They might as well publish average average book lengths for categories on Amazon. Want a young adult novel? Average price 4¢ per page! Oh you want a specific novel? Sorry we advertise averages only.
What you'll actually get is determined by factors entirely out of their control. Whether that's an install engineer using a reel of Cat5e because that's what was in the truck even though the work order said crappy Category 3 phone cable would do, or even the neighbours are all using the most expensive package and that causes electrical interference that reduces everybody's bandwidth.
You can measure your actual characteristics, and the ISP will offer to do a half decent estimate of that, that's useful to at least decide which service to buy but the average is essentially worthless.
Doing god’s work
Testing microSD card speed on a Mac is rather pointless because performance is (severely) limited on the Raspberry Pi through it's terribly slow interface, not necessarily the actual card.
Remember RPi 1-3 use a plain SPI interface, and even the new 4 only has UHS-1.
I have no idea what your point is. Testing on a mac is pointless because RPi performance is limited?
Except the article showed the exact same performance in both places.
It started with: Why don't these cards designed for things like RPi actually show any improvement? Oh, they don't show improvement on any device at all, they are full of crap.
He explained in the article why he did this: to check if the manufacturers claims are incorrect, or if its a RPi hardware issue.
It's neither.
it's clearly the former.
Manufacturer's claims are not incorrect. OS vendors just didn't intoroduce support yet.
The point is if the cards actually meet spec and if they behave differently on the RPi.
So the first step is to show how the cards behave in an environment with no artificial performance limitations. This is the baseline "peak performance".
The next step is to show whether the RPi's performance ceiling is below that baseline, or if any specific card behave particularly poorly on the RPi. 2 cards may be equal on a high performing machine but for some reason have big differences when running in the RPi.
Even at 25MBps, it's very easy to see if you're bottlenecked to less than a thousand 4K writes per second (4MBps), and pretty easy to see if you're bottlenecked to less than three thousand 4K reads per second (12MBps).
Rpi1-3 definitely have an MMC controller.