> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this.
A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies.
Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
> I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime
Can you share more about this? I have a APC Back UPS PRO USV 1500VA (BR1500G-GR) and it would be nice to know if this is possible with that one as well.
That UPS eventually died, and I'm not sure if it was because it was hooked up to a larger battery than it was designed for, but it's still only 12 volts so I don't think the electronics would notice. What they may notice is extended run-time in the event of a power failure.
It was a crude mod. Take the cover off and remove the existing little security alarm battery, use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS), and feed the battery cables out through the hole. I probably got some additional cables with appropriately sized terminations to effectively extend the short existing ones (since they were only designed to be used within the device). And then connect it up to a car battery.
Cover any exposed metal on the connectors with that shrink rubber tubing or electrical tape. Be very careful with exposed metal around it anywhere, especially touching the RED POSITIVE pole of the battery. Get a battery box - I got one for the big-ass AGM battery.
Test it out on a laptop that's had it's battery removed or disconnected that, just in case, you don't care too much about losing.
Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity.
Personally, I think it's safer a less hassle to go for a LiFePo4 (LFP) Power Station style device that has UPS capabilities. LFP batteries have 3,000-ish cycle lifetimes, which could be nearly ten years with daily use.
No, don't do it. I understand his thought process because they are both 12v batteries with more capacity, but car batteries are made for high burst of energy which a car engine ignition requires, whereas UPS batteries are made for slow drains. Also, these UPS are made for charging battery cells in a certain way, if you start to stack a bank of batteries of the same model in parallel hoping for more capacity, even then its a problem for the UPS's charger, they won't charge evenly and eventually becoming a problem.
> I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet
If your OS is using systemd, you can fix that pretty easily by adding an After=network-online.target (so the ExecStart doesn't even try to check if there is no networking yet) and an ExecCondition shell script [1] to actually check if nfs / smb on the target host is alive as an override to the fs mounts.
Add a bunch of BindsTo overrides to the mounts and the services that need the data, and you have yourself a way to stop the services automatically when the filesystem goes away.
I've long been in the systemd hater camp, but honestly, not having to wrangle with once-a-minute cronjobs to check for issues is actually worth it.
If anyone is looking to get started with a homelab at a good price, I can highly recommend checking ebay for a Dell Wyse 5070. They flooded the market for $50 and are likely powerful enough for many needs. They have a M.2 slot that support SATA. The 'extended' version also has space for a small pcie card and has a parallel and 2 serial ports for a blast to the past.
I built mine around an N150 board off of aliexpress. 6 SATA slots, 4x2.5G ethernet, 2x m.2 slots. Find a cheap second-hand case, a bit of RAM and you're ready to go. It's got hardware transcoding, handles 4K without breaking a sweat. And it consumes 6W!
Another cheap option are Fujitsu Futro's. They're meant to be thin server clients meant to operate larger server racks (I guess?). Anyway, they come with 4-8GB RAM, an SSD (most people upgrade them), and even have a PCI-e slot (depending on the model) to use with a 2.5 or 10 Gbit ethernet card, for example. Around $50 on ebay.
For a bit more money, Optiplex Micro / Lenovo Tiny / HP Mini series with at least 8th gen i5 are a good option too. Can be found from Ebay for about 70 - 120 USD, much more powerful than Wyse 5070 while still quite power efficient (about 10W idle, as opposed to 5W of Wyse). Usually they come with one NVME, one SATA 2.5" slot, some premium models even with PCIE.
All good options, just noting that based on some searches I made they all seem to lack serial ports compared to the Wyse if that is something you care about (I personally do). There could be variants out there though with serial ports and if would be happy to hear about them and even more happy if there are fanless variants/alternatives for those of us with very limited space at home and a need to avoid noise.
I don't know about fanless, especially with an i5 as opposed to n-something.
But not all those minis are the same. G4 (intel 8th gen) and G5 (intel 9th gen) HPs are horrendous. The fan makes an extremely aggravating noise, and I haven't found a way to fix it. Bonus points for both the fan and heatsink having custom mounts, so even if you wanted to have an ugly but quiet machine by slapping a standard cooler, you couldn't.
G6 versions (intel 10th gen) seem to have fixed this, and they're mostly inaudible on a desk, unless you're compiling something for half an hour.
Another tip: second hand gaming PCs! They can be incredibly cheap and powerful due to upgrade cycles just make sure to put a raid 1 on it as second hand gamer gear might be less reliable.
Looking at Raspberry Pi prices inside EU, I can get 8 core laptops for a cheaper price, with display, dGPU et al.
No idea what happened, but Raspberry Pis are super expensive for the last couple years, which is why I decided to just go with used Intel NUCs instead. They cost around 80-150EUR and they use more electricity but they are a quite good bang for the buck, and some variants also have 3x HDMI or Gbit/s ethernet or m2 slots you can use to have a SATA RAID in them.
No. The main pain point with RPi's is that they're SD card based – which are slow and prone to failure. Configuring an SSD to be used as the main storage has also been a pain in the past (not sure if that's changed recently).
With an n100, you get a better, more upgradable system for around the same price and same power usage. On top, you will also have an x64 system that isn't limited to some ARM quirks. I made the switch n100's over a year ago and have had no issues with them so far.
I have a somewhat bigger machine that hosts my homelab, an HP 800 G2 SFF. It takes "normal" components, so can ben modified. The only custom thing is the PSU, but the standard one is good enough for my needs. Bonus points for not requiring an external power adaptor.
It has an i5-6500, 32 GB RAM (16 + 2x8 DIMMs), 2 SATA SSDs and a 2x10Gb Connect-X3. It runs 24/7 hosting OpnSense and HomeAssistant on top of KVM (Arch Linux Hardened – didn't do anything specific to lower the power draw). Sometimes other stuff, but not right now.
I haven't measured it with this specific nic, but before it had a 4x1Gb i350. With all ports up, all VMs running but not doing much, some power meter I got off Amazon said it pulled a little over 14W. The peak was around 40 when booting up.
Electricity costs 0.22 €/kWh here. The machine itself cost me 0 (they were going to throw it out at work), 35 for the nic and maybe 50 for the RAM. It would take multiple years to break even by buying one of these small machines. My plan is to wait out until they start having 10 Gb nics and this machine won't be able to keep up anymore.
Quick search online tells me ~5W for the Dell Wyse 5070, which does not sound unrealistic as I have similar boxes that draw ~10W. So, 32 to 62kWh per year and then we have ~USD 6.5 to 13 per year assuming 20 cents per kWh which another online search told me was reasonably realistic for the US.
local dns, static site hosting, local apt cache, various other network services (unifi controller if you've got those APs for example), remote/headless dev machine (maybe not for kernel or bigcorp java development), or whatever else you want. mail if you want. Anything :)
Those little thin clients aren't gonna be fast doing "big" things, but serving up a few dns packets or whatever to your local network is easy work and pretty useful.
Even these low-power CPU's are surprisingly capable. As an example of more fancy thing, one could slap in some external storage, install Jellyfin, and run their own local streaming service off such a machine. The CPU is modern enough for efficient hardware transcoding of a stream.
While i get the whole homelab thing is exiting and a great learning experience, it's simply not worth the time and effort for the majority of people.
You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime).
In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever.
A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period).
It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks.
Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money.
Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.
But on a homelab you can host any service you want and start/stop it whenever you need to. Sure cloud storage might cost less in the short-term, but if you need more storage or more services, a selfhosted option is far cheaper.
VPS are very expensive for what you get. If you have the capital, doing it yourself saves you money very quickly. It's not rare to pay $50 for a semi-decent VPS, but for $2000 you would get an absolute beast that would last 10 years at the very least.
With Docker, maintenance is basically zero and unused services are stopped or restarted with 1 command.
How many services do you need ? And how much CPU power do those services need ?
I've also self hosted for decades, but it turns out i don't really need that much, at least not public.
I basically just need mail, calendar, file storage and DNS adblocking. I can get mail/calendar/file storage with pretty much any cloud provider (and no, there is no privacy when it comes to mail, there is always another participant in the conversation), and for €18/year i can get something like NextDNS, Control D, or similar.
For reference, a Raspberry Pi 4 or 5 will use around 50 kWh per year, which (again in europe) will translate to €15/year. For just €3 more per year i get redundancy and support.
I still run a bunch of stuff at home, but nothing is opened to the public internet. Everything runs behind a Wireguard VPN, and i have zero redundant storage. My home storage is used for backing up cloud storage, as well as storing media and client backups. And yes, i also have another cloud where i backup to.
My total cloud bill is around €20-€25/month, with 8TB of storage, ad blocking DNS, mail/calendar/office apps and even a small VPS.
I did not do the price calculations (in France, and I prefer not to know :) but I host many things except for mail and calendar (mail is tricky to host). Of these 29 services, I use maybe 4 daily and 15 monthly. They are well protected, easy to maintain, and serve family and friends.
As a Northern European (Finland) I can tell you that the electricity cost here for last year was closer to 0,1 € per kWh including the transfer fee and taxes. Additionally, more than 40 % of houses here have electric heating. The heating season starts in early autumn and ends in late spring, lasting 8 to 9 months depending on the year. As the electricity used by the device is turned into heat, during the heating season running it costs nothing.
Yeah, northern scandinavia has plenty of renewable energy.
As for electric heating, that is true in 1:1 heating scenarios, but i assume you guys are also using heat pumps these days, and while you still get heat "for free", it will not be anywhere as efficient as your heat pump.
Yes, it's probably peanuts in the grand scheme of things, i know our air to water heat pump in Denmark uses around 4500-5500 kWh per year, so adding another 100 kWh probably won't mean much.
As a bare minimum, you should update your server and docker images daily, or at least whenever there's an update (which you won't know unless you check).
If you only access your homelab over VPN or similar, then by all means, update whenever you feel like it, but if you expose your services to the internet, you want to be damned sure there are no vulnerabilities in them.
The internet of today is not like it was 20 years ago. Today you're constantly being hammerede by bots that scan every single IPv4 address for open ports, and when they find something they record it in a database, along with information on what's running on that port (provided that information is available).
When (not if) a vulnerability for a given service is discovered, an attacker doesn't need to "hunt & peck" for vulnerable hosts, they already have that information in a database, and all they need to do is start shooting at their list of hosts.
You can use something like shodan.io to see what a would be attacker might see (can check your own IP with "ip:xxx.xxx.xxx.xx".
Try entering something like Synology, Proxmox, Truenas, Unraid, Jellyfin, Plex, Emby, or any of the other popular home services.
Have you seen the prices for Google Drive et al? The NAS setup you describe (which I wouldn't consider worth the money for that little space) is what, 12GB with 1 parity drive?
Google One for 10TB is 274,99€/mo (at least in my country) so you'd make the entire nas price and subscription cost within a few months, let alone years.
There just aren't compelling public cloud for large sizes (My NAS is 30TB capacity and I'm using 18 right now) and even if you go the more complex loops with like S3 and whatnot you still get billed more than it's worth. Public cloud is meant for public files, there's a lot of costs you're paying for stuff you don't need like being fast to access from everywhere.
do you need more space than 2TB ? (excluding things you've downloaded from the internet)
Very few people i know has use for that much storage. Yes, you can download the entire netflix catalog, and that will of course require more storage, and no, you probably shouldn't put it in the cloud (or even back it up, or use redundant storage for it).
Setting up your own homelab to be your own netflix, but using pirated content, is not really a use case i would consider. I'm aware people are doing it, and i still think it's stupid. They're saving money by "stealing" (or at least breaking laws), which is hardly a lifehack.
My wife is a professional photographer, and while we do archive most of her RAW files somewhere else, pretty much everything HEIC, JPEG or any other compressed format lives in our main cloud.
We have 2.2TB in total for “direct storage”, and we’re currently using around 1.5TB, and that’s including myself and our kids.
My personal photo library has just short of 90,000 photos, and about 5,000 videos. My wife’s library is roughly twice that. I have no idea how many photos the kids have, but they each take up around 200GB for photos.
And then we have backups, which actually take up about 1TB per person, mostly because that’s the space I’ve allocated for each, so history just grows until it’s filled. Photos ideally won’t change much. We backup originals along with XMP metadata for edits, so the photos stays the same, and changes are described in easily compressed text files. Backups of course also have deduplication enabled.
How much space do you realistically need high availability, redundant storage for ?
For my personal use case, that involves photos and documents, all things i cannot easily recreate (photos less so). Those are what matters to me, and storing them in the cloud means i not only get redundancy in a single data center, but also geographical redundancy as many cloud providers will use erasure coding to make your data available across multiple data centers.
Everything else will be just fine on a single drive, even a USB drive, as everything that originated on the internet can most likely be found there again. This is especially true for media (purchased, or naval aquisition). Media is probably the most replicated data on the planet, possibly only behind the bible and IKEA catalog.
So, back to the important data, i can easily fit an entire family of 4 into a single 2TB data plan. That costs me somewhere around €85 - €100 per year, for 4 people, and it works no matter what i do. I no longer need to drag a laptop with me on vacation, and i can basically just say "fuck it" and go on vacation for 2 weeks.
The premium plan from Google has 2 TB and costs about the same annually as the electricity for the NAS that the GP comment suggested for comparison (at 100% usage). So at the same ONGOING cost (not even counting initial investment), the NAS has 8 times more storage. 16 times if you assume it will be mostly idle. Except if you want high availability with RAID, then you're back to 8 times. And haven't yet thought about backups.
All this assuming that you even need that much storage, which most people definitely do not.
Encrypted data on the servers is only useful if your server is just dumb storage. I want the server to actually do something, e.g. serving media, running home assistant etc.
OP has admittedly over-engineered their setup. Depending on your goals (cost, speed, space, autonomy), there are less-rigorous solutions for the layperson.
I, for one, don't want to have Google, etc. as a dependency[1], so I will pay some energy cost to do that.
Building a homelab is an awesome way to learn a lot of things.
I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people.
I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.
The encryption is interesting but I wouldn’t call this over engineered at all, in fact it’s rather basic compared to a lot of homelabs I see people building particularly where people are doing K8s or similar over multiple machines.
Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.
As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case.
When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.
Your VPS provider likely uses servers with ECC RAM, this home server doesn't. For most people it doesn't seem to matter but for me it does - a home server where I store my data needs to have ECC RAM.
This was exactly a use case I had in mind when building https://canine.sh -- also uses k3s as a provider, and provides a Heroku-like devex.
How to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying
I've got basically raw internet coming in to my OPNSense device, although I had to request certain ports to be removed from the ISP's by-default-blocked policy, since I host a mail server - but the ISP is fine with this, they have a form for it, super easy.
Some family members are behind CGNAT, and I'm not sure if their ISP has the option to move out from behind that, but since they don't self-host it's probably slightly more secure from outside probes. We're still able to privately share communications via my VPN hub to which they connect, which allows me to remotely troubleshoot minor issues.
I haven't looked into cloudflare tunnels, but haven't felt the need.
This is such a great post. I have a small collection of posts for inspiration in creating my homelab and this is getting added to it. Current have a Pi 4 with PiHole and a Beelink. Going to add one or two more machines.
I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments.
One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door.
At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me.
I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along.
I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks.
I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.
I can relate to this. I still run my own x86 box as a router to have the AP controller, but I'm strongly considering dropping this.
I need to update it and patch it, hoping nothing goes wrong in the process. If something breaks I'm the only one that can repair it, and I really don't want to hear my wife screaming at me at 7am when I wake up.
Eventually I came to your same conclusion, but I still run a hybrid setup that allows me to keep the router (for now), and a NAS for backup (3-2-1) and some local services. I run a dedicated server from Hetzner for "always on" services, so that the hardware, power redundancy and operational toil are offloaded. I gave up long ago on email: any hosting service will be way better than me doing it - I know I can do it, but is it worth my sanity? Nope.
> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this.
A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies.
Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
> I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime
Can you share more about this? I have a APC Back UPS PRO USV 1500VA (BR1500G-GR) and it would be nice to know if this is possible with that one as well.
That UPS eventually died, and I'm not sure if it was because it was hooked up to a larger battery than it was designed for, but it's still only 12 volts so I don't think the electronics would notice. What they may notice is extended run-time in the event of a power failure.
It was a crude mod. Take the cover off and remove the existing little security alarm battery, use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS), and feed the battery cables out through the hole. I probably got some additional cables with appropriately sized terminations to effectively extend the short existing ones (since they were only designed to be used within the device). And then connect it up to a car battery.
Cover any exposed metal on the connectors with that shrink rubber tubing or electrical tape. Be very careful with exposed metal around it anywhere, especially touching the RED POSITIVE pole of the battery. Get a battery box - I got one for the big-ass AGM battery.
Test it out on a laptop that's had it's battery removed or disconnected that, just in case, you don't care too much about losing.
Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity.
Personally, I think it's safer a less hassle to go for a LiFePo4 (LFP) Power Station style device that has UPS capabilities. LFP batteries have 3,000-ish cycle lifetimes, which could be nearly ten years with daily use.
No, don't do it. I understand his thought process because they are both 12v batteries with more capacity, but car batteries are made for high burst of energy which a car engine ignition requires, whereas UPS batteries are made for slow drains. Also, these UPS are made for charging battery cells in a certain way, if you start to stack a bank of batteries of the same model in parallel hoping for more capacity, even then its a problem for the UPS's charger, they won't charge evenly and eventually becoming a problem.
This really doesn't seem like something one would want to mess around with if they don't know what they're doing (fire hazards and all...)
> I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet
If your OS is using systemd, you can fix that pretty easily by adding an After=network-online.target (so the ExecStart doesn't even try to check if there is no networking yet) and an ExecCondition shell script [1] to actually check if nfs / smb on the target host is alive as an override to the fs mounts.
Add a bunch of BindsTo overrides to the mounts and the services that need the data, and you have yourself a way to stop the services automatically when the filesystem goes away.
I've long been in the systemd hater camp, but honestly, not having to wrangle with once-a-minute cronjobs to check for issues is actually worth it.
[1] https://forum.manjaro.org/t/for-those-who-use-systemd-servic...
I just use a small UPS to make sure all data is written to the drives properly before the battery runs out.
Do you have power outages often? Even if I have one, my services can come up automatically without doing anything, when the power is restored.
If anyone is looking to get started with a homelab at a good price, I can highly recommend checking ebay for a Dell Wyse 5070. They flooded the market for $50 and are likely powerful enough for many needs. They have a M.2 slot that support SATA. The 'extended' version also has space for a small pcie card and has a parallel and 2 serial ports for a blast to the past.
I built mine around an N150 board off of aliexpress. 6 SATA slots, 4x2.5G ethernet, 2x m.2 slots. Find a cheap second-hand case, a bit of RAM and you're ready to go. It's got hardware transcoding, handles 4K without breaking a sweat. And it consumes 6W!
Another cheap option are Fujitsu Futro's. They're meant to be thin server clients meant to operate larger server racks (I guess?). Anyway, they come with 4-8GB RAM, an SSD (most people upgrade them), and even have a PCI-e slot (depending on the model) to use with a 2.5 or 10 Gbit ethernet card, for example. Around $50 on ebay.
For a bit more money, Optiplex Micro / Lenovo Tiny / HP Mini series with at least 8th gen i5 are a good option too. Can be found from Ebay for about 70 - 120 USD, much more powerful than Wyse 5070 while still quite power efficient (about 10W idle, as opposed to 5W of Wyse). Usually they come with one NVME, one SATA 2.5" slot, some premium models even with PCIE.
All good options, just noting that based on some searches I made they all seem to lack serial ports compared to the Wyse if that is something you care about (I personally do). There could be variants out there though with serial ports and if would be happy to hear about them and even more happy if there are fanless variants/alternatives for those of us with very limited space at home and a need to avoid noise.
I don't know about fanless, especially with an i5 as opposed to n-something.
But not all those minis are the same. G4 (intel 8th gen) and G5 (intel 9th gen) HPs are horrendous. The fan makes an extremely aggravating noise, and I haven't found a way to fix it. Bonus points for both the fan and heatsink having custom mounts, so even if you wanted to have an ugly but quiet machine by slapping a standard cooler, you couldn't.
G6 versions (intel 10th gen) seem to have fixed this, and they're mostly inaudible on a desk, unless you're compiling something for half an hour.
Another tip: second hand gaming PCs! They can be incredibly cheap and powerful due to upgrade cycles just make sure to put a raid 1 on it as second hand gamer gear might be less reliable.
i bought dell wyse 5070 for building talos cluster using proxmox. pretty great and you can upgrade ram to 32gb
I would say raspberry pi 5. cheap, small and widely used so much of the stuff is already done many times
Looking at Raspberry Pi prices inside EU, I can get 8 core laptops for a cheaper price, with display, dGPU et al.
No idea what happened, but Raspberry Pis are super expensive for the last couple years, which is why I decided to just go with used Intel NUCs instead. They cost around 80-150EUR and they use more electricity but they are a quite good bang for the buck, and some variants also have 3x HDMI or Gbit/s ethernet or m2 slots you can use to have a SATA RAID in them.
Is it better than a n100 setup from China? When you factor in the storage, power supply, case, (fan), and so on?
No. The main pain point with RPi's is that they're SD card based – which are slow and prone to failure. Configuring an SSD to be used as the main storage has also been a pain in the past (not sure if that's changed recently).
With an n100, you get a better, more upgradable system for around the same price and same power usage. On top, you will also have an x64 system that isn't limited to some ARM quirks. I made the switch n100's over a year ago and have had no issues with them so far.
What if power consumption is taken into account? Are there any devices in that category that are ok to leave on 24/7 ?
I have a somewhat bigger machine that hosts my homelab, an HP 800 G2 SFF. It takes "normal" components, so can ben modified. The only custom thing is the PSU, but the standard one is good enough for my needs. Bonus points for not requiring an external power adaptor.
It has an i5-6500, 32 GB RAM (16 + 2x8 DIMMs), 2 SATA SSDs and a 2x10Gb Connect-X3. It runs 24/7 hosting OpnSense and HomeAssistant on top of KVM (Arch Linux Hardened – didn't do anything specific to lower the power draw). Sometimes other stuff, but not right now.
I haven't measured it with this specific nic, but before it had a 4x1Gb i350. With all ports up, all VMs running but not doing much, some power meter I got off Amazon said it pulled a little over 14W. The peak was around 40 when booting up.
Electricity costs 0.22 €/kWh here. The machine itself cost me 0 (they were going to throw it out at work), 35 for the nic and maybe 50 for the RAM. It would take multiple years to break even by buying one of these small machines. My plan is to wait out until they start having 10 Gb nics and this machine won't be able to keep up anymore.
Quick search online tells me ~5W for the Dell Wyse 5070, which does not sound unrealistic as I have similar boxes that draw ~10W. So, 32 to 62kWh per year and then we have ~USD 6.5 to 13 per year assuming 20 cents per kWh which another online search told me was reasonably realistic for the US.
Tangent, but it's always crazy to hear what other countries pay per kWh compared to the 0.4€/kWh in Germany.
Yeah, and Germany is expensive compared to the Nordics. 6.35 c/kWh right now in Finland, 2.54 c/kWh average over the last 30 days.
(clarification: that's euro cent, so 0.0635€ etc)
Bay area, California: $0.61 base, $0.80 from 16:00 to 21:00.
And in Iceland the average is around 0.07€/kWh.
Yea, I own Wyse 5070 extended, and measured around 5W from the wall when nothing attached to the PCIE slot.
curious to know what you would use this for?
local dns, static site hosting, local apt cache, various other network services (unifi controller if you've got those APs for example), remote/headless dev machine (maybe not for kernel or bigcorp java development), or whatever else you want. mail if you want. Anything :)
Those little thin clients aren't gonna be fast doing "big" things, but serving up a few dns packets or whatever to your local network is easy work and pretty useful.
I use it for media hosting. Backups (connected USB disks), home assistant, syncthing
Even these low-power CPU's are surprisingly capable. As an example of more fancy thing, one could slap in some external storage, install Jellyfin, and run their own local streaming service off such a machine. The CPU is modern enough for efficient hardware transcoding of a stream.
While i get the whole homelab thing is exiting and a great learning experience, it's simply not worth the time and effort for the majority of people.
You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime).
In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever.
A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period).
It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks.
Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money.
Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.
But on a homelab you can host any service you want and start/stop it whenever you need to. Sure cloud storage might cost less in the short-term, but if you need more storage or more services, a selfhosted option is far cheaper.
VPS are very expensive for what you get. If you have the capital, doing it yourself saves you money very quickly. It's not rare to pay $50 for a semi-decent VPS, but for $2000 you would get an absolute beast that would last 10 years at the very least.
With Docker, maintenance is basically zero and unused services are stopped or restarted with 1 command.
How many services do you need ? And how much CPU power do those services need ?
I've also self hosted for decades, but it turns out i don't really need that much, at least not public.
I basically just need mail, calendar, file storage and DNS adblocking. I can get mail/calendar/file storage with pretty much any cloud provider (and no, there is no privacy when it comes to mail, there is always another participant in the conversation), and for €18/year i can get something like NextDNS, Control D, or similar.
For reference, a Raspberry Pi 4 or 5 will use around 50 kWh per year, which (again in europe) will translate to €15/year. For just €3 more per year i get redundancy and support.
I still run a bunch of stuff at home, but nothing is opened to the public internet. Everything runs behind a Wireguard VPN, and i have zero redundant storage. My home storage is used for backing up cloud storage, as well as storing media and client backups. And yes, i also have another cloud where i backup to.
My total cloud bill is around €20-€25/month, with 8TB of storage, ad blocking DNS, mail/calendar/office apps and even a small VPS.
I did not do the price calculations (in France, and I prefer not to know :) but I host many things except for mail and calendar (mail is tricky to host). Of these 29 services, I use maybe 4 daily and 15 monthly. They are well protected, easy to maintain, and serve family and friends.
Not to mention that I love them.
As a Northern European (Finland) I can tell you that the electricity cost here for last year was closer to 0,1 € per kWh including the transfer fee and taxes. Additionally, more than 40 % of houses here have electric heating. The heating season starts in early autumn and ends in late spring, lasting 8 to 9 months depending on the year. As the electricity used by the device is turned into heat, during the heating season running it costs nothing.
Yeah, northern scandinavia has plenty of renewable energy.
As for electric heating, that is true in 1:1 heating scenarios, but i assume you guys are also using heat pumps these days, and while you still get heat "for free", it will not be anywhere as efficient as your heat pump.
Yes, it's probably peanuts in the grand scheme of things, i know our air to water heat pump in Denmark uses around 4500-5500 kWh per year, so adding another 100 kWh probably won't mean much.
The maintenance time is a bit overestimated if you keep it simple.
On my homelab, I update everything every quarter and it takes about 1 hour, so 4 hours a year is pretty reasonable. Docker helps a lot with this.
And I’ve almost never run into trouble in years, so I have very few unexpected maintenance tasks.
EDIT: I am referring to a homelab that is only accessible for private purposes through a VPN.
As a bare minimum, you should update your server and docker images daily, or at least whenever there's an update (which you won't know unless you check).
If you only access your homelab over VPN or similar, then by all means, update whenever you feel like it, but if you expose your services to the internet, you want to be damned sure there are no vulnerabilities in them.
The internet of today is not like it was 20 years ago. Today you're constantly being hammerede by bots that scan every single IPv4 address for open ports, and when they find something they record it in a database, along with information on what's running on that port (provided that information is available).
When (not if) a vulnerability for a given service is discovered, an attacker doesn't need to "hunt & peck" for vulnerable hosts, they already have that information in a database, and all they need to do is start shooting at their list of hosts.
You can use something like shodan.io to see what a would be attacker might see (can check your own IP with "ip:xxx.xxx.xxx.xx".
Try entering something like Synology, Proxmox, Truenas, Unraid, Jellyfin, Plex, Emby, or any of the other popular home services.
> which you won't know unless you check
RSS feeds FTW
Sorry, I should have mentioned that my services are only accessible through a VPN. Otherwise, I completely agree with you.
Have you seen the prices for Google Drive et al? The NAS setup you describe (which I wouldn't consider worth the money for that little space) is what, 12GB with 1 parity drive?
Google One for 10TB is 274,99€/mo (at least in my country) so you'd make the entire nas price and subscription cost within a few months, let alone years.
There just aren't compelling public cloud for large sizes (My NAS is 30TB capacity and I'm using 18 right now) and even if you go the more complex loops with like S3 and whatnot you still get billed more than it's worth. Public cloud is meant for public files, there's a lot of costs you're paying for stuff you don't need like being fast to access from everywhere.
Google Premium storage, is $100/yr (paid annually) for 2TB of storage.
Even at the high end estimate the homelab is giving you several times the storage for the same cost.
do you need more space than 2TB ? (excluding things you've downloaded from the internet)
Very few people i know has use for that much storage. Yes, you can download the entire netflix catalog, and that will of course require more storage, and no, you probably shouldn't put it in the cloud (or even back it up, or use redundant storage for it).
Setting up your own homelab to be your own netflix, but using pirated content, is not really a use case i would consider. I'm aware people are doing it, and i still think it's stupid. They're saving money by "stealing" (or at least breaking laws), which is hardly a lifehack.
I know many people that would fill that space with home videos from phones and digital cameras. Millennials with kids especially.
You can fit A LOT of photos and videos in 2TB.
My wife is a professional photographer, and while we do archive most of her RAW files somewhere else, pretty much everything HEIC, JPEG or any other compressed format lives in our main cloud.
We have 2.2TB in total for “direct storage”, and we’re currently using around 1.5TB, and that’s including myself and our kids.
My personal photo library has just short of 90,000 photos, and about 5,000 videos. My wife’s library is roughly twice that. I have no idea how many photos the kids have, but they each take up around 200GB for photos.
And then we have backups, which actually take up about 1TB per person, mostly because that’s the space I’ve allocated for each, so history just grows until it’s filled. Photos ideally won’t change much. We backup originals along with XMP metadata for edits, so the photos stays the same, and changes are described in easily compressed text files. Backups of course also have deduplication enabled.
a NAS has like 10000x more storage than google drive and is also way faster locally.
How much space do you realistically need high availability, redundant storage for ?
For my personal use case, that involves photos and documents, all things i cannot easily recreate (photos less so). Those are what matters to me, and storing them in the cloud means i not only get redundancy in a single data center, but also geographical redundancy as many cloud providers will use erasure coding to make your data available across multiple data centers.
Everything else will be just fine on a single drive, even a USB drive, as everything that originated on the internet can most likely be found there again. This is especially true for media (purchased, or naval aquisition). Media is probably the most replicated data on the planet, possibly only behind the bible and IKEA catalog.
So, back to the important data, i can easily fit an entire family of 4 into a single 2TB data plan. That costs me somewhere around €85 - €100 per year, for 4 people, and it works no matter what i do. I no longer need to drag a laptop with me on vacation, and i can basically just say "fuck it" and go on vacation for 2 weeks.
I just need you to make comparisons that are fair.
The premium plan from Google has 2 TB and costs about the same annually as the electricity for the NAS that the GP comment suggested for comparison (at 100% usage). So at the same ONGOING cost (not even counting initial investment), the NAS has 8 times more storage. 16 times if you assume it will be mostly idle. Except if you want high availability with RAID, then you're back to 8 times. And haven't yet thought about backups.
All this assuming that you even need that much storage, which most people definitely do not.
It’s great to learn on, and if you happen to have a place with free electricity then even more fun :)
It’s also an excuse for me to stay in most summer days.
Encrypted data on the servers is only useful if your server is just dumb storage. I want the server to actually do something, e.g. serving media, running home assistant etc.
OP has admittedly over-engineered their setup. Depending on your goals (cost, speed, space, autonomy), there are less-rigorous solutions for the layperson.
I, for one, don't want to have Google, etc. as a dependency[1], so I will pay some energy cost to do that.
1: see: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
>Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you
Who are you to tell people how to spend their time? Let people have hobbies ffs
A server will never love you back.
Neither will the majority of hobbies for self-enrichment.
It isn’t unreasonable to want some alone time.
Building a homelab is an awesome way to learn a lot of things.
I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people.
I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.
The encryption is interesting but I wouldn’t call this over engineered at all, in fact it’s rather basic compared to a lot of homelabs I see people building particularly where people are doing K8s or similar over multiple machines.
Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.
I have a setup that is perhaps not as robust, but where my primary aim was that I should be able to incrementally encapsulate the parts. https://wiki.roshangeorge.dev/w/One_Quick_Way_To_Host_A_WebA...
As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case.
When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.
Your VPS provider likely uses servers with ECC RAM, this home server doesn't. For most people it doesn't seem to matter but for me it does - a home server where I store my data needs to have ECC RAM.
Seconded, but hard to find for small boxes. I have seen in-band ECC on Asus Nucs, but that is as good as it gets from what I can tell.
There is the new Minisforum N5 Pro, which supports DDR5 ECC RAM. I'm keeping an eye on it.
totally agree. That's why my homelab storage server(s) are 2nd hand enterprise machines. They come with ECC.
This was exactly a use case I had in mind when building https://canine.sh -- also uses k3s as a provider, and provides a Heroku-like devex.
How to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying
I've got basically raw internet coming in to my OPNSense device, although I had to request certain ports to be removed from the ISP's by-default-blocked policy, since I host a mail server - but the ISP is fine with this, they have a form for it, super easy.
Some family members are behind CGNAT, and I'm not sure if their ISP has the option to move out from behind that, but since they don't self-host it's probably slightly more secure from outside probes. We're still able to privately share communications via my VPN hub to which they connect, which allows me to remotely troubleshoot minor issues.
I haven't looked into cloudflare tunnels, but haven't felt the need.
What do you mean by "one port at a time"?
I run cloudflared on one machine, and it proxies one subdomain to one port, and another to a unix socket (could have been a second port, no pb).
This is such a great post. I have a small collection of posts for inspiration in creating my homelab and this is getting added to it. Current have a Pi 4 with PiHole and a Beelink. Going to add one or two more machines.
Yeah good luck with that one.
I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments.
One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door.
At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me.
I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along.
I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks.
I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.
I can relate to this. I still run my own x86 box as a router to have the AP controller, but I'm strongly considering dropping this.
I need to update it and patch it, hoping nothing goes wrong in the process. If something breaks I'm the only one that can repair it, and I really don't want to hear my wife screaming at me at 7am when I wake up.
Eventually I came to your same conclusion, but I still run a hybrid setup that allows me to keep the router (for now), and a NAS for backup (3-2-1) and some local services. I run a dedicated server from Hetzner for "always on" services, so that the hardware, power redundancy and operational toil are offloaded. I gave up long ago on email: any hosting service will be way better than me doing it - I know I can do it, but is it worth my sanity? Nope.