Assumptions
- You want to have a secure, flexible, and high performance home network
- Secure: you control the (mostly) open source software stack
- High performance: your gear isn’t going to be limiting your realized performance in any meaningful way. A household of hungry Internet users can take maximum advantage of the connectivity you’ve purchased
- Flexible: You’re able to self host a wide variety of workloads
- You're willing to spend some reasonable money getting there (~$2000 USD all in for this setup, although you likely already have most pieces or can do it incrementally)
- We're not going to waste money here building a "home lab" just for practice, but we are going to buy modern equipment that is somewhat future-proofed.
- You're results oriented: We are going to use a lot of "off the shelf" stuff here, because it works well and saves time.
- Yes, you can hand build your own server PC etc but it takes time, and component built PCs can be flakey. Unless you have a lot of extra parts laying around to troubleshoot, you can consume a lot of time for little cost savings.
Step 1 - Dump your ISP's modem
Comcast is charging a ridiculous rental fee for their cable modems, now up to $15/month. Buying your own cable modem, something like this Arris model will pay itself back within months.
But there's a much deeper reason to dump your ISP's cable modem: You have no idea what it's doing or how secure it is. Generally, your ISP controls any firmware on the device. There could be outstanding security bugs or "features" enabled that you'd rather not have.
Take for example, Comcast's XfinityWifi network. You may have heard of it. If you're a Comcast customer, you get access to the XfinityWifi wireless network wherever it may be found, just use your Comcast username & password. But you may be wondering, how is this Wifi SSID seemingly all over the place? Well it's because when you use a Comcast cable modem + WiFi router combo, the device quietly uses it to extend the XfinityWifi network as well.
Comcast swears up and down that this is totally transparent to you, but do you really want to pay a monthly fee for the privilege of enabling Comcast's business model?!? Also, if you are in a location where the cable infrastructure can barely support the speeds you've purchased, there's no way having extra Wifi freeloaders on your connection isn't going to have some performance impact.
Aside 1a - Bridge mode for your modem
To maximize flexibility, don’t buy a cable modem with integrated wireless. We are going to have a separate wireless solution, so any wireless functionality here is just going to get in the way and add unnecessary cost.
We are going to run our cable modem in “ethernet bridge” mode. It’s just going to pass packets transparently from one interface to the next – no assigned IP of its own on your network.
Step 2 - Edge / Gateway Router
We are going with a Protectli Vault to implement our edge router. We want it to be fast and reliable. The Protectli is effectively a solid state PC – no moving parts, flash storage, heat dissipation via the case – no fans. Since it’s a PC, we can run any OS .. but our choice for a router is PFSense. It’s open source and based on FreeBSD. It comes with a pretty web based UI and has a lot of detailed options. We’ll run WireGuard VPN on this device, but not much else. We don’t want to do anything on this device that’ll increase latency or increase potential attack surface.
Aside 2a - 2.5Gig Ethernet
1 gig ethernet has been ubiquitous now for years. 2.5 gig ethernet is starting to become more commonly available and typically adds little to the effective cost of most devices. You’re getting 150% more speed and future proofing your setup a bit, so it’s well worth trying to get. As of early 2023, for a lot of devices, only the most recent generation comes with 2.5 gig, so if you try to save a few bucks and buy last year’s model, you might miss out. The Protectli router above is a good example: only the most recent models come with 2.5 gig but are similarly priced to the previous era models which only have 1 gig.
There’s also 10gig networking but it’s considerably more complicated and costly. Wolfgang’s video below gives a good overview (1:30 - 6:30): While 2.5 gig can likely re-use your existing cabling, 10 gig comes with a lot of special considerations (fiber likely). Avoid for now.
Step 3 - Ethernet Switch
You likely already have this, but you need an Ethernet switch as opposed to a hub. Optimally one with 2.5 gig ports. Lots of ways to go here: features, number of ports etc will vary greatly depending on your use case. A few thoughts:
- It’s probably easier to buy multiple switches for different areas of a large home rather than trying to cable everything back to one central point. For e.g, in my house I have a second 8 port switch specifically for the media area, since every game console, cable box etc benefits from an Ethernet uplink.
- Don’t cheat yourself here: Wire everything you can in your house. WiFi networks tend to get flaky and performance degrades when you have a lot of devices connected, particularly older devices. If you are doing any smart home stuff, you are likely to already be forced into having 10s of devices connected to your WiFi at all times (smart light switches, security cameras etc). Anything you can wire via Ethernet will be much more reliable (especially video streaming devices).
- Consider PoE ports (power over Ethernet), which allow you to power & connect some devices like WiFi routers and security cameras with a single cable.
Step 4 - WiFi Mesh Router
As discussed above, we are going with a standalone WiFi solution. Since we already have a dedicated PFSense edge router, we don’t care much about the WiFi router firmware (and its capabilities or lack thereof) beyond the WiFi space.
WiFI routers seem to be hit or miss – you’ll hear about specific models that are “known good” but then they are “refreshed” with cost reduced versions with less CPU and RAM and don’t perform as well. Firmware revisions also play a big role – be careful about upgrading (or not upgrading) before doing some research on your particular model.
I’ve had good luck with the ASUS ZenWiFi AX6600 series. You want a mesh system so you can add nodes as needed to extend coverage. And you want the option for wired/Ethernet “backhaul”: the WiFi network should be extended on a wired backbone. If you don’t use Ethernet (and the ad copy on these mesh systems make it seem like it’s totally unnecessary), most mesh systems will use some channels in the 5GHz spectrum to run a private backhaul network in addition to your SSID. If your wireless situation was marginal to start with, this is unlikely to work well.
Step 5 - Network Attached Storage (NAS)
Again lots of options here. As a minimum, you’re going to want some kind of basic file share with a reasonable backup story. You can build yourself here, but the Synology devices work well, don’t consume much power (see UPS section below), and are a cinch to set up. Something simple like the 2 bay NAS DiskStation DS220jworks. Buy the shell, then buy a couple of SATA harddrives to stick into it. 220J is getting a bit long in the tooth these days and doesn’t have a particularly impressive hardware spec, but I’d argue that we are going to do more complex workloads like media transcoding elsewhere. We just need a cheap and reliable file server. Don’t worry too much about getting lots of bays and trying to do something like RAID5 at home to improve data reliability. Instead, just configure the Synology to use BackBlaze as an external cloud backup. BackBlaze is surprisingly cheap, cheaper than AWS S3 even. At $5/per TB/per month, it just doesn’t make sense to host your own backups (which can fail, and are also subject to the “all in the same house burning down” problem).
Step 6 - VM & Docker Host - Intel NUC
You want a small, reliable, and power efficient PC to host all of your self-hosted services. Enter the Intel NUC line of computers, which is pretty much Intel’s equivalent to the Mac Mini. Mac Mini’s are pretty nice, but running anything other than MacOS on them is annoying.
There’s a wide variety of Intel NUC form factors and options to choose from. The kit versions basically come with the case & CPU, and you bring the memory and storage devices. These seem to be the best deal for a VM host.
In late 2022, a sweet spot seemed to be the 12th gen units, like Intel NUC 12 Pro Kit (RNUC12WSHi5). You get a Core I5 CPU @ 4+ GHZ and 2.5 gig ethernet.
To round out the kit you’ll want to get 64 GB RAM (the max this system will take – and yes this weird laptop memory format is what the NUCs need), and some kind of SSD. We want a lot of RAM because each VM needs dedicated memory. Give each kid a virtual Linux box to mess up :).
I had trouble finding these on Amazon, but NewEgg works as well. Make sure they are throwing the power cord in as a bonus, or buy it separately :(.
Step 7 - Uninterruptible Power Supply (UPS)
Protect all these expensive devices that you’ve purchased from brown-outs and flickering power in winter storms. In a short power outage, not having everything reset on you is awesome, especially for self-hosted services. It also gives you time for a clean shutdown in longer outages. To just protect your core network assets, something like an APC UPS 1500VA will work well. Buy the better models that actually regulate and clean-up the output voltage, especially if you live in an area with flaky power. Typically I only UPS protect critical network components, and I’ve selected them here to be somewhat power efficient so that battery backing them isn’t too expensive.
Sizing UPSes is a bit annoying, as VA (Volt Amps) do not translate to watts directly, as it depends on the types of electrical loads connected. APC has a load calculator hidden in the details section for each of their models. Typically though, a 1500VA UPS like the APC unit above will have the following runtimes:
Watts Run time
100 1hr 17mins
200 39 mins
300 24 mins
400 17 mins
500 12 mins
600 9 mins
You can buy something like a Kill-A-Watt meter to measure real life power draws for each device. Here’s what I measured with the recommend devices above:
Arris Cable Modem 9W
Ethernet Switch 4.3W
ProtectLi Router PC 6.7W
Synology NAS 21W
WiFi router (ASUS) 10.5W
Intel NUC ~10W (idle)
--------------------------------
TOTAL ~62W
Running only ~60W off this UPS, you’ll get 1hr+ of runtime.