I’m thinking about starting a self hosting setup, and my first thought was to install k8s (k3s probably) and containerise everything.
But I see most people on here seem to recommend virtualizing everything with proxmox.
What are the benefits of using VMs/proxmox over containers/k8s?
Or really I’m more interested in the reverse, are there reasons not to just run everything with k8s as the base layer? Since it’s more relevant to my actual job, I’d lean towards ramping up on k8s unless there’s a compelling reason not to.
Unless you have multiple systems, I don’t think k8s will yield much benefit over plain docker.
So, if I plan to build a pi cluster I should get familiar with k8s?
I’m running a 3 pi cluster with k3s at the moment. The main benefit I’ve found is that all my pis run exactly the same software setup as a base so it’s easy to add new ones or replace/update one. I use a deployment management application to push my deployments too which means it’s super easy to redeploy everything if something goes funky.
The basics can be useful there. The whole idea with k8s is to be able to run applications across multiple hosts in a given fleet. Your cluster can be that fleet! :)
Also k8s is in high demand in the sector, so those are good skills that could be turned into $$
I get why too. I’m a full stack (including devops) software engineer, and docker/k8s is just completely opaque to me. I’m not sure why, but I really just can’t wrap my head around it. Thankfully my current company has a devops team that takes care of it, but jeez
Tbh those stuff aren’t really intuitive. But, as was my case for instance, that’s something that can be “easily” learnt as a hobbyist like us. And when you understand those concepts, at least from an abstract point, my stance is that you can become a better dev/ops/sys :) I strongly advice anyone in the field to at least play a little with Docker/containers to grasp what it is.
I can’t even get my head around Traefik, let alone Kubernetes (and k3 vs k8???)
Personally I always use containers unless there is a good reason to use a VM, and those reasons do exist. Sometime you want a whole, fully functional OS complete with custom kernel, in that situation a VM is a good idea, sometimes a utility only comes packaged as a VM.
But absent of a good reason, containers are just better in the majority of cases
What I did is install proxmox on the bare metal, setup a vm in which I put the containers.
Proxmox itself stays (almost) completely stock. The only changes I’ve made to it were to add the NUT client package so it could gracefully shut down if my NUT server indicates that the UPS is running out of power during an outage.
In your VMs you can do whatever. Setup OMV, or a stock Ubuntu or Debian vm and install your services on the VM or use Docker/Podman. Setup Fedora CoreOS or IoT vms and host all your services in Podman containers.
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
You can also spin up new vms as needed to just try to fuck around with new techs or just a new way of setting up your home lab. It gives you a ton of flexibility and makes backing stuff up way easier.
Another great thing you can do is if 3 years down the line you are looking to replace your server hardware with some newer or more powerful stuff you can just add the new device as a node to the cluster. Then you can migrate all your existing VMs over to your new hardware and decommission your old one with very little to no downtime on anything.
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
Hopefully you put a giant asterix by this point. You need the snapshot AND the original backup. Snapshots are only diffs and can’t survive without their base backup.
This is my exact setup as well. Proxmox with one beefy vm dedicated just to docker and then a few other vms for non docker workloads (eg, home assistant, pihole, jelltfin). I can probably run those in docket as well, but the to worked better as vms when I set them up
Appreciate your take on this and specifically mentioning that you have a VM for Home Assistant. That was a lightbulb moment for me as I like how easy it is to manage updates as an OS install rather than in a Docker container. If I ever get around to rebuilding my server architecture I’m definitely going to do this!
I have a similar setup, but 2 VMs on each of my 2 servers, then on server 1, I have VM A running one test K3s node and VM B running one live (Production) K3s node with the same on server 2, so I can take one server full down for maintenance, but keep my test and live sites running. It’s way overkill, but allows me to learn about how to set up and maintain resilient systems. One day, I’ll do the same for my network :-(
Container processes are just ordinary linux processes, so they don’t need extra overhead (cpu and ram reservation) to run, which means your machine can run more of them. If you have a machine with 32GB of ram, can probably run 15 VMs with 2GB of ram each where the actual app running inside the VM might only consume about 50% of the VM ram, or you can run them as container and they all would just consume 15GB of ram, leaving you extra to run more containers. I found this to be ideal for self hosting because all apps are your personal apps so interprocess isolation is not as important compared to running in public cloud.
I’ve always been unclear of why people choose to run VM’s. I would think you’d want to try Docker first, LXC second, and VM only in the last instance, if you need to emulate a different architecture? But if the stuff you need to run has been ported to your server’s architecture why add the overhead?
There’s been some nasty buggery with avahi instances on containers clashing with host ones in the past
Some programs just don’t like to run without access to parts to your system like /proc /sys and /run.
Rather than bother with crafting bespoke permissions, non-default cgroups and elevated rights for certain containers, I’ve definitely opted for just installing a VM.
It was always a time/functionality choice, and not one I make often - crafting the right solution is always better; but I have done it
Containers, unless you have a specific need for a VM.
With a VM you have to reserve resources exclusively. If you give a VM 2gb of ram, then that’s 2gb of ram that you can’t use for other things, even if the guest OS is using less.
With Containers, you only need as many resources as the process inside the container requires at the time.
I think it depends on your scale. If homelab stuff docker is awesome IMO.
VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.
Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.
They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.
Why not use both? I have PVE installed on all of my hosts and then use k3s/docker in VMs. If there ever is anything you don’t want to or just can’t deploy as a container (e.g. opnsense, hassio, truenas, windows [for whatever reason you might have]), you can just spin it up as a VM and not worry about adding and maintaining another physical machine
If you are using PvE for linux “VMs” those probably aren’t actually VMs but LXC containers. And if you are running docker in one of those, you’ve got containers in your containers.
Welcome to the club.
Yo dawg. I heard you like containers.
My brother in Christ, how would one confuse a VM with an LXC in Proxmox? They couldn’t be more clearly labelled as different things than they already are. But don’t let this distract you from the fact that in 1998, The Undertaker threw Mankind off Hell In A Cell, and plummeted 16 ft through an announcer’s table.
#NeverForget
You can try casaOS.
https://github.com/IceWhaleTech/CasaOS
Its something like k8s. But it is easy to use and works very well with docker containers.
I’d suggest looking into k8s. It’s definitely a bit more complex on the start, but so much more power once you get to the details. VMs you don’t share the base OS layer and the hardware, you have to pre-define the resources you need per app in a more constrained manner, while containers can move freely in their little sandbox to pickup whatever it needs.
It is also much easier to manage replicas, upgrades, scale and a bunch of other things once you are using containers and an orchestrator like Kubernetes. Let me know if you need any help/insights. I’ve been trying to post more videos/answers about things that could be complicated.
Removed by mod
Why not both?
Like many others here, I went with Proxmox as the base host. But most of my services are Docker containers , running in a “dockerVM” on top of Proxmox.
Having Proxmox as the base is just so flexible, which is very handy for a homelab.
- For instance I set up a VM with Wireguard back when Wireguard had only just been merged in to the mainline kernel, without affecting the other
- You can have separate VM for docker testing, and docker production
- You can run multiple VMs for multiple Kubernetes hosts, to try it out and get your feet wet without affecting the “production” containers
- If you get additional servers, you can just migrate those Kubernetes VMs
- You can run Windows VM should you need, and BSD (and thus pfSense/opensense or TRUE AS)
- You can run a full graphical environment if you want
- Proxmox has easy setup for firewalls for each VM
- I have a VM running a legacy bare metal system (from the same server now running proxmox) that I’ve been slowly de-commissioning piece by piece
What is your system backup solution like? Having it separated seems convenient for that since you can just back up the vm storage somewhere I’m guessing?
Not OP, but similar setup (Proxmox with docker on a VM). The VM (plus a few LXCs) are backed up daily using the backup built into Proxmox, and those backups are mirrored to the cloud with rclone.
Proxmox Backup Server: Incremental de-duplicateed image backups of the whole VM, with possibility of individual file restore. It’s like magic
For the legacy bare metal system I have rsnapshots of the data folder (set it up ages ago, and never changed it)
An nginx LXC container has a single static backup of the container, with the nginx config file stored in a git repo
I personally really, really like (Docker) containers and I host most of my stuff with it, on a Raspberry Pi and on (free tier) Oracle Cloud VPS’s. I also plan to (re)install Proxmox on a spare old laptop and run some stuff in VMs on that (namely Home Assistant) and might try a NixOS server too.
So really, use both. Use the right tool for the job. And you can also run containers in VMs and even use Ansible to configure everything with playbooks, allowing you to re-run said playbooks when things go wrong.
My backup solution is rsync and so I really like docker-compose since it usually means there is zero config for restoration of backups on a new computer besides installing docker-compose (which is usually one line on the terminal).
I have a pretty low power server at home (Pentium G4560), and the previous one was even slower J3160, so I don’t want to unnecessarily hog the CPU with a VM, and the few services I need at home run perfectly fine in containers.
I run pihole, unbound, wireguard, plex, unifi controller in containers, and I run some additional services directly on the host (samba, transmission).
I have a Windows VM on my Windows PC for work, so it’s isolated from my main rig (various VPN clients and work files etc), and if I needed some Linux stuff on my Windows PC I’d also run a VM, but more VMs also mean more updating and patching, which is much easier with containers.
