Pros and cons of Proxmox in a home lab?

Hi all. I was curious about some of the pros and cons of using Proxmox in a home lab set up. It seems like in most home lab setups it’s overkill. But I feel like there may be something I’m missing. Let’s say I run my home lab on two or three different SBCs. Main server is an x86 i5 machine with 16gigs memory and the others are arm devices with 8 gigs memory. Ample space on all. Wouldn’t Proxmox be overkill here and eat up more system resources than just running base Ubuntu, Debian or other server distro on them all and either running the services needed from binary or docker? Seems like the extra memory needed to run the Proxmox software and then the containers would just kill available memory or CPU availability. Am I wrong in thinking that Proxmox is better suited for when you have a machine with 32gigs or more of memory and some sort of base line powerful cpu?

fortera ,

I use Proxmox/virtualisation because I want to be able to run services within their own OS. I've got a VM dedicated to docker both at home and in my colocation, since a lot of services I'm happy to just chuck on there, but there's others with more complex setups, and other services/systems that just running them in docker isn't an option.

Pika , (edited )
@Pika@sh.itjust.works avatar

I'm currently running proxmox on a 32 gig server running a ryzen 5600 G, it's going fine the containers don't actually use all that much RAM and personally I'm actually seeing a better benchmarks than I did when I just ran as a Bare Bones Ubuntu server, my biggest issue has actually been a larger IO strain than anything, because it's a lot more IO heavy now since everything's containerized. I think I easily could run it with a lower amount of ram I would just have to turn off some of the more RAM intensive items

As for if I regret changing, no way Jose, I absolutely love the ability of having everything containerized because I can set things up how I want it when I want it and if I end up screwing something up configuration wise or decide that I no longer need that service I can just nuke the container without having to remember well what did I install on this program so I can remove it and do other programs need this dependency to work. Plus while I haven't tinkered as much in this area, you can hard set what resources you want a lot to each instance, so if you have a program like say a pi hole that you know is never going to use x amount of resources to be able to appropriately work you can restrict what it can do so if something does go wrong with it it doesn't use all of your system resources

The biggest con out of it is probably having to figure out how to do the networking side because every container is going to have a different IP address, I found using a web dashboard is my friend because I can have heimdel tell me where all my services are and I just have to click the icon to bring me to the right IP address, it took a lot of work to figure out how it's operational and how to get it working, but the benefits I've gotten of having it is amazing. Just make sure you have a spare disk to temporarily clone partitions to because it's extremly difficult to use existing disks in the machine. I've been slowly going one at a time copying it over to an external drive nuking the and then reinitializing the disc as part of the proxmox lvm and then copying the data back over onto their appropriate image file.

LordCrom ,

I use it on reclaimed hardware ... Works great for me. Has all the features you'd want for a home lab, and I run a few production hosts there as well

specialseaweed ,

I used Proxmox for awhile, then went to Unraid. I learned a lot using Proxmox but for ease of homelabbing, it’s tough to beat Unraid. It depends on what you’re wanting from your lab.

DeltaTangoLima , (edited )
@DeltaTangoLima@reddrefuge.com avatar

It all depends on how you want to homelab.

I was into low power homelabbing for a while - half a dozen Raspberry Pis - and it was great. But I'm an incessant tinkerer. I like to experiment with new tech all the time, and am always cloning various repos to try out new stuff. I was reaching a limit with how much I could achieve with just Docker alone, and I really wanted to virtualise my firewall/router. There were other drivers too. I wanted to cut the streaming cord, and saving that monthly spend helped justify what came next.

I bought a pair of ex enterprise servers (HP DL360s) and jumped into Proxmox. I now have an OPNsense VM for my firewall/router, and host over 40 Proxmox CTs, running (at a guess) around 60-70 different services across them.

I love it, because Proxmox gives me full separation of each service. Each one has its own CT. Think of that as me running dozens of Raspberry Pis, without the headache of managing all that hardware. On top of that, Docker gives me complete portability and recoverability. I can move services around quite easily, and can update/rollback with ease.

Finally, the combination of the two gives me a huge advantage over bare metal for rapid prototyping.

Let’s say there’s a new contender that competes with Immich. They offer the promise of a really cool feature no one else has thought of in a self-hosted personal photo library. I have Immich hosted on a CT, using Docker, and hiding behind Nginx Proxy Manager (also on a CT), accessible via photos.domain on my home network.

I can spin up a Proxmox CT from my custom Debian template, use my Ansible playbook to provision Docker and all the other bits, access it in Portainer and spin up the latest and greatest Immich competitor, all within mere minutes. Like, literally 10 minutes max.

I have a play with the competitor for a bit. If I don’t like it, I just delete the CT and move on. If I do, I can point my photos.domain hostname (via Nginx Proxy Manager) to the new service and start using it full-time. Importantly, I can still keep my original Immich CT in place - maybe shutdown, maybe not - just in case I discover something I don’t like about the new kid on the block.

That's a simplified example, but hopefully illustrates at least what I get out of using Proxmox the way I do.

The cons for me is the cost. Initial cost of hardware, and the cost of powering beefier kit like this. I'm about to invest in some decent centralised storage (been surviving with a couple li'l ARM-based NASes) to I can get true HA with my OPNsense firewall (and a few other services), so that's more cost again.

glizzyguzzler ,
@glizzyguzzler@lemmy.blahaj.zone avatar

Incus is way easier to work with than Proxmox, and it sits on your OS of choice instead of being the OS you must use. For home use it’s way easier to use with the web ui, it even has clustering if you want to go hard.

So you can install Incus when you want a VM/LXC container and not have to commit to a VM/LXC container OS from the start.

Also Proxmox free just had a bad update that björked some stuff if you updated when it was live. Proxmox free is rolling and apparently lacks basic sanity checks for updates.

Lemongrab ,

I remember updating (maybe a year ago now) and it making all my containers unaccessable.

glizzyguzzler ,
@glizzyguzzler@lemmy.blahaj.zone avatar

Incus or Proxmox (e.g., should I shift to Incus LTS or something?)

Lemongrab ,

If incus works for yoy, use it. Proxmox locks you out of the option to choose your base server distros.

glizzyguzzler ,
@glizzyguzzler@lemmy.blahaj.zone avatar

Ah, I was wondering which one you updated and it made your containers inaccessible!

Lemongrab ,

Sorry, misunderstood. Proxmox Free broke my containers on updating a while ago.

Now I use Docker-style application containerizing, but I think LXC (the base technology powering Incus/LXD) is useful in a number of situations and perfectly viable for use. I think Incus-containerized applications are easier to upgrade individually (like software updates of your apps, no need to recreate the container image) and gives a closer to native experience of managing. You do lose out on automated deployment of applications from widely available image sources like docker.io, but the convenience-loss is minimal.

glizzyguzzler ,
@glizzyguzzler@lemmy.blahaj.zone avatar

Good to know Proxmox’s bad updates are more pervasive than the latest bad update.

I have been able to install Docker in the LXC containers and pull images in with the normal commands. I do that container-in-container to get effectively rootless docker containers for stuff that I couldn’t figure out how to run rootless. So you don’t even lose out on docker if you’re determined! And as you said incus goes on any OS, you can docker just fine on the base OS of your choice and use incus for specific things!

rottedmood OP ,
@rottedmood@lemmy.linuxuserspace.show avatar

i played around with Incus yesterday on one of my VPSs that I really don't care about. I did find it really interesting. But im just wondering if its still a bit too much for what i use my home lab for (running local services like jellyfin, gitea, etc.). I would prefer to containerize all of those, but unless im misunderstanding something somewhere (and I probably am), running Incus to then run another instances of ubuntu 22.04 (or whatever) so i can set up Jellyfin or Gitea inside of that seems like a bit of overkill. However, as im trying to get nextcloud set up and running, having it exposed to the internet would mean spousal factor would go way up. Honestly they are about to kill me for using pihole, so having them have to turn on tailscale to connect to nextcloud, well sometimes it feels like its asking too much. So this is where running something in an isolated container would make me feel a bit more at ease. ah if only my spouse would just learn to turn on tailscale when they need it, but i don't see that happening any time soon.

glizzyguzzler ,
@glizzyguzzler@lemmy.blahaj.zone avatar

I do use it to hold internet-exposed things in LXC containers to sidestep having to figure out how to not run things as Docker root.

You do not need it for everything, but since it’s not an OS that makes it your everything, that’s ok! Run Docker containers as you need, put internet-exposed ones in an LXC container, put home assistant in a VM because it’s special.

TechnicallyColors , (edited )

I used Proxmox for a couple years and it's good if you run a lot of VMs or LXCs, but I found that I'm not really the target audience. I ended up only running one Debian VM for my Docker containers. It was fine, but I eventually felt that Proxmox added no value for me, and the end result was sacrificing some memory and performance from using virtio emulations for CPU/GPU/RAM/filesystems. If your machines only have 8-16GB of RAM I don't think it would be a good idea, as I've seen the rule of thumb is to dedicate 2GB for Proxmox's usage, which is in addition to any guest OS's requirements. Meanwhile I have a Debian install on a VPS that takes about 450MB of RAM.

For me, pros:

  • Native ZFS support - invaluable, ZFS is terrific. MergerFS+SnapRAID is a decent replacement but the dodgy tooling and laundry list of footguns makes me nervous to use it on important data. ZFS is idiot-proof, as long as you know what you're doing during the initial setup. RAIDZ expansion is coming this year and you can still use mixed-size disks in a RAIDZ as long as you accept that all disks are equivalent to the smallest one, so I personally feel ZFS is acceptable for grab-bag disk usage now
  • Separation of bare metal and server environment, which means you can spin up another server VM from scratch without impacting the previous one, then switch with zero downtime. In the end, I replaced Proxmox with Debian on ZFS root (ZFSBootMenu) and wrote a few hundred lines of bash to automate the installation, so when I switched it only took about 30 minutes of downtime start to finish.
  • Isolation of different environments. If my VM gets hacked, it will have a harder time reaching my Proxmox host etc. I run all services in isolated Docker environments anyway so this isn't that big of a perk for my threat profile.

Cons:

  • Partitioning RAM for ZFS ARC, Proxmox, and VM leads to inherent inefficiencies at the margins.
  • I usually give my VM n-1 CPU cores, which is still less power than if I had just used the CPU natively.
  • GPU passthroughs to VM can be less efficient, depending on the GPU and how it handles it. My iGPU is less performant when using its ~SR-IOV feature
  • Learning requirement - not a huge learning curve but it's a lot of knowledge that I will not use now that I've stopped using Proxmox
  • Hosting your data pool on the Proxmox host or a dedicated data VM means that your server VM needs to use NFS to access its data, which lacks a handful of features (e.g. inotify) and is a pain
  • Need to maintain two systems for updates, downtimes, etc
  • More points of failure
  • Extra startup time
  • Run by a company that thinks it's okay to use winrar-style nag popups every time you load the console, and requires you to manually dig through the source to disable that. I understand it's their business model, it doesn't change how it affects me the end user who lacks $120/year to spend on disabling a popup
hjpoijnerflkjn ,

I went exactly the same route. Years of proxmox realizing it is not KISS in any way for my use cases. Switches to Nixos on ZFS root (so no bash installation scripts ;) ).

However, docker has not the same level of isolation and security as VMs. I am currently looking into gVisor for that.

k_rol ,

$550? For a homelab you should only need to pay €110/year. What am I missing here?

TechnicallyColors ,

Yeah it's €110/year here: https://shop.proxmox.com/index.php?rp=/store/proxmox-ve-community

I remember evaluating the price a long time ago and thinking it was too much for disabling a pop-up, and on writing my post I navigated to their site and saw the standard subscription and thought that's what I had looked at a few years back: https://shop.proxmox.com/index.php?rp=/store/proxmox-ve-standard

k_rol ,

Ah yes I did the same thing as you as I checked again but from my PC.

I just paid myself the 110 after using it for 2 years. It's not for the popup since I was using a script to remove it. It's more to get the production ready updates. My server has too many important things now, I don't want less tested changes. That and to support the devs.

thatsnothowyoudoit ,
@thatsnothowyoudoit@lemmy.ca avatar

Pros:

  • you run a home lab

Cons:

  • you run a home lab
rottedmood OP ,
@rottedmood@lemmy.linuxuserspace.show avatar

ahhh thanks, i needed that laugh. too true....

corsicanguppy ,

I don't prefer proxmox, but I will say that when you have even a machine with 8 or 16gb RAM, virtualizing a workload on it just makes sense. At that point the cost is 12% resources, and the benefits IMHO farrr outweight that.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

Virtualization has a 1-2% performance penalty

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

You need Proxmox

Seriously though it is nice to have

some_guy ,

Proxmox is available free. You pay for support and maybe other things with a license, but you can download it and give it a spin at no cost. I just switched to Proxmox around 1m ago when I restarted my homelab project after years on hiatus. I used to use Esxi before Broadcom bought VMware and decided to suck. I like it so far.

It might be overkill for your needs. I'm running it because I want to play with setting up and managing Win Server (I only have experience managing existing servers on Win), so there's a distinct reason for me to be on Proxmox even though I'm a Mac and Linux person. I agree that it might be overkill for your i5 if you only plan to run one Ubuntu instance on it. However, a lot of homelabbing is about having an environment to try out and learn new skills. If that's something that's interesting to you, it might be worthwhile.

Keep in mind that you could also run KVM for virtualization if you find reason for VMs. You're not limited to Proxmox. And if you see no need for VMs, you already have three devices to do the things you bought them to do.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

For stability you want the enterprise subscription which is not free but is fairly reasonable

WeirdGoesPro ,
@WeirdGoesPro@lemmy.dbzer0.com avatar

Using ProxMox has been extremely useful for me. It has allowed me to experiment with a lot more things than I ever did before—it is very easy to spin up a new VM to test things out.

I would recommend it to anyone running a home server.

Decronym Bot , (edited )

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
HA Home Assistant automation software
~ High Availability
IP Internet Protocol
LTS Long Term Support software version
LXC Linux Containers
NAS Network-Attached Storage
NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency
VPS Virtual Private Server (opposed to shared hosting)
ZFS Solaris/Linux filesystem focusing on data integrity

[Thread for this sub, first seen 29th Jun 2024, 15:25]
[FAQ] [Full list] [Contact] [Source code]

ikidd ,
@ikidd@lemmy.world avatar

VMs under KVM are pretty much bare metal and Proxmox doesn't use much for resources itself, it's basically a headless Debian with a webserver interface to do all the KVM stuff.

Proxmox, especially if you use ZFS for the VM datastore, makes a home lab so much easier to revert, backup and deploy/clone VMs and LXCs. I highly recommend it if you're just starting out. Once you wrap your head around it, it gets out of the way and lets you just tinker with your projects, and not have to manually do everything in VirtManager or at the command line.

Combined with Proxmox Backup Server, it's a production ready hypervisor for anything you decide to keep. Also, the HA features work well enough that I had my main routing OPNsense VM jump between nodes when the primary node lost a drive, and I didn't notice for a week, it was that seamless.

SaintWacko ,

Seconding this. Especially if you're still learning and making mistakes, it's so nice to just be able to destroy a VM/CT and start over, rather then potentially breaking other things or the OS itself.

DarkDarkHouse ,
@DarkDarkHouse@lemmy.sdf.org avatar

Also needs mentioning: clustering. I have a years old cluster with none of the hardware I originally started with, but my Pi-hole is still there. Having the ability to migrate guests between hosts is a game changer when you frequently replace or rebuild said hosts. With the right setup, migration can have as little as a few seconds of downtime, or even no downtime at all. You can’t do that with bare metal installs.

monkeyman512 ,

If you want to get things working then never "tinker" with things, maybe it's not worth it. But if you want to learn and be able to try new things it is really helpful. Having a new VM not breaking existing VMs reduces risk when trying something new.

Lifebandit666 ,

It's the same reason I like running things in Docker; you can just wake up and read about something while enjoying your morning shit, then switch the computer on and try and boot it before that thing you're meant to be doing. If you can't do it you can just delete it and try again later.

I started Self Hosting with Proxmox 4 months ago and so far my only real snafu has been mapping drives directly to Proxmox with Fstab. If you're gonna do it, add "nofail" FFS.

I pass my drives through to my NAS VM to handle rather than Proxmox because it's easier to fix my NAS if it fucks up using Proxmox, than to try and fix a none-booting Proxmox.

Anyway now I'm at a point of stability and dim sat thinking about redoing it bare bones, but I love tinkering so I'm sure this is just the plateau before I discover something new to play with, so I'm keeping Proxmox

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • selfhosted@lemmy.world
  • test
  • worldmews
  • mews
  • All magazines
    •