Pros and cons of Proxmox in a home lab?

Hi all. I was curious about some of the pros and cons of using Proxmox in a home lab set up. It seems like in most home lab setups it’s overkill. But I feel like there may be something I’m missing. Let’s say I run my home lab on two or three different SBCs. Main server is an x86 i5 machine with 16gigs memory and the others are arm devices with 8 gigs memory. Ample space on all. Wouldn’t Proxmox be overkill here and eat up more system resources than just running base Ubuntu, Debian or other server distro on them all and either running the services needed from binary or docker? Seems like the extra memory needed to run the Proxmox software and then the containers would just kill available memory or CPU availability. Am I wrong in thinking that Proxmox is better suited for when you have a machine with 32gigs or more of memory and some sort of base line powerful cpu?

phanto ,

I have tried a couple of Proxmox clusters, one with overkill specs and one with little Mini PCs. Proxmox does eat up a fair amount of memory, but I have used it with Ceph for live migrations. Its really useful to me to be able to power off a machine, work on it, then bring it back up, and have no interruptions in my services. That said, my Mini PCs always seemed to be hurting for RAM. So that's my pros and cons.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

Proxmox doesn't have a lot of overhead. However, Ceph is a beast and requires very power hardware with at least a dedicated 10g network between hosts for transfers. You also need 5 or more nodes for it to be reliable. I wouldn't recommend Ceph as there isn't a lot of point to it. You can get a similar functionality with NFS or ZFS replication.

phanto ,

I have it working with LaCP'd 4gb networking for the transfers. Five nodes. I agree though, It's a beast on RAM.

peregus ,

I run it on a 4GB Fujitsu Futro S920! 😆
All the RAM seems to be used by 3 VMs. Some SWAP is been used, ok, but the Proxmox overhead doesn't seem that much.

TCB13 ,
@TCB13@lemmy.world avatar

C'mon just move to Incus: https://lemmy.world/comment/10896868 :P

peregus ,

No way! For just 1 reason: I will have to learn another new thing and replace it in about 6 servers. I value my time and for now Proxmox is fine.

P.s. Incus seems nice though! NO, stop tempting me!!! I'm already in the rabbit hole with a gazilion of self hosted services and dozens piling up in the to do list 🙈🙈🙈

TCB13 ,
@TCB13@lemmy.world avatar

Well, I understand your POV... but real software freedom instead of messages asking you to buy a license and a questionable kernel is always a good choice :P

kitnaht ,

Does Incus support things like Kernel Samepage Merging? How does it handle Windows VMs? Does the WebUI give a nice and easy novnc window that just works?

TCB13 , (edited )
@TCB13@lemmy.world avatar

Yes ksmtuned is your friend. For VMs it can be managed / enabled like any other Linux Kernel + QEMU/KVM running with KSM enabled.

On LXC containers it may be a bit harder as it depends a LOT, best results if you're using systemd both the host and containers. It may work out all out of the box or you'll have to resort to ksm_wrapper in both the Incus executable and the stuff running inside your containers.

Don't forget that:

KSM only operates on those areas of address space which an application
has advised to be likely candidates for merging, by using the madvise(2)
system call: int madvise(addr, length, MADV_MERGEABLE).
https://www.kernel.org/doc/Documentation/vm/ksm.txt

How does it handle Windows VMs

As one would except from QEMU... https://blog.simos.info/how-to-run-a-windows-virtual-machine-on-incus-on-linux/

Does the WebUI give a nice and easy novnc window

Yes it works fine. https://youtu.be/wqEH_d8LC1k?feature=shared&t=508

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

Just wait until you get a few machines. You can live transfer things and dynamically allocate resources.

TCB13 , (edited )
@TCB13@lemmy.world avatar

If you know your way around Linux you most likely don’t need Proxmox and its pseudo-open-source... you can try Incus / LXD instead.

Avoid Proxmox and safe yourself a LOT of headaches down the line. Go with Debian 12 + Incus/LXC, it runs VMs and containers very well. Proxmox ships with an old kernel that is so mangled and twisted that they shouldn’t even be calling it a Linux kernel. Also their management daemons and other internal shenanigans will delay your boot and crash your systems under certain circumstances.

LXD/Incus provides a management and automation layer that really makes things work smoothly - essentially what Proxmox does but properly done. With Incus you can create clusters, download, manage and create OS images, run backups and restores, bootstrap things with cloud-init, move containers and VMs between servers (even live sometimes).

Another big advantage is the fact that it provides a unified experience to deal with both containers and VMs, no need to learn two different tools / APIs as the same commands and options will be used to manage both. Even profiles defining storage, network resources and other policies can be shared and applied across both containers and VMs.

I draw your attention to containers (not docker), LXC containers because for most people full virtualization isn't even required. In a small homelab if you can have containers that behave like full operating systems (minus the kernel) including persistence, VMs might not be required. Either way LXD/Incus will allow for both and you can easily mix and match and use what you require for each use case. Hell, you can even run Docker inside an LXC container.

For eg. I virtualize the official HomeAssistant image with Incus because we all know how hard is to get that thing running, however my NAS / Samba shares are just a LXD Debian 12 container with Samba4, Nginx and FileBrowser. Same goes for torrent client that has its own container. Some other service I've exposed to the internet also runs a full VM for isolation.

Like Proxmox, LXD/Incus isn’t about replacing existing virtualization techniques such as QEMU, KVM and libvirt, it is about augmenting them so they become easier to manage at scale and overall more efficient. I can guarantee you that most people running Proxmox today it today will eventually move to Incus and never look back. It woks way better, true open-source, no bugs, no delayed security updates, no BS licenses and way less overhead.

Also, let's consider something, why use Proxmox when half of it’s technology (the container part) was made by the same people who made LXD/Incus? I mean Incus is free, well funded and can be installed on a clean Debian system with way less overhead and also delivers both containers and VMs.

Yes, there's an optional WebUI for it as well!

https://lemmy.world/pictrs/image/9caa6ea8-17b1-48f6-a8c2-ff3f606f3482.png
https://lemmy.world/pictrs/image/a5a110b2-ed6f-431f-a767-0a21fb337a6b.png

Some documentation for you:

barsquid ,

I think I was on a previous account the last time I saw you, glad to see you're still posting. You convinced me to move from Proxmox to Incus a while back. Sure, I had some growing pains, but it's pretty smooth now.

I like that I can switch out my distros underneath Incus instead of being stuck on one weird kernel. IME you were absolutely right about that. I'm getting into atomic distros to manage homelab machines. I would not be able to do that on Proxmox.

I also don't need to edit a giant Javascript file to remove a nag about enterprise software repos, which is nice.

TCB13 ,
@TCB13@lemmy.world avatar

I'm glad to know that I could help.

I like that I can switch out my distros underneath Incus instead of being stuck on one weird kernel

This is an interesting take that I never considered before, my experience (be it corporate or at home) is usually around Debian machines running Incus and I never had the need to replace the distro underneath it.

barsquid ,

Yeah, I think it's an unusual case, but I wanted to bring it up to support your point about rejecting their kernel and distro. You can put Incus on a lot of different systems. Don't like systemd? Put it on Void. Want a declarative setup? NixOS. Minimalist? Alpine.

Do I want to maintain a full operating system just to run this one type of software? No, that's absurd. I want to choose the distro I want to work with and then have the software work on top of it.

TCB13 ,
@TCB13@lemmy.world avatar

You can put Incus on a lot of different systems. Don’t like systemd? Put it on Void. Want a declarative setup? NixOS. Minimalist? Alpine.

This is great, yeah.

rottedmood OP ,
@rottedmood@lemmy.linuxuserspace.show avatar

Thanks for all this. I’m familiar with Linux and I just think for my need, something like Proxmox is overkill. I do need to learn LXD on its own. Typically I just run binaries of the services I use, and I don’t tend to use docker or other things. I had toyed with the thought of using Proxmox for management purposes because let’s face it management of several on prem and off prem servers can be a pain. But keeping things running fast and smooth (for spouse approval) is important. I’ll look over the links you provided as it’s probably just good for me to learn LXD directly.

TCB13 ,
@TCB13@lemmy.world avatar

Typically I just run binaries of the services I use, and I don’t tend to use docker or other things

That's essentially what I do in my NAS with LXD, it's a great use case for it.

Enjoy.

catloaf ,

In what scenarios have you found Proxmox to be unstable? I've had almost no issues with it, despite using it in several unsupported ways.

TCB13 ,
@TCB13@lemmy.world avatar

Check the bottom of reply, there’s a link there with my experience over the years.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

LXC is worse than virtualization as it pins to a single core instead of getting scheduled by the kernel scheduler. It also is quiet slow and dated. Either run Podman, Docker or full VMs. Proxmox has a really nice GUI that allows for more advanced management and live transfers between hosts. It also ships with a newer kernel than Debian although it shouldn't matter as you are using it for virtualization.

TCB13 ,
@TCB13@lemmy.world avatar

LXC is worse than virtualization as it pins to a single core instead of getting scheduled by the kernel scheduler. It also is quiet slow and dated. Either run Podman, Docker or full VMs.

First what you're saying about the scheduler isn't even what happens by default, that was some crap that Proxmox pulled when they migrated from OpenVZ to LXC. To be fair, they had a bunch of more or less valid reasons to force that configuration, but again it due to kernel related issues that were affecting Proxmox more than regular Ubuntu and those issues were solved around the end of 2021.

Now Docker and LXC serve different purposes and they aren't a replacement for each other. Docker is a stateless application container solution while LXC is a full persistent container aimed at running full operating systems...

Docker and LXC share a bunch of underlaying technologies at on the beginning Docker even used LXC as their backed, they later moved to their execution environment called libcontainer because they weren't using all the featured that LXC provided and wanted more control over the implementation.

For those who really need full systems is LXC definitely faster than a VM. Your argument assumes everything can and should be done inside Docker/Podman when that's very far from the reality. The Docker guys have written a very good article showcasing the differences and optimal use cases for both.

Here two quotes for you:

LXC is especially beneficial for users who need granular control over their environments and applications that require near-native performance. As an open source project, LXC continues to evolve, shaped by a community of developers committed to enhancing its capabilities and integration with the Linux kernel. LXC remains a powerful tool for developers looking for efficient, scalable, and secure containerization solutions. Efficient access to hardware resources (...) Virtual Desktop Infrastructure (VDI) (...) Close to native performance, suitable for intensive computational tasks.

Docker excels in environments where deployment speed and configuration simplicity are paramount, making it an ideal choice for modern software development. Streamlined deployment (...) Microservices architecture (...) CI/CD pipelines.

Anyways...

It also ships with a newer kernel than Debian although it shouldn’t matter as you are using it for virtualization.

It matters, trust me. Once you start requiring modules it will suddenly matter. Either way even if they ship a kernel that is newer than Debian it is so fucked at that point that you'll be better with whatever Debian provides out of the box.

mouse ,
@mouse@midwest.social avatar

As a small homelabber I agree with this.
I started with a baremetal and using Docker, and switched to Proxmox, and now over to Incus, actually currently I am using Debian with cockpit + cockpit-machines. I do like Incus, I keep hopping back and forth between cockpit, I need to settle on one.

TCB13 ,
@TCB13@lemmy.world avatar

Actually it would be interesting to see cockpit-machines move to Incus as a virtualization backend and support both LXC containers and QEMU VMs tat way.

habitualTartare ,

I'm using a commercial desktop with an i5 Sandy bridge. I maxed out to 32Gb of ram only because I'm running trueNAS, debian with containers, and home assistant. Most RAM goes to trueNAS and trueNAS doesn't accurately report ram. For CPU, mostly just task limited but I don't really think thats a proxmox issue.
Obviously it's not going to support an enterprise or even small business but it works for what I need of less than 4 users on my budget.

Proxmox doesn't really ask for much but I probably would recommend docker for your arm devices.

phucyall ,

You need to understand what Proxmox gives you, which primarily is ability to run/manage/backup/etc VMs easily. If you don’t care about that, don’t use it. I have a fairly well spec’d desktop I use for homelab and I use proxmox because I often do experiments in VMs where snapshots and ability to jump to snapshots is essential. So is being able to spin up a new VM with new OS (like Windows) for example to do some testing. You can still do VMs without proxmox, but proxmox does make it a lot easier for living with daily.

TCB13 ,
@TCB13@lemmy.world avatar

You need to understand what Proxmox gives you, which primarily is ability to run/manage/backup/etc VMs easily

Yeah and after understanding what it gives you then you move to Incus because while it might be a bit harder to setup it delivers around 80% of what Proxmox does without the overhead, mangled kernel and licensing issues.

https://cockpit-project.org/ also does VMs and can work for people without cluster needs.

snekerpimp ,

Proxmox is based on kvm/qemu, and is very resource conservative. There is virtually no impact on performance due to the hypervisor, even on older processors. Scheduling on the cpu and hypervisor makes running multiple VMs at the same time trivial as well. RAM and I/O bandwidth are the two things that can affect performance. Running out of RAM due to too many VMs will grind you to a halt, but so would running too many applications or containers on bare metal. Running everything off of one spinning sata disk will make it impossible, but again, same downfall on bare metal.

Those minimal impacts to performance are a minor nuisance compared to the ability to run experiments and learn on sandboxed VMs. Now that TrueNAS has better virtualization support, it has caught my eye as a better homelab solution, but I will always have a proxmox server running somewhere in my stack just due to the versatility it gives me.

machinin ,

For me, pros are:

  • Fun to learn something new
  • Easy to test different systems. For example, I can play with different router or NAS software without having a separate computer around.
  • I've been able to create different "computers" that serve different needs and require different levels of security.
  • Currently, a cluster is probably overkill, it was a fun experiment.

Cons

  • Updating all the different systems can be a pain. I could probably automate it, but I haven't made the time to learn it yet.
  • As a beginner, I'm throwing a bunch of parts together and hoping it will work. I should probably be more strategic in my implementation, but I don't know what to prioritize. I'm sure I'll have to start over in the future.
  • With the previous point, the storage setup doesn't seem very intuitive. I probably need to set up that better.
  • I haven't quite figured out backups yet. My VM backups all seem too big. I need to figure that out and automate it.

Hope this is helpful.

umbrella ,
@umbrella@lemmy.ml avatar

a simple cron job pointing to an update.sh with an apt update && apt upgrade -y does the trick.

i wouldnt recommend you to completely automate it though

debian has unattended-updates by default and generally takes care of itself

rutrum ,
@rutrum@lm.paradisus.day avatar

It seemed nice at first, but one major issue: GPU passthrough was a nightmare. It cant be done in the UI and I didnt understand fully how it worked. There are many different tutorials not by promox that are outdated or may not work. It was frustrating enough I jumped to NixOS. Other hiccups included having to go to the terminal to passthrough drives for openmediavault, but that one was kind of straightforward atleast, and it worked first time.

In hindsight, I didnt actually need to virtualize everything at that level, so I never really had a good use case for it anyway. I use containers over entire VMs.

monkeyman512 ,

I think GPU passthrough has improved since you have used it. Some command line prep work is still necessary, but the passthrough config is done in the GUI.

Swarfega ,

I did it a week ago and it was just a case of passing through the video card. I came across a lot of guides and they were all in the CLI. I assume things have improved or maybe it differs per card. I was just using onboard graphics from an N100 CPU.

philpo ,

A lot of guides are still for Proxmox 7 or even 6 on that matter.

Proxmox 8 has changed a lot in that regard.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

The onboard iGPU doesn't need anything special once you turn on IOMMU. You just click add ePCI device.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

It really depends on the hardware. Also PVE 8 adds some improvements. I would just follow the wiki.

monkeyman512 ,

If you want to get things working then never "tinker" with things, maybe it's not worth it. But if you want to learn and be able to try new things it is really helpful. Having a new VM not breaking existing VMs reduces risk when trying something new.

Lifebandit666 ,

It's the same reason I like running things in Docker; you can just wake up and read about something while enjoying your morning shit, then switch the computer on and try and boot it before that thing you're meant to be doing. If you can't do it you can just delete it and try again later.

I started Self Hosting with Proxmox 4 months ago and so far my only real snafu has been mapping drives directly to Proxmox with Fstab. If you're gonna do it, add "nofail" FFS.

I pass my drives through to my NAS VM to handle rather than Proxmox because it's easier to fix my NAS if it fucks up using Proxmox, than to try and fix a none-booting Proxmox.

Anyway now I'm at a point of stability and dim sat thinking about redoing it bare bones, but I love tinkering so I'm sure this is just the plateau before I discover something new to play with, so I'm keeping Proxmox

ikidd ,
@ikidd@lemmy.world avatar

VMs under KVM are pretty much bare metal and Proxmox doesn't use much for resources itself, it's basically a headless Debian with a webserver interface to do all the KVM stuff.

Proxmox, especially if you use ZFS for the VM datastore, makes a home lab so much easier to revert, backup and deploy/clone VMs and LXCs. I highly recommend it if you're just starting out. Once you wrap your head around it, it gets out of the way and lets you just tinker with your projects, and not have to manually do everything in VirtManager or at the command line.

Combined with Proxmox Backup Server, it's a production ready hypervisor for anything you decide to keep. Also, the HA features work well enough that I had my main routing OPNsense VM jump between nodes when the primary node lost a drive, and I didn't notice for a week, it was that seamless.

SaintWacko ,

Seconding this. Especially if you're still learning and making mistakes, it's so nice to just be able to destroy a VM/CT and start over, rather then potentially breaking other things or the OS itself.

DarkDarkHouse ,
@DarkDarkHouse@lemmy.sdf.org avatar

Also needs mentioning: clustering. I have a years old cluster with none of the hardware I originally started with, but my Pi-hole is still there. Having the ability to migrate guests between hosts is a game changer when you frequently replace or rebuild said hosts. With the right setup, migration can have as little as a few seconds of downtime, or even no downtime at all. You can’t do that with bare metal installs.

Decronym Bot , (edited )

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
HA Home Assistant automation software
~ High Availability
IP Internet Protocol
LTS Long Term Support software version
LXC Linux Containers
NAS Network-Attached Storage
NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency
VPS Virtual Private Server (opposed to shared hosting)
ZFS Solaris/Linux filesystem focusing on data integrity

[Thread for this sub, first seen 29th Jun 2024, 15:25]
[FAQ] [Full list] [Contact] [Source code]

WeirdGoesPro ,
@WeirdGoesPro@lemmy.dbzer0.com avatar

Using ProxMox has been extremely useful for me. It has allowed me to experiment with a lot more things than I ever did before—it is very easy to spin up a new VM to test things out.

I would recommend it to anyone running a home server.

some_guy ,

Proxmox is available free. You pay for support and maybe other things with a license, but you can download it and give it a spin at no cost. I just switched to Proxmox around 1m ago when I restarted my homelab project after years on hiatus. I used to use Esxi before Broadcom bought VMware and decided to suck. I like it so far.

It might be overkill for your needs. I'm running it because I want to play with setting up and managing Win Server (I only have experience managing existing servers on Win), so there's a distinct reason for me to be on Proxmox even though I'm a Mac and Linux person. I agree that it might be overkill for your i5 if you only plan to run one Ubuntu instance on it. However, a lot of homelabbing is about having an environment to try out and learn new skills. If that's something that's interesting to you, it might be worthwhile.

Keep in mind that you could also run KVM for virtualization if you find reason for VMs. You're not limited to Proxmox. And if you see no need for VMs, you already have three devices to do the things you bought them to do.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

For stability you want the enterprise subscription which is not free but is fairly reasonable

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

You need Proxmox

Seriously though it is nice to have

corsicanguppy ,

I don't prefer proxmox, but I will say that when you have even a machine with 8 or 16gb RAM, virtualizing a workload on it just makes sense. At that point the cost is 12% resources, and the benefits IMHO farrr outweight that.

possiblylinux127 ,
@possiblylinux127@lemmy.zip avatar

Virtualization has a 1-2% performance penalty

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • selfhosted@lemmy.world
  • test
  • worldmews
  • mews
  • All magazines
    •