jerry ,
@jerry@infosec.exchange avatar

Just ordered the new Infosec.exchange DB server 😅

Increasing to 512GB of RAM on Hetzner’s dell 48 core Epyc with a 10g network 🫠

Going to try using zfs instread of ext4 this time since it’s what all the cool kids seem to be doing with Postgres these days.

gnyman ,
@gnyman@infosec.exchange avatar

@jerry my colleagues who went to pgconf 23 came back and said no zfs for new server , went with ext4 in the end

I think it was this talk that made them decide we don't want to do it, I can check https://youtu.be/JwAIOtpAjg4

Also are you using FDE/LUKS? I guess not as it's "public" data but if you do check the block size, LUKS picked the wrong one and got only 20% of non-encrypted performance before we noticed then we got 80%, also the "cloudflare patches" are important (now mount flags not patches)

gh0sti ,
@gh0sti@mastodon.social avatar

@jerry just run everything on RAM.

nxadm ,
@nxadm@infosec.exchange avatar

@jerry

ZFS? Hence the 512 GB RAM!

We old Solaris kids used to joke about ZFS memory usage ;)

pearofdoom ,
@pearofdoom@cloudisland.nz avatar

@jerry I've heard ensuring the postgres and zfs have the same block size can unlock extra perf

jerry OP ,
@jerry@infosec.exchange avatar

@pearofdoom I heard the same

kfanyo ,
@kfanyo@infosec.exchange avatar

@jerry It's true. Two words: multiple vdevs. You will be pleased. Congrats!

bplein ,
@bplein@bvp.me avatar

@jerry My gut tells me xfs is the fs of choice for performance with database workloads. (Amended later -> )

For SSD backed storage.

RandomDamage ,
@RandomDamage@infosec.exchange avatar

@bplein @jerry I wouldn't use anything but SSD storage for high IOPS DBs

It's just not worth the pain to try to get the performance out of spinning rust

jerry OP ,
@jerry@infosec.exchange avatar

@RandomDamage @bplein I’m all nvme ssd. I tried regular ssd and wow is that slow.

RandomDamage ,
@RandomDamage@infosec.exchange avatar

@jerry @bplein yep, and regular SSD is even more faster than mechanical HDDs.

Add in a reasonable set of tuning parameters to take advantage of all that RAM and this one will be so fast

rossmadness ,
@rossmadness@infosec.exchange avatar

@jerry I'm sure @jimsalter will be happy to hear this.

caffinepwrd ,
@caffinepwrd@infosec.exchange avatar

@jerry just setup a recurring donation, love this community and your work

jerry OP ,
@jerry@infosec.exchange avatar

@caffinepwrd :blobheartcat: thank you for your support and for being part of the community

gary_alderson ,
@gary_alderson@infosec.exchange avatar

@jerry @caffinepwrd

a couple pretty expansive links - maybe you can garner something interesting out of them - pls do a followup on new server thread - interesting stuff

https://vadosware.io/post/everything-ive-seen-on-optimizing-postgres-on-zfs-on-linux/

https://wiki.postgresql.org/images/8/86/PostgreSQL_on_ZFS.pdf

gary_alderson ,
@gary_alderson@infosec.exchange avatar

@jerry try the caching with zfs - i think it works pretty well?

rk ,
@rk@well.com avatar

@jerry

Since it’s the second server you’ve tried, you could argue it’s a…sequel server.

Josh ,
@Josh@infosec.exchange avatar

[Thread, post or comment was deleted by the author]

  • Loading...
  • jerry OP ,
    @jerry@infosec.exchange avatar

    @Josh the database server also serves:
    Infosec.space
    Pixel.infosec.exchange
    Infosec.place
    Fedia.social
    Infosec.town
    Fedia.io
    Meetup.infosec.exchange
    Books.infosec.exchange
    Infosec.pub

    It’s a bit memory bound now, so I’ll upgrade to this one, and return one of the existing servers.

    xabean ,
    @xabean@infosec.exchange avatar

    @jerry @Josh if the current DB server is memory bound without ZFS, you're going to need a lot more RAM for ZFS. As I understand it, ZFS needs a lot of RAM.

    jerry OP ,
    @jerry@infosec.exchange avatar
    jerry OP ,
    @jerry@infosec.exchange avatar

    @xabean @Josh ok. I guess I’m staying with ext4 :)

    chrisvest ,
    @chrisvest@mastodon.social avatar

    @jerry @xabean @Josh When I worked at Neo4j we generally recommended XFS as a slightly better ext4. I get it if you don’t want to touch the boat on this one, though.

    ithoughtisawa2 ,
    @ithoughtisawa2@infosec.exchange avatar

    @xabean @jerry @Josh My limited understanding of zfs is that it uses any extra ram for caching. It doesn't necessarily require more ram but it will use whatever you give it. Obviously more ram means more caching which means better performance.

    jerry OP ,
    @jerry@infosec.exchange avatar

    @ithoughtisawa2 @xabean @Josh I guess the question is whether Postgres or zfs benefits more from the extra ram

    xabean ,
    @xabean@infosec.exchange avatar

    @jerry @ithoughtisawa2 @Josh The database. Always the database. The less it has to touch or spill to disk, the better.

    feld ,
    @feld@bikeshed.party avatar

    @jerry @ithoughtisawa2 @xabean @Josh don't do it. It can be done, but you're asking for headaches. In the future Postgres will actually have the ability to run optimally on ZFS when a direct I/O implementation is finished, but it's not ready yet

    https://freebsdfoundation.org/wp-content/uploads/2023/02/munro_ZFS.pdf

    feld ,
    @feld@bikeshed.party avatar

    @jerry Put it on XFS and I'd enable Postgres checksums if you haven't done that, and let it fly

    jerry OP ,
    @jerry@infosec.exchange avatar

    @feld that is the feedback I’ve been getting tonight and what I will likely do

    FritzAdalis ,
    @FritzAdalis@infosec.exchange avatar

    @jerry @feld
    I was going to suggest btrfs as an alternative to zfs because the zfs cache isnt unified with the linux fs cache, but apparently performance is poor with postgres. They say copy on write kills it, which would also affect zfs.

    Josh ,
    @Josh@infosec.exchange avatar

    [Thread, post or comment was deleted by the author]

  • Loading...
  • jerry OP ,
    @jerry@infosec.exchange avatar

    @Josh .exchange is a busy little instance.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • test
  • worldmews
  • mews
  • All magazines