mookie ,
@mookie@troublemaker.social avatar

I was wondering why my instance was overloaded. Then I saw the wonderful @lisamelton boosted a post. Thank you Lisa! 🥰

xobb ,
@xobb@mstdn.social avatar

@mookie @lisamelton why aren’t these requests cached? Asking for a friend

mookie OP ,
@mookie@troublemaker.social avatar

@xobb @lisamelton

Let your friend know that there's a fat cache in front of this instance that takes care of all the static assets (I worked with a crew of smart people who taught me about CDNs).

The CPU spikes are a result of any instance that didn't already have the post reaching out for the post body.

The CDN serves out the media since it's static.

But, the post itself is treated as a dynamic page, so each instance of people that follow the booster will reach out... all at once.

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@mookie

Any idea how many requests are in this big spike? And which process is driving the CPU load?

@xobb @lisamelton

mookie OP ,
@mookie@troublemaker.social avatar

@argv_minus_one @xobb @lisamelton

I'm not sure about the number of requests, but this is a 5min chart of performance during that period. I'm assuming it's some percentage over 100% of the 23k followers that Lisa has.

My instance runs on a puny 4-core (Intel Celeron N5105 @ 2.00GHz) mini-PC and I have this tweak in my mastodon-web.service.

Environment="WEB_CONCURRENCY=8"
Environment="MAX_THREADS=8"

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@mookie

From the bottom graph, it looks like you were getting around 100 requests per second on average, with a peak of around 140, and a total of roughly 12000 requests.

For a static website, that would be child's play even for your humble mini-server, but Mastodon must be doing a lot more work to serve each request than a static web server would.

I've never run Mastodon myself, so it's kind of interesting to see stuff like this.

@xobb @lisamelton

mookie OP ,
@mookie@troublemaker.social avatar

@argv_minus_one @xobb @lisamelton The numbers are usually higher. I thing Datadog takes an average for the period (5min) as the number looks lower when looked at in high intervals.

You’re right, that traffic would be peanuts for a static site. The large process chewing up CPU is Mastodon’s bundle.

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@mookie

Not the database? That's odd. I would think that serving those requests would just involve fetching from the database and reformatting everything into ActivityPub, with the database doing the heavy lifting.

I say this because, in a recent project of mine, that's how it went: the application itself did everything basically instantly, but the database had to do a lot of work for each request.

My project was non-HTTP and write-heavy, though, unlike Mastodon, so 🤷‍♂️.

@xobb @lisamelton

mookie OP ,
@mookie@troublemaker.social avatar

@argv_minus_one @xobb @lisamelton Maybe. lol. I’m sure Postgres was having a party on the machine too (along with Sidekiq and redis).

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@mookie

Three different processes besides Mastodon itself? That must be a pain to manage if you're not using containers.

@xobb @lisamelton

mookie OP ,
@mookie@troublemaker.social avatar

@argv_minus_one @xobb @lisamelton There a lot of moving parts for Mastodon. @gotosocial is still developing, but I am keeping an eye out on it. It’s just one binary and is really easy to maintain, especially for single user instances.

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@mookie

Yeah, definitely not a fan of long lists of out-of-process dependencies. I wish I could use Postgres as a library.

@xobb @lisamelton @gotosocial

mookie OP ,
@mookie@troublemaker.social avatar

@argv_minus_one @xobb @lisamelton @gotosocial My favorite thing with GoToSocial is that it can use SQLite as its datastore (with the option of Postegres too).

snott ,
@snott@teabag.ninja avatar

I'm curious how snac would handle this 🤔 (is what I'm running now)

CC: @argv_minus_one @xobb @lisamelton @gotosocial

argv_minus_one ,
@argv_minus_one@mstdn.party avatar

@snott @xobb @lisamelton @gotosocial

It is so neat that people can just write their own software and connect it to the Fediverse like it's no big deal. Open protocols are ✨magic✨ and I'm here for it!

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • test
  • worldmews
  • mews
  • All magazines