mttaggart ,

Sigh.

Another night of instability here on infosec.town. It's seeming to happen once everyone gets home and pops on to engage.

But what's more, I'm finding that blocks on here are essentially useless. Maybe that's a broader ActivityPub issue, but it's quite irksome to see posts from blocked accounts in boosts and searches.

singe ,
@singe@chaos.social avatar

@mttaggart https://chaos.social/@ordnung/112685013091080144 chaos.social saw it too. But if a technical explanation relating to Lemmy too. Might be useful for @jerry in debugging.

mttaggart OP ,

@singe @jerry That's super good to know; thanks!

jerry ,
@jerry@infosec.exchange avatar

@mttaggart @singe I just restarted the services and flushed redis for infosec.town. Let me know if you see a difference

mttaggart OP ,

@jerry @singe Yep, snappy as ever now, although it had been before. It does seem to be happening about every 12 hours?

jerry ,
@jerry@infosec.exchange avatar

@mttaggart @singe I set the job to flush redis weekly after the last bought of problems - I'll up that to twice a day and see if it helps.

mttaggart OP ,

@jerry @singe Incredible; thank you! Although the timing does suggest that the issue may be independent of the cache. I guess we'll find out tomorrow morning!

jerry ,
@jerry@infosec.exchange avatar

@mttaggart to be clear, the cause is definitely external - I see huge numbers of failed incoming messages, but I think iceshrimp is not handling them super elegantly and ends up with overly large queues sitting in redis that cause a slow down. I have to increase logging verbosity to try to find the source of the errors - it may will be the same lemmy.world issue

singe ,
@singe@chaos.social avatar

@jerry @mttaggart looks like chaos.social was due to an overheating CPU https://leah.is/notes/hot-servers-are-not-good/

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • test
  • worldmews
  • mews
  • All magazines