Mander

willya , in This instance is flyiiing (super fast)!
@willya@lemmyf.uk avatar

The latest release forced the more efficient backend updates.

Sal Mod , in This instance is flyiiing (super fast)!
@Sal@mander.xyz avatar

Thank you for the feedback!

Blaze , in This instance is flyiiing (super fast)!
@Blaze@reddthat.com avatar

Mander is great indeed!

Bl4ze , in Version update and server upgrade!
@Bl4ze@lemmy.world avatar

New test, 23:01

Blaze ,
@Blaze@reddthat.com avatar

Great!

Sal OP Mod ,
@Sal@mander.xyz avatar

It works now :)

Blaze ,
@Blaze@reddthat.com avatar

It does, that's great! Well done!

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks! All seems to be in order now 😌

lil , in Version update and server upgrade!
@lil@lemy.lol avatar

Now update to version 0.19.5, please

Blaze ,
@Blaze@reddthat.com avatar

I hope that's a joke, give this poor admin some rest!

Sal OP Mod ,
@Sal@mander.xyz avatar

As soon as it comes!

Blaze ,
@Blaze@reddthat.com avatar
Sal OP Mod ,
@Sal@mander.xyz avatar

Aaaah, thanks! 😆 It is done now.

Blaze ,
@Blaze@reddthat.com avatar

Already? Impressive!

Sal OP Mod ,
@Sal@mander.xyz avatar

This was a small update so it only required me to get to my computer. I just had to change 0.19.4 to 0.19.5 and restart the docker containers :)

The 0.19.3 -> 0.19.4 was lot more involved because of the database upgrade and all of the extra server stuff that I needed to catch up on.

fossilesque , in Version update and server upgrade!
@fossilesque@mander.xyz avatar

🎉 🌞

lvxferre , (edited ) in Version update and server upgrade!
@lvxferre@mander.xyz avatar

Sal, I got some error here: couldnt_get_posts. That appears every time that I try to visualise the instance through the "Subscribed" communities.

The error does not appear if I browse the instance through "Local" or "All", nor if I check the list of communities that I'm subscribed to. The error happens across devices and front-ends (tested with Linux Firefox, Android Firefox, Jerboa).

EDIT/UPDATE: I unsubscribed manually to a few communities each time, checking if the front page loaded fine. Eventually, it did.

Re-subscribing to either !technology or !fediverse seems to trigger the error again.

Shdwdrgn ,

That's the same error I was asking about below, and wondering if I had to do with the current status of collecting all the content. If so, it sounds like it could be another day or two before we can view our front pages again?

lvxferre ,
@lvxferre@mander.xyz avatar

For now, one option is to temp change your settings to browse by "Local" or "All". Or alternatively temporarily unsubscribe off a few comms; if you're subscribed to a small number of comms, some trial and error might help you to find which ones are causing the issue. (For me it was two LW comms).

lvxferre ,
@lvxferre@mander.xyz avatar

Sorry for the double reply.

Check if you're subscribed to any of those communities: !technology , !comicstrips or !memes. If you are, try to unsubscribe and see if your front page works again. For me it did the trick.

!fediverse was also giving me problems, but apparently it's fine now, so odds are that the problem will eventually go away on its own.

Shdwdrgn ,

All of this sounds like the issue is related to the new server still loading up the communities. I'm subbed to a pretty good list so it's probably best just to wait for the server to catch up. Thanks for the info though!

acockworkorange ,

Unsubscribing from comic strips did the trick a couple days ago, thanks! And just now I’ve subscribed back and everything is normal. So feel free to subscribe again.

Sal OP Mod ,
@Sal@mander.xyz avatar

(@Shdwdrgn ) I think it is related with the processing of so many activities per second while syncing with Lemmy.World.

I have tried changing some rate limits and the number of maximum database connections, but I get similar errors while moving around the site. The server is running well below 100% on all metrics, and the logs don't produce an obvious error. I will wait until we catch up with Lemmy.World and see if that fixes the issue. If it doesn't, then I will trouble shoot more in-depth.

lvxferre ,
@lvxferre@mander.xyz avatar

Thank you!

RobotToaster ,
@RobotToaster@mander.xyz avatar

There's some potential solutions to this mentioned here https://github.com/LemmyNet/lemmy/issues/4277

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks! The database is currently fetching lemmy world data as fast as possible, so this might be saturating the database. Once the syncronization is tested, I will try to tune postgres if it still happens.

lvxferre ,
@lvxferre@mander.xyz avatar

The issue seems to be gone for me. And the instance "feels" considerably faster than it did before :-D

Sal OP Mod ,
@Sal@mander.xyz avatar

Great, thanks for letting me know!! I did a bit of research and I think that the next time I need to upgrade the instance I will move it to a dedicated server instead of a virtual private server. But hopefully it runs smoothly for some time! :)

Sal OP Mod ,
@Sal@mander.xyz avatar

Hey @RobotToaster , @lvxferre , and @Shdwdrgn - we are now in sync with lemmy.world and the database under much less stress. This seems to have fixed the problems that I observed on my end. Are your problems also resolved?

Shdwdrgn ,

Ah! It looks like it JUST fixed it, I had checked about 15 minutes ago and still couldn't get my subscribed list to load, but it does appears to be loading up now. I'll reset my preferences and keep an eye on it through the day -- hoping for the best!

Sal OP Mod ,
@Sal@mander.xyz avatar

It should have been fixed about 13 minutes ago! Thanks for checking! If problems persist I will try to fine-tune the database settings as suggested above.

Shdwdrgn ,

There was a brief outage soon after I left that last reply, assume you were doing another round of maintenance at the time, but otherwise it's still looking good from here.

Sal OP Mod ,
@Sal@mander.xyz avatar

Ah, thanks. I think that was when I reboot the instance to change a setting that controls how post thumbnails are generated, as the new setting was failing to generate thumbnails for many posts.

RobotToaster ,
@RobotToaster@mander.xyz avatar

I just resubbed to the communities that were causing issues and it looks like everything is working well now, thanks.

acockworkorange ,

Letting you know I no longer see the issue. Thanks, Sal!

Sal OP Mod ,
@Sal@mander.xyz avatar

Great to hear! Thanks!

RobotToaster ,
@RobotToaster@mander.xyz avatar

Having the same issue here unfortunately.

ratel , in Version update and server upgrade!

Well done and many thanks!

Blaze , in Version update and server upgrade!
@Blaze@reddthat.com avatar

Well done!

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks!

Since the upgrade, I can see (here) that activities from lemmy.world are being fetched significantly faster. I will let it run for some time and hopefully this speed-up is enough to catch up.

Blaze ,
@Blaze@reddthat.com avatar

Great, looks promising!

You seem also to be catching up with the backlog somehow? https://grafana.lem.rocks/d/cdfzs0dwal3pca/federation-health-time-behind?orgId=1&var-instance=All&var-remote_instance=mander.xyz&var-remote_software=All&from=now-12h&to=now

Hopefully the trend will continue, let's wait a few hours and see

Bl4ze ,
@Bl4ze@lemmy.world avatar

Test from a LW account to see how long it takes to show up (I posted at 18:24 GMT+1)

Sal OP Mod ,
@Sal@mander.xyz avatar

Well it is here now! Almost caught up :-)

Blaze ,
@Blaze@reddthat.com avatar

FYI, I just tried posting from a LW account 4 minutes, the comments still don't show up: https://lemmy.world/comment/10702133

Sal OP Mod ,
@Sal@mander.xyz avatar

Ah, thanks!

Hmm, I am not sure about how the queue gets filled. It might be that the instance needs to process all of the older content first.

I will give it some hours and if it does not look like it will catch up, I will look into other methods.

Blaze ,
@Blaze@reddthat.com avatar

Sounds good!

Sal OP Mod ,
@Sal@mander.xyz avatar

The instance is catching up at a rate of ~15 activities/s, and we are 2.5M activities behind. So it should take about 2 days to catch up.

I have tried fiddling with some parameters to see if this could be improved, but I did not find the right knob to tune. Best is probably to just let it run 😄

Blaze ,
@Blaze@reddthat.com avatar

Nice!

Shdwdrgn ,

Does this have anything to do with the front page giving an error message? I get a lemmy server error, however I was able to log out and log back in again, and I can get to the communities page which got me back to this discussion, but even forcing a browser refresh of the front page continues to give an error message for me. FYI, ctrl-shift-R also fails, I'm not sure what else to try.

Blaze ,
@Blaze@reddthat.com avatar

The trend seems to go down at a nice pace, there's something quite satisfying to see it go down like that ha ha

M0oP0o ,
@M0oP0o@lemmy.world avatar

Yay, down to one week delay. Looking forward to catching up.

Shdwdrgn , in Version update and server upgrade!

Wow the new server is just flying along!

Sal OP Mod ,
@Sal@mander.xyz avatar

Good to hear!

BlastboomStrice , in Version update and server upgrade!
@BlastboomStrice@mander.xyz avatar

Lets gooooo🔥🔥🔥

Well done!

Sal Mod , in kbin.social federation issue
@Sal@mander.xyz avatar

Yes, I can check!

At the moment I see the message:

We are working on resolving the issues
Over the next few days, there will be a change in server infrastructure. Temporary problems with the website's functionality may occur during this time.

I will check again later

BlastboomStrice OP ,
@BlastboomStrice@mander.xyz avatar

Oh indeed, it's down again (yesterday I could read the posts)😬..

Sal Mod ,
@Sal@mander.xyz avatar

I also saw it working yesterday but didn't have a chance to respond and do tests 😅

BlastboomStrice OP ,
@BlastboomStrice@mander.xyz avatar

Yeah cool, no rush, besides it seems like kbin has some issues to solve first😅

Sal Mod ,
@Sal@mander.xyz avatar

Checking back in. Still not working :(

BlastboomStrice OP ,
@BlastboomStrice@mander.xyz avatar

Yeah indeed.. I too check the site from time to time. I had seen on their last posts that they were thinking of moving out of kbin.social due to the downtimes..

I will try to update you if/when they come back online.😅

acockworkorange , in [Done] Migrating images to object storage, images will be inaccessible for a bit

Thanks for all your work keeping the lights on, Sal!

M0oP0o , in kbin.social federation issue
@M0oP0o@lemmy.world avatar

Lets hope this migration fixes a bunch of federation issues. Right now I am hoping these issues are all related.

Can't wait until we can see all of lemmy again.

lvxferre , in [Done] Migrating images to object storage, images will be inaccessible for a bit
@lvxferre@mander.xyz avatar

Late to the party, but I hope that the transfer goes well!

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks!

Almost a day and only about 50% transferred. I was expecting it to be much faster... I just hope once it finishes it all works smoothly 😄

fossilesque ,
@fossilesque@mander.xyz avatar

🦎 🦎 🦎 🦎 🦎

lvxferre ,
@lvxferre@mander.xyz avatar

On a lighter note, it would take far longer if you delayed it! Just imagine some years from now, the amount of data being transferred!

Sal OP Mod ,
@Sal@mander.xyz avatar

Exactly! It is something that I wanted to do over a year ago, but I was not sure how... Pretty much all my knowledge about servers comes from setting up and trying not to break a lemmy instance, so this was another lesson that I was a bit scared to learn 😂 It turns out that it was very simple as the pict-rs (Lemmy's image backend) developer made a robust migration tool that automates this process. But since I don't know that beforehand, I need to triple check backups and read the instructions several times in preparation so even if it is easy and the hard work has been done by others it still requires some focus.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • random
  • mander@mander.xyz
  • test
  • worldmews
  • mews
  • All magazines