Thank you! I used to have more accounts on other instances for these cases but recently I have been loving this instance so much that I'd rather just wait instead of using my alts. So thanks for all your hard work!
Yes, I have been told about the slow federation. I have tested a few things - for example, I am able to fetch up-to-date data from Lemmy.World immediately. But I am not entirely sure why the federation is so slow in the background. Has anyone noticed if this affects only lemmy.world?
I have two things planned:
Move images to object storage, which will be a more efficient way of using space. I can also use what I gain from decreasing disk resources by increasing the RAM and CPU count.
Upgrade the instance to 0.19.4
My plan is to do these first and hopefully the federation issues are fixed during the process, and if not the next step will be to troubleshoot. But I have been so busy recently, and I do want to have a long block of uninterrupted time in case that things go wrong and I have to spend some time troubleshooting. I will try to do these tomorrow.
EDIT: I have looked through the search for others experiencing similar issues, and it seems to be something experienced in the past by other instances, and so I will try to get some inspiration from those previous experiences: https://mander.xyz/post/10489815
Are you on the admins Matrix channel? The APAC instances (Reddthat, lemdro.id, Aussie.zone, Lemmy.nz, Blahaj) had the same issues a few weeks back, they managed to solve them somehow, they'll probably be able to help you out
I have collected a few of the posts... I have lagged behind on a few things (object storage, migrating pictrs database to postgres, upgrade to 0.19.4), so I am trying to catch up and during this process I will see if I can implement their solutions.
I have also noticed a generally slower experience when using the instance.. After I move the files to the much cheaper object storage I will be able to allocate more resources to RAM, CPU, and bandwidth. I was under the impression that the slow federation could be related to this decrease in performance, but from what I have read now it seems that it happens independent from this.
EDIT: I have looked through the search for others experiencing similar issues, and it seems to be something experienced in the past by other instances, and so I will try to get some inspiration from those previous experiences: https://mander.xyz/post/10489815
Hi Sal, thanks for the update and everything you do here! Glad to hear that everything is okay and I completely understand being busy.😊 Fingers crossed on the new version fixing most of the issued.
There's another overlooked issues regarding the slow federation. And I just found out a few days ago, there's de-peering drama since mid February that caused a huge of delays from APAC to Europe.
Around that time until now, I had trouble accessing some sites, like I couldn't access some github projects, f-droid repos also failed, and also some websites. I've tried cloudflare warp, it didn't help either.
And if I'm not mistaken our admin has a solution to this issue a while ago, you may ask Dave.
Thanks!! I faced an unexpected problem, but now the files are being transferred ^_^
The transfer is quite slow. I have timed it for a bit and if the "Used Space" data on the object storage dashboard is correct, files are being transferred at about 120 MB / minute... If that rate is the average, this will take about 40 hours :/ But hopefully the object storage dashboard is lagging. We will see.
Wow that's depressingly slow! My previous file server would transfer at around 75MB/s (not minute) and I thought that was pretty bad. Hopefully you can figure out where the bottleneck is at.
I thought that worse case scenario this would be an overnight transfer, but only 40 GB have been transferred so far.
The server's bandwidth use is well below the limit, so I think that the issue might be on the receiving end (Contabo's S3-compatible object storage).
I had to decrease the number of parallel connections because otherwise I would get too many errors and the transfer would stop, so it might not be the transfer speed of a single file, but rather the amount of time the server has to wait between initiating transfers. I am not sure, but at ~50GB per 12 hours this will end up taking almost 4 days... I will ask in the Matrix chat to see if this was also the experience of others who migrated.
Exactly! It is something that I wanted to do over a year ago, but I was not sure how... Pretty much all my knowledge about servers comes from setting up and trying not to break a lemmy instance, so this was another lesson that I was a bit scared to learn 😂 It turns out that it was very simple as the pict-rs (Lemmy's image backend) developer made a robust migration tool that automates this process. But since I don't know that beforehand, I need to triple check backups and read the instructions several times in preparation so even if it is easy and the hard work has been done by others it still requires some focus.
Since the upgrade, I can see (here) that activities from lemmy.world are being fetched significantly faster. I will let it run for some time and hopefully this speed-up is enough to catch up.
The instance is catching up at a rate of ~15 activities/s, and we are 2.5M activities behind. So it should take about 2 days to catch up.
I have tried fiddling with some parameters to see if this could be improved, but I did not find the right knob to tune. Best is probably to just let it run 😄
Does this have anything to do with the front page giving an error message? I get a lemmy server error, however I was able to log out and log back in again, and I can get to the communities page which got me back to this discussion, but even forcing a browser refresh of the front page continues to give an error message for me. FYI, ctrl-shift-R also fails, I'm not sure what else to try.
I don't know if it's my app (Thunder on android), but all operations seem slow (I think it used to be faster, changing to guest from another instance ~everything is faster). Even saving this post resulted in a timeout.
I'll report my main issues:
My profile page and inbox take about 5seconds to load and sometimes they load indefinetly
Opening any post takes about 3seconds to load
Saving any image takes around 5seconds to load
Is this the experience for the rest as well? I'd like to know if it's just me or that mander is slow. I'm not angry or something, just reporting stuff to let the owners and users know. If it could be fixed, it would be nice. If it's an infrastructure issue, well I get it.
Not yet. :) We just chillin in our simple little instance. You'll probably want to make sure to trigger the indexing manually on the big servers once the community is created. Our federation syncing a bit squirrley right now. I messaged Sal yesterday but he's probably busy. It'll get sorted I'm sure. 😅 He's good about that.
Mander
Top