[Done] Migrating images to object storage, images will be inaccessible for a bit

I am performing a migration that moves the images files from disk storage to object storage. During this migration the image back-end needs to be turned off, and so images will not be accessible.

It should still be possible to comment.

I am not sure how long this will take.

This is part of a few optimizations that I plan to do before I upgrade the instance to version 0.19.4.

EDIT: Update, June 17 06:08 UTC

The transfer is working, but, after 12 hours, only ~50 GB have been transferred to Contabo's object storage (rate of about 1.16 MB/s). This is a lot slower than I imagined, and, if this rate continues, it will take about 3 more days to transfer the remaining 300 GB.

I am not sure if this slow rate is to be expected for moving files to object storage, or if something is wrong. If any of you has experience with this I would appreciate some tips! I have also asked in the Matrix chat in case someone there can provide me some feedback as well.

EDIT: Update, June 18, 07:44 UTC

Migration is done!

272.16 GB of image data has been transferred to object storage. This took about 37 hours. This means that transfer occurred at a rate of 2 MB/s. From other admin's experience, this transfer was quite slow. I am making use of Contabo as my object storage provider, and I suspect that the speed of the transfer was limited on their end. Another admin reported that they used backblaze b2 as the object storage provider and they experienced much faster transfers. All of this should have no influence on how the site performs from the end-user's point of view, I am documenting this for other admins who learn from other's experiences by searching through lemmy.

Object storage is a much cheaper way to save files in the cloud. Before this, I have stored the image files into the same SSD disk where the instance lives (rented from serverspace.io). The cost of this disk space goes up as the disk space needs increase. At the current disk size of 400 GB renting that space was costing $36/month and increased $0.9/mo for every 10 GB.

Object storage is a specialized way of saving and serving files from the cloud. Object storage is much cheaper, at €6 for 500 GB (from Contabo), and so it is a much more scalable way of storing the instance's files. The only reason I did not do this earlier was because I was not sure about how to do it and kept pushing it back. But now, it's done! I would recommend other admins to do this as early as possible, as migrating many files is more of a hassle than migrating a few.

Istill need to do a few more tasks (moving the instance to a new server with a smaller disk, upgrading the postgres database to be compatible with 0.19.4, changing the pict-rs database format from sled to postgres, and then upgrading to 0.19.4). But first I will let the instance run for a few days with object storage to make sure nothing is breaking.

Thank you for your patience :D

acockworkorange ,

Thanks for all your work keeping the lights on, Sal!

lvxferre ,

Late to the party, but I hope that the transfer goes well!

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks!

Almost a day and only about 50% transferred. I was expecting it to be much faster... I just hope once it finishes it all works smoothly 😄

fossilesque ,
@fossilesque@mander.xyz avatar

🦎 🦎 🦎 🦎 🦎

lvxferre ,

On a lighter note, it would take far longer if you delayed it! Just imagine some years from now, the amount of data being transferred!

Sal OP Mod ,
@Sal@mander.xyz avatar

Exactly! It is something that I wanted to do over a year ago, but I was not sure how... Pretty much all my knowledge about servers comes from setting up and trying not to break a lemmy instance, so this was another lesson that I was a bit scared to learn 😂 It turns out that it was very simple as the pict-rs (Lemmy's image backend) developer made a robust migration tool that automates this process. But since I don't know that beforehand, I need to triple check backups and read the instructions several times in preparation so even if it is easy and the hard work has been done by others it still requires some focus.

01101000_01101001 ,

I never knew there was a salamander that accompanied the mander nomenclature.

Sal OP Mod ,
@Sal@mander.xyz avatar

Yeah! Hello! 🦎

Blaze ,
@Blaze@reddthat.com avatar

Good luck!

Sal OP Mod ,
@Sal@mander.xyz avatar

Thanks!! I faced an unexpected problem, but now the files are being transferred ^_^

The transfer is quite slow. I have timed it for a bit and if the "Used Space" data on the object storage dashboard is correct, files are being transferred at about 120 MB / minute... If that rate is the average, this will take about 40 hours :/ But hopefully the object storage dashboard is lagging. We will see.

Blaze ,
@Blaze@reddthat.com avatar

Fingers crossed 🤞

Shdwdrgn ,

Wow that's depressingly slow! My previous file server would transfer at around 75MB/s (not minute) and I thought that was pretty bad. Hopefully you can figure out where the bottleneck is at.

Sal OP Mod ,
@Sal@mander.xyz avatar

I thought that worse case scenario this would be an overnight transfer, but only 40 GB have been transferred so far.

The server's bandwidth use is well below the limit, so I think that the issue might be on the receiving end (Contabo's S3-compatible object storage).

I had to decrease the number of parallel connections because otherwise I would get too many errors and the transfer would stop, so it might not be the transfer speed of a single file, but rather the amount of time the server has to wait between initiating transfers. I am not sure, but at ~50GB per 12 hours this will end up taking almost 4 days... I will ask in the Matrix chat to see if this was also the experience of others who migrated.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • random
  • mander@mander.xyz
  • test
  • worldmews
  • mews
  • All magazines
    •