Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @[email protected] and @[email protected] for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn’t too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @[email protected] had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we’re very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

  • Tontoz@lemmy.world
    link
    fedilink
    arrow-up
    12
    ·
    1 year ago

    Really like seeing posts like this that explain what’s going on. Thanks for all the hard work!

      • gfdoto@lemmy.world
        link
        fedilink
        arrow-up
        0
        ·
        1 year ago

        Seeing this post immediately made my ditch my neglected Lemmy instance to sign up for an account on this server. Thanks for the hard work.

    • lenninscjay@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      agreed. as others have stated. Transparency. But also, new software, new tech (in a way, the fediverse/activity pub), and hopefully a new beginning. Happy to be here. If not technically on the ground level (lemmy has been around for a short time before we knew reddit was going to implode), close to it!

    • deweydecibel@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      How so?

      Admin communicating information about the site to users like genuine human beings, instead of the corporate-sanitized pablum reddit admins speak in? Yeah that’s refreshing.

  • flamekhan@lemmy.world
    link
    fedilink
    arrow-up
    2
    ·
    1 year ago

    Huge thanks to the lemmy.world team over the last couple of days to scale and maintain the instance! There’s a link for donating on the sidebar for lemmy.world - just a couple bucks a month can help us support this instance!

    • infotainment@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      It seemed to work fine at first, but starting today I’m noticing a huge number of async failures; I’ve had to refresh the page in order to get almost anything (comments, posts, communities, list of communities, profiles, etc) to load.

    • guyman@lemmy.world
      link
      fedilink
      English
      arrow-up
      0
      arrow-down
      1
      ·
      1 year ago

      Yep. So much lag, lots of ‘network error’ in Jerboa.

      Never change, free software </3

  • Anaralah_Belore223@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    edit-2
    1 year ago

    The update is good. Just needs more troubleshooting

    Trying to edit this comment

    Editing took between 4 to 11 seconds. on https://lemmy.world/c/lemmyworld

    Meanwhile, on other communities… 0.5 to 1 seconds

  • F3F3N@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    A bit off topic, but does anyone else hate how when you click on a post and then go back, the page auto-resets to the top? Wish it would remember how far down you scrolled and return to that point.

  • xikufrancesc@lemmy.world
    link
    fedilink
    Español
    arrow-up
    1
    ·
    1 year ago

    How beautiful and cute the new lemmy world looks. thanks boss.

    Que bonito y lindo se ve el nuevo lemmy world. gracias jefe.

  • Magister@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    1 year ago

    I cannot connect with Jerboa, it always says user or password incorrect 🙁

    Edit: worked after a few minutes

  • Secret300@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

    Had no idea nginx did that. There’s so much I need to learn

  • tmpod@lemmy.pt
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Awesome work!

    I’d like to know more about the exact container topology you have, since I may try something similar on my instance as well.
    Is it something like this?

    ┌───┐       ┌───┐
    │WEB│       │WEB│
    └─┬─┘       └─┬─┘
    ┌─┴─┐ ┌───┐ ┌─┴─┐
    │BE ├─┤IMG├─┤BE │
    └─┬─┘ ├───┤ └─┬─┘
      └───┤DB ├───┘  
          └───┘      
    

    Thank you! :3