We managed to resolve the issue and I have many ideas how to avoid this in the future 🤓

Time to come back, it is working again. Lucky break 😅

Slrpnk.net is unexpectatly back online 🥳

Nein, die alternativen, bereits quell offenen Treiber waren eh besser und jetzt halt die einzigen.

Ah, right they just swapped out without migrating anything. Thats not really what we are aiming for, but their comment confirmes that remote subscriptions might get lost.

I would need to check at database level, but I guess on the subscribing side it is only a list of communities to display? The real “work” for subscriptions happens on the remote instance to decide with activities to send.

But that indeed raises the interesting point on how to retain subscriptions of remote users. Anyone got an idea how that is handled on database level? Is it just the same list but for the non-local users that are mirrored in the database?

They actually migrated stuff from Lemmy? I thought it was a new instance?

I think we will try to come up with a theme for Piefed with similar colors.

The backups are most likely fine 😅

Sure, that seems reasonable. We will add that when the instance is back online.

If I forget, please remind me. For sure we will not just ignore you 😊

Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?

We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.

I agree that more admins with remote access would be good to have, and we have been working towards adding more (there are already more than one, who are not directly involved with slrpnk, but rather the mother project f-hub.org), but physical access will always be limited by the location (Azores) unless we move to an entirely different data-center (which in turn would make it hard for me to access the physical servers).

Yes, I also suspect that at a certain scale it would make a difference, but with a single Postgres database and no advanced clustering or so, the real-life performance metrics of a medium sized Lemmy instance strongly point to Postgres being the bottleneck and not the Python or Rust codebase of the rest of the software.

For regular users there is not a huge difference, but the web-interface is significantly faster and you can subscribe to topics that combine multiple communities. The disadvantage is less mobile app support. Right now only Interstellar supports Piefed.

The real advantage is for moderators and admins, as the Piefed developers actually listened to community feedback and implemented a lot of nice moderation features that require bots or annoying work-arounds on Lemmy.

And there are a few technical differences that make Piefed easier to administrate and troubleshoot from a sysadmin perspective.

I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.

We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.

I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.

Sure, but they will not replace hardware and set up remote access for you, which seems like this would have been required in this case.

It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a “real” data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).

Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.

Thanks.

Yeah at the very least this will result in some changes in the hardware setup to avoid similar situations. But of course it is impossible to forsee all possible problems.

I think now that Piefed has an API for apps, we will see some of them adding support soon. Overall I think the benefits of a Piefed migration outweight the disadvantages, but it remains to be seen if it is doable.

Yeah, the xmpp server is down too. That is something that bugs me quite a bit and I will probably move that one to an external small VPS to retain a more secure backup communication channel.