Sometime early this morning, our database started having issues, most likely due to a disk corruption issue.
Unfortunately the postgres database was unrecoverable, and we had to restore from a nightly backup, so a few hours of posts and comments will be missing.
Apologies again for the downtime, and for the disruption.
You must log in or register to comment.
Oh holy cow, those servers are running at some pretty high temps!
Liquid cooling required
Holy fuck. I thought this was a joke before I saw who was posting it.
Awesome that it’s back up! Thanks for all your hard work
o7
No worries! Is there an announcement anywhere in case it’s down? Mastodon? Matrix?
We posted in some of the lemmy matrix chats, but ya we should also use our mastodon too.
Yep.
No apologies necessary. Y’all have been working overtime to get things back in order, and I appreciate it!
No worries, glad to hear you got it up and running again :)
looks like some replay action too. my instance just got spanked.
When I saw LemmyMl was down I went to sleep and had a dream where I was eating fries extremely large fries that that were labeled “Sadistically Large” (included chicken fries as well as potato fries) and drinking probiotic shakes so dangerous they literally had to be bundled with a free antibiotic shake by law which of course I took one sip of in front of the clerk to calm them down before guzzling the other shake down with impunity. Woke up before I found find out what the burger was. Maybe those anti-woke types are on to something.
How does lemmy federation work in this case? Conceivably after being restored from backup the lemmy.ml instance could see those few hours of lost history as federated to other lemmy instances and resync it back as the host instance. Obv I’m vastly oversimplifiying things but what happens today?
I believe they’re lost, as recieved apub content doesn’t have the original post / comments, but I’m not totally sure. cc @nutomic@lemmy.ml
It strikes me that there is the potential to use trusted remote servers as a means of recovering the lost data. I mean, nearly every lemmy instance except lemmy.ml will have copies of the missing data, and given the hugely redundant availability of that data (including the ability to compare from multiple sources to establish/verify trust), using that data to rebuild missing content seems like it could be useful functionality.
Welcome back!
So happy it’s back!
Only was a cable
Do you know that you are doing to me?! I actually had to work while lemmy was down…
Join us! Enjoy being among posters that never stop posting
Idk what Lemmy.ml is is it some sort of liberal instance that refuses to engage with chapochat
the owner, dessalines, is a massive comrade and writes fantastic essays. you should check out his stuff
I hope none of you have blocked my community because would make me angry
