It seems like just recently, everything broke. Posts from some other instances (Lemmy 0.19 or kbin) aren’t showing up here reliably anymore, and then sometimes posts from a few days ago will suddenly show up. kbin.social has a banner talking about unusual “problems” without going into detail.
Does anyone know what’s going on or why? I’ve heard that Lemmy 0.19 has a problem where its outbound federation queue will sometimes die until it’s restarted manually, but that doesn’t seem like it should affect a lemmy.world <-> kbin interaction. Is it just Lemmy 0.19, or just lemmy.world, or are there multiple issues, or what gives?
lemm.ee took a while to upgrade from 0.19 to 0.191 and then had some serious performance degradation before undergoing some backend upgrade which finally completed a few hours ago. This should finally clear up the lag and backlog between lemm.ee and other instances, particularly lemmy.world, with which it had a lot of lag issues.
I believe kbin is updating the API, working on bug fixes and improvements and such, so the site is spotty right now. It said this would be happening for a few days, but that was a few days ago…
Interesting, thank you… so maybe it’s just a combination of 0.19 breakage and kbin.social breakage coming unfortunately at the same time.
Everything seems kinda fucked at the moment. Here’s the German community on kbin, fedia.io (mbin) and its original Lemmy instance:
https://kbin.social/m/[email protected] https://fedia.io/m/[email protected] https://feddit.de/c/dach
kbin.social really struggled with its issues affecting performance a lot which ernest seems to got a hold of by now. This didn’t happen on fedia.io (mbin) but the federation issues seem to persist with being almost 2 weeks behind now. Something is really off but it also seemed to all have kicked off around the same time? Kinda feels like Lemmy pushed something that affected both platforms differently. Either way, kinda makes the Fediverse somewhat unusable and pointless if stuff like this constantly happens.
Oh, just to add on to my previous comment to address the specifics: My guess is that feddit.de’s outbound federation is not running because of this bug in Lemmy 0.19. It will probably stay that way until the queue is manually unborked somehow, and even then, feddit.de will probably not be able to federate real reliably in general until the fix is released and they upgrade. Without knowing too much about it beyond some guesswork that’s my prediction.
Like I said in my other comment, I sort of prefer it that way. Call it the fediverse’s version of the “good fast cheap, pick 2” decision. You can have one only of the following options:
- You’re the product people are making money off
- It costs you money
- It doesn’t work perfectly all the time
I’m happiest with #3 personally.
Eh. I mean, stuff breaks sometimes; I get it. No one’s getting paid to maintain the servers, so any degree to which they function is sort of a bonus. Honestly I kind of prefer it that way, even if NGL it’s definitely irritating when stuff is broken for extended periods of time like this.
I’m currently working on installing kbin on a server of my own so I can try to make that my new home. I like its codebase much better than Lemmy’s, to the very superficial extent I’ve investigated the two of them, and I like that kbin attempts to interoperate well with both Mastodon’s part of the fediverse and Lemmy’s part. Presumably if I can get it working as a little server for me, it’ll be better-behaved without the load of a ton of users on it, and I can just hang out there and wait it out until people sort all of this out and it all starts working well.
Did you have a look at mbin, mentioned in an earlier comment ? It looks quite promising too
I looked at it super-briefly. Honestly, I tend to prefer the kbin “I have a vision for what I want this to be and I don’t want democratically inspired changes to my codebase” system over the mbin “if it looks good we’ll merge it” system. Based on my little bit of experience with it, the backend of kbin seems relatively well-organized, and obviously he put it together well enough for it to be able to work in production on a busy site, so I trust his judgement a little. I didn’t look in detail enough to make a real informed decision about it though. I do have some nitpicks with kbin (definitely as far as the frontend in particular) but relative to other Fediverse software it seems above average (although, that’s a low bar.)
Looking over mbin now, the one thing that seems like a serious point in its favor is talking about security fixes. Do you know anything more about that? Are there security things that the maintainer(s) of kbin are not wanting to address or something?
Edit: Oh, also, update: It looks to me like lemmy.world is currently configured to break all incoming federation from default-configured kbin instances. Maybe I am wrong about that, but that’s how it looks to me right now. I posted about it.
Glad to hear that kbin is also updating!
Kbin has separate problems because it is a different software. Cant speak about that as I dont know what their problems exactly are.
As for Lemmy, if posts show up after days it’s probably because 0.19 is more resource-hungry than 0.18 which causes some servers/instances to lag behind in outgoing federation.
0.19 has functional problems beyond simple resource consumption. It is fine, this is the nature of bleeding-edge software, just curious if anyone knew what was going on or if there were additional problems affecting the system beyond that one that I’m aware of.
I’m also aware that kbin is different software from Lemmy. Just curious if anyone happened to know what was up with them, or whether the problems were in some way related (e.g. failures to federate out from Lemmy 0.19 instances to kbin leading to some queue filling up so that kbin can’t get its usual work done and enters unusual failure states), and this seemed like a decent place to ask since some people on kbin obviously read this community.
Have a look at https://github.com/LemmyNet/lemmy/issues/4288, hopefully the fix is released soon
Oh look, it’s the issue that I linked to the fix for, in my comment 🤗. (Edit: It’s apparently merged now, who knows when that’ll translate to a release, but presumably it’s a priority since everything’s broken until that happens)
It is already in 19.1
0.19.2 you mean? 0.19.1 is from 3 weeks ago and the fix was merged 4 days ago.
Granted my instance is tiny, but have had zero noticeable issues with 0.19.1 since updating weeks ago. It had some increase in CPU on deployment due to DB and Pictrs also doing DB upgrades (or upgrade prep - depending if you stayed on 4.7 or went to 5), but has been running fine since.
Mines also tiny and my timeline is all fucked up since updating.
Posts I just make are 11 hours old the second I send them and I’m getting about 60% of the content I see when I browse the original instance of a community coming through on my side.
Check this
https://phiresky.github.io/lemmy-federation-state/site?domain=quokk.au
AFAIK, in order to fix outgoing federation issue, you need to restart Lemmy and DB every few hours. This is a temporary solution
Cheers for that!
1337 lagging instances
Woo!
Weird, did you run any of the alpha/beta/rc builds? I did and maybe that somehow made my upgrade smoother?
Posted with the wrong account below sorry lol I was saying that, without pointing fingers as i think Lemmy is a w e s o m e, this is a major drawback for new users that looks interoperability.
deleted by creator
I’d just like my pre-0.19.x
Prevbutton back, please.
