Hi all, We will be attempting to migrate Pict-rs tomorrow starting at 1700 UTC. Between now and 1700, image uploads will be disabled to try and avoid any data loss. You will still be able to like, comment and post until then. During the upgrade window, Pict-rs will be offline, so viewing images might not work. In theory CloudFlares CDN cache should cover us during the window, but YMMV. Hopefully everything should go smooth, but we do our best to make sure we have backups!
You can monitor our site health at https://dash.lemmy.world and https://status.lemmy.world
We will post updates to https://mastodon.world/@LemmyWorld as well.
Cheers, LW Infra Team 💗
Update This change is done. Images are now hosted on S3 storage and uploads are re-enabled.
Love your work in managing Lemmy’s infrastructure at scale. Would like to join you sometime
Feel free to apply https://team.lemmy.world 🤓
Love how openly you guys communicate about the management of LW. It’s interesting for anyone with an interest in self hosting things to see how you’ve scaled up. Keep up the good work!
Thanks so much 💗
Is this migrating from pict-rs to object storage or upgrading pict-ra from 0.4 to 0.5?
Object storage. Sorry if that was unclear ❤️
Ah nice congrats on the move
Thanks! It’s a lot of work for the behind the scenes stuff, but we have a great team that really cares 👼
Where can I read more about why this is happening?
Actually 2 reasons why. First and most important, this was needed so we can use tooling to detect and remove CSAM material. Secondly, it’s because this is more cheap and scalable than having it on disk.
Thank you for doing what you do.
I’ve heard that being a content moderator is absolutely soul-crushing work. Here’s hoping the new tooling lets you automate away the worst of it.
I have no insight into why it’s being done in this instance, but object storage is typically used when you want to move away from storing things on your web server with “fixed” storage, and instead store it in an “infinitely” expandable storage system. It is also much easier to manage when you have multiple servers as it’s separate and shared.
Right on. This seems relatively important considering pict-rs is the lemmy default. Seems like a change that should be considered for the core repo.
The problem with making S3 config the default is that it would add friction for new admins. If you’re just starting to play around with self-hosting Lemmy then it’s 100% certain you have a local disk but not nearly as likely that you’ve got an S3 bucket ready to go. Making S3 the default would force those new admins to either tweak their configs or figure out how to get an S3 bucket before they could get going. Using local disk for images is fine for smaller instances and self-hosts, but for cost and scaling reasons having an option to switch to S3 is really valuable for larger instance like lemmy.world. Only the top 10-20% of instances are likely to really need to make the switch to S3, so Pictrs defaulting to local disk makes sense I think.
While I agree on the technical aspects, I’ll say that small instances benefit a lot from s3. My single user instance currently has over 25 Gb of media cache… Object storage (hosted on my NAS ) has been a godsend.
Have you used MinoIO? I’ve used that before for local S3 and it works quite well.
What kind, or host?
That we moved to S3 from filesystem
Right, but you said not AWS so I’m curious what either cloud-based S3 service, or baremetal appliance / stack you’re using with S3.
Curious as well. Could be Cloudflare R2 since they said they’re using Cloudflare elsewhere, and it has zero egress fees. Or something like Minio. Obviously just wild speculation on my part.
Almost seems like he’s being cagey about not answering for some reason
I saw “Pictrs Migration” and I thought was about to open a map porn post regarding the movement of ancient Scots.
Guyz, what is Pictr? Is it like Imgur?
Also, [excellent]
Kind of like Imgur, it’s a barebones Image hosting server which is what Lemmy uses to serve images uploaded to the instance. It makes it easier otherwise people would need to use things like catbox or imgur itself to host images.
I do feel like support could be better with it though, like if it gave us an interface where we could access our images from our account, or at the very least one for admins.
Congratulations on the move!
Is it amazon s3 storage? How much are we paying for it?
We are not using AWS S3, we try and keep costs down where it makes sense 😉
Bro who are you? I don’t see your name in the mod list
It says in their profile.
Lemmy.world tech lead.
Oh shit! Sorry I missed if
In case you’ve not visited the team page. Here is the LW staff.
Like myqqq attached is
deleted by creator
Ight awesome
Thanks. Can we also please block threads?
There will be a announcement soon and even if we dont defederate you can simply block threads yourself in 0.19!
I hope you give us the option. I want to see what threads is like
What is threads?
Meta’s version of twitter.
0.19
ETA for update to that? Seems to work fine on the servers I’m aware of.
In the next week we will upgrade our test environment and test our automation there, if everything works fine we will then be doing it probably the week after that.
Thank you for your work.
Thank you! You are making lemmy great!
Want to say this because the release notes make it perfectly clear. Instance blocking is not and cannot be considered an alternative to defederation.
Users can now block instances. Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesn’t affect users from the blocked instance, their posts and comments can still be seen normally in other communities.
Not a replacement for defederation, it’s still needed to keep instances healthy. Plus the fact that Lemmy doesn’t really even have blocks, they’re just mutes called blocks (which are only useful under the assumption that the blocked account is innocent, and the blocker is just having a fit or doesn’t have credibility), if we had a way to restrict malicious users then I would say being more hands off isn’t that bad but as it stands there isn’t so defederation and moderation is still important and can’t be substituted with blocking.
But for those “I hate that instance BLOCK IT!!! FOR MURICA!!!” A instance wide defederation is too much. But for those really bad like csam or racist, radical instances it is the best to defederate.
That is a fair point, and I would agree if the only people calling for threads defederation were the snowflakes (the overly sensative opininated users you speak of). However many people have highlighted important reasons why threads federation isn’t a good idea.
Reasons such as:
- Embrace Extend Extinguish Concerns (Already discussed in great detail elsewhere I won’t beat a dead horse)
- Facebook has a very bad track record when it comes to shady and malicious practices
- Facebook has a very poor track record when it comes to effectively moderating spam and hate speech
So while there are some bad and dumb reasons people argue for defederation there are legitimate concerns over this and threads.
Side note (also a bit of a rant):
I feel like the sensitive snowflake users ruin it for people who have legitimate and valid concerns, they undermine efforts to make spaces better because people are so used to dealing with them that when a person brings up something that could be a problem or should be addressed the default response is to dismissively just tell them to block users or instances, under the assumption that a user is one of those snowflakes. (It happened back with the Rammy situation, when the user posted to alert the fediverse that their instance was rogue).
Personally I feel like there should be more severity when it comes to users who make false/opinionated claims so that any claims in general aren’t treated as false or non-credible by default, but I also realize that would be difficult given the size of the instance and also isn’t really my pace to decide since I’m not responsible for user moderation outside of my own communities or ones I mod in.
Just some things to think about. Also sorry for the late response, been having trouble on my Lemmy app. I guess better late than never though (lot of times it is never when I’m late to check or respond).
We will see. We will be of course not cu ksu king if we see a lot of just spam, hate or just ads to “Come over to threads” ads in any way we will look into defedarating them.
I implore you to read the article I just published regarding how people become radicalized. Your decision is ultimately going to affect the well-being and safety of the users of Lemmy.World.
I see the suggestion of personally blocking Threads purported as a solution, but that is limited to a personal level. That does not solve the problem of radicalization of the user-base in general. Users who have not blocked them or people viewing without a profile will be potentially exposed to misinformation and extremism.
As documented in my article, radicalization happens through a social contagion effect, in which repeated exposures reinforce the radical beliefs. People who otherwise would not believe such things become radicalized through repetition, and social media websites (including reddit and lemmy) serve as catalysts for radicalizing such vulnerable people.
Incels were spawned through their echo-chamber on reddit, and slowly radicalized to the point in which they have committed mass murder and acts of domestic terrorism. Read sources The_Donald on reddit was originally a satire subreddit that became a pro-trump echo-chamber, which snowballed into real-world extremism and had to be banned.
Threads has been subject to mass amounts of radicalizing, extremist content, and there have also been instances of users having personal information doxxed on Threads due to Meta’s information-harvesting practices. [1]
Threads was marketed to be open to ‘free speech’ and encouraged the Far-Right movement to join, who have spread extremism, hate, and harassment on Threads already. [3] Threads has been a hotbed of Israel-Palestine misinformation/propaganda. [4]
They fired fact-checkers just prior to Threads’ launch [1], however they claim they will have 3rd party fact-checkers next year. [5]
Meta/FB has a rampant history of illegal and unethical practices, including running experiments on their users which affected their moods and induced depression in many uninformed, non-consenting subjects. [6]
Meta/FB/Instagram also have a strong history of facilitating the spread of misinformation and extremism, which contributed to the January 6th insurrection attempt. [7]
If exploding heads was defederated with because of this sort of toxic extremism, why would you want to federate with a platform plagued by that same content? One known for shortcomings moderating it? And one which comes from a company with a long history of unethical practices regarding users?
With their rampant history of unethical and illegal business practices, I encourage you (the Lemmy. World team) not to federate with Threads for the well-being of the users in this instance. I have consistently seen a larger consensus of Lemmy.World users opposing federation than being interested in it.
A significant portion of Lemmy (in general as well as Lemmy.World) users migrated here from reddit after being disappointed in the reddit administration’s failure to represent their users’ interests. Unlike most of the users on reddit, a significant proportion of the users on Lemmy.World are the active users who cared enough about their principles to leave and start over here.
I have really valued the leadership of Lemmy.World thus far and have donated to you since my first week here, because I believed in this instance and our community. And I still do.
But I will be cancelling my donations and likely finding a different instance whose ethics are more aligned with my own, should you choose to federate with Threads. I have tried to be an active user and promote discussion and sharing content on Lemmy.World. But in federating with Threads, I believe you will lose a lot of active community members like myself.
I sincerely hope you will take this into consideration in making your final decision.
Yadda yadda yadda… You can just block them yourself. No need to force everyone to miss out on them, just because you may think it’s wrong to federate with them.
I already acknowledged how that does not protect the community itself. Including from potential unethical practices/non-consenting experiments by Threads against users, seeing as how that is an established practice of the parent company.
If Threads is federated with Lemmy.World, could they not carry out actions that would in-turn affect users from this instance?
For example, say they manipulate the hot posts on Threads to be depressing content to experiment on moods (again). Would that manipulated feed not affect users of this instance with the federated content?
Or what about how FB and Instagram promoted misinformation many times over, and which is a current problem on Threads? How is this different than defederating from Exploding Heads?
It seems wild to me to federate with a notorious bad actor like Meta/Threads. It’s so clearly in opposition to cultivating a healthy fediverse IMO. The immediate payoff is more users in the Fediverse, but I think it will be a mistake in the long run.
People advocating for trusting Meta/Threads this time after their history is like believing that touching a hot stove won’t burn me this time.
If it seems wild to you to fedrate with Metq, why don’t you just defederate yourself then?
Because you actually have that power. You yourself can decide.
If .world deferates the entire instance, people like me who wants to federate with them no longer has a choice.
So do you feel the same way for exploding heads? I don’t think platforms should be given exposure when they’re going to lead to more radicalization.
And I was intending to create some more communities here, but I’m reluctant to do that with potential Threads integration.
For instance, I was thinking about making a community for my city, but I wouldn’t want Threads integration to be incorporated into the community. Would I have a way of the community itself blocking Threads?
But what it ultimately boils down to is the danger that the far-right poses, especially with trump’s dictator remarks and threats against political opponents as well as the justice department.
These platforms and the way they are managed have real-world consequences. January 6th is an example. The_Donald and incels on reddit lead to real-world violence and murders. 8chan users have a history of encouraging mass shooters.
This is such a tumultuous time with dangerous and fascist rhetoric becoming normalized, and there is a danger of real-world violence. There’s also plenty of examples already. Far-right extremism has been on the rise around the world and is a global concern.
I think the ethical decision is not to enable such platforms that enable hatespeech, harassment, and radicalization. But I understand that you disagree with me.
Actually I don’t feel that for any instance, be they extreme right left or center.
Let the users pick for themselves, instead of forcing them into a position.
From the changelog:
Users can now block instances. Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesn’t affect users from the blocked instance, their posts and comments can still be seen normally in other communities.
Not how instance blocking seems to work in 0.19, it’s the equivalent of community blocking, won’t affect users or their posts. So not a replacement to defederation or moderation.
This stance is very weird since it implies that users are entitled to something on a server we don’t own or control, but really we’re as entitled to federation as we are to these accounts. An admin can ban your account off their server forever because they felt like it, and those same admins can defederate your favorite server for the same reason. The difference between the fediverse and social media is that there are other instances with different rules you can go, and different federation stances. Your freedom is having other options.
Plus this is made even easier given that Lemmy now has a native way to export user settings in 0.19, which definitely does make life much easier for those who wish to migrate.
From the changelog:
Users can now export their data (community follows, blocklists, profile settings), and import it again on another instance. This can be used for account migrations and also as a form of backup. The export format is designed to remain unchanged for a long time. You can make regular exports, and if the instance becomes unavailable, register a new account and import the data. This way you can continue using Lemmy
