Agreed, I think the biggest issue though is just scale. It’s over 100 petabytes of data. Not outside the realm of big cloud providers to mirror, but they don’t really give a shit. It would require some sort of significant distributed software solution for the community to work with. Not impossible, but as far as I know, nobody’s taken up the mantle yet as I think it would need custom software just to begin the solution of how to distribute it as a sharded set of community mirrors, different people just mirroring individual pieces.
IPFS is the way to go IMO, it’s so perfect for archival that it pains me that it’s still pretty unknown
the fact that you don’t need any sort of central organization for everyone to help seed data is amazing, no more duplicate torrents splitting seeders, so long as you have identical data the network just figures it out.
If you have the hash for a piece of data you can just set a computer to watch for someone to start seeding it, even if the last time anyone saw the data was decades ago and a dude just found a CD in their recently passed dad’s basement, if that dude seeds it overnight and then their computer explodes, you’ve now downloaded it and it’ll remain available. It’s so fucking good.
HexOS has a plan for shared encrypted data. With the simplicity of installation and management it could take off mainstream as personal NAS are gaining popularity, but its still in early development.
We are screwed if the Internet Archive goes down, right?
Seems like a huge point of failure for one entity.
Agreed, I think the biggest issue though is just scale. It’s over 100 petabytes of data. Not outside the realm of big cloud providers to mirror, but they don’t really give a shit. It would require some sort of significant distributed software solution for the community to work with. Not impossible, but as far as I know, nobody’s taken up the mantle yet as I think it would need custom software just to begin the solution of how to distribute it as a sharded set of community mirrors, different people just mirroring individual pieces.
IPFS, GnuNet?
IPFS is the way to go IMO, it’s so perfect for archival that it pains me that it’s still pretty unknown
the fact that you don’t need any sort of central organization for everyone to help seed data is amazing, no more duplicate torrents splitting seeders, so long as you have identical data the network just figures it out.
If you have the hash for a piece of data you can just set a computer to watch for someone to start seeding it, even if the last time anyone saw the data was decades ago and a dude just found a CD in their recently passed dad’s basement, if that dude seeds it overnight and then their computer explodes, you’ve now downloaded it and it’ll remain available. It’s so fucking good.
HexOS has a plan for shared encrypted data. With the simplicity of installation and management it could take off mainstream as personal NAS are gaining popularity, but its still in early development.
Interplanetary File System can do it