Server-side Backbone Ideas

created Apr 16, 2019

I'm ignoring the obvious admin tax issue, which is a significant problem when discussing a decentralized/federated/independent web that involves non-tech users.

Apr 16, 2019 HN comments from this thread, started yesterday:

There are two problems with decentralized / federated p2p approach.

(1) Everyone pays for the infrastructure. That is, you pay for storing and serving the videos in a p2p video service. You pay for crawling and forwarding the search results to users of a p2p search engine. It may be unnoticeable if you run a beefy desktop / server on a 100 Mbit connection from a major metropolis. It becomes worse on your laptop or mobile on a metered connection, or on a rural DSL.

(2) The speed and reliability of a distributed system depends on all its nodes. SREs running a bunch of servers at many nines is one thing. You and me running goodwill p2p nodes is another. To achieve comparable reliability, many more nodes would be needed. Also, with enough nodes, p75 latency may be okay, but p99 is going to be poor. Compare to torrents, where last month's content may arrive in seconds, and obscure content from 5 years ago may take days.

The sad thing is that most people are not ready to shell out the amount which marketing companies pay for the ads they see. Even less are they ready to make their experience slower and requiring more technical chops.

For the few that care, there's PeerTube, ipfs, distributed search, etc.

If Mozilla only tries to pursue goals that have a chance of global success, I'm fine with that. Let them keep Firefox and Rust in a good shape; more than enough for me.

Reply comment where I was interested in the first paragraph.

You're talking like we're still back in 2010 or something.

(1) Digital Ocean, Vultr, and Linode Boxes have a 300 Mbps port size, under polite use. That goes for every instance type. Search engines like Manticore or anything besides memory eating java things like elastic search can run under constrained environments in a cluster and when used with SSDs are really damn fast. Then there is Cloudflare who's services, as long as you don't abuse them, can help with massive distribution. Not to mention all of the storage backends that exist with really, really cheap storage. Backblaze, Wasabi, hell when some ingenuity you could probably use Dropbox apis for long term unlimited storage. This is your backbone.

(2) For extra redundancy and better distribution you can add in p2p which if used correctly can propagate new data to users within reasonable time frames and allow for data storage to be more resilient. Easily less than a minute in streaming environments.

Also, while IPFS is nice it doesn't have the resiliency necessary for the future. Mozilla won't do this but neither will any other browser making company.

Obvious reply to that reply:

Sure, the infrastructure is there for the tech-savvy.

But people who has at least a vague idea how to configure their own server are a small minority.

Which is why people use Facebook, Instagram, YouTube, etc.

Yada, yada, yada. This part of the thread interested me: