This article touches on something interesting: community hosting.
I'd like to explore that. Specifically, the idea of small communities where a group of people maintains the underlying tech, and - kicker here - everyone in the community knows more or less everyone else in the community.
That offers a bit more security/safety/continuity than just self-hosting everything, while still not ceding control to a faceless corporation.
Granted, there will always be other reliances outside of the community - like internet and electricity providers - but a line has to be drawn somewhere.
Glad to answer questions if you'd like. I started and helped run a bandwidth cooperative from 2000-2015 or so. I decommissioned my last box in the coop at the beginning of this year.
The basic story, though, is that before the dot-com crash, a lot of SF nerds kept their pet projects on work bandwidth. That became risky during the crash, so I and some pals rented a fractional cabinet in a colo provider and split the costs. I think we ended up using 4 providers over the years and peaked at a full cabinet, almost all 1U servers.
I was glad I did it and at the end I was glad to be done with it. A co-op is hard to wrangle and it's basically impossible to make sure that the workload is evenly spread, so you have to be comfortable with the fact that somebody, probably you, is going to be doing a bunch of unpaid work, even if it's only keeping track of what needs doing and herding people into doing it.
Eventually, I decided running physical hardware was more hassle than it was worth to me. Trying to solve mysteries like, "Why does google sometimes decide my email is spam" was a multi-year effort that I never did solve, even though I knew people at Google. And I grew to dread the chance that something would break and I'd have to rush down to the colo, possibly having to return from vacation (or beg a friend to be remote hands). So eventually I shifted some of the stuff I was hosting off to service providers (yay Fastmail!) and the rest into Terraform-built slices of AWS.
I do sometimes miss the ability to fully run down a problem (e.g., by looking at mail server logs). But mostly it's a relief. I'm happy now to get my hardware kicks on things where uptime doesn't matter.
This is one of my main arguments for why we need to fill the gap of both turnkey disk arrays and data replication. Family photos, especially of kids, should not have to be in Instagram or Facebook, if you have three members of the extended family with any basic technical chops at all. You should be able to self host a triply redundant copies of the family photos, complete with bandwidth aggregation.
People in my parents’ generation all have stories of some grandma’s house fire eating the family hoard of photos, including the only copies of Great Grandpa Frank as a child. We don’t want Uncle Steve losing those pictures just because his house is in the 100 year flood plain.
I'd be happy with an E2EE solution that could be deployed on a cheap VPS. Unfortunately, all the photo and video hosting solutions I've come across lack E2EE. Building it out myself is a daunting prospect because it would mean putting together mobile apps for both Android and iOS that supported it as well.
It should be fairly trivial to deploy a number of existing FOSS photo hosting solutions to a server in your house though. Syncthing should make for trivial replication between the local instance and one at a relative's house. It recently gained support for E2EE (ie untrusted servers) so you could even throw it up on a cheap VPS as an extra backup and to allow for more reliable distribution between nodes. There's no bandwidth aggregation to be had here with current tooling though.
What software would you use for incremental sync from phone of photos and videos though?
Client applications have been my greatest struggle. I use nextcloud but their app (android) has plenty of flaws when the only purpose is one big rsync (no mirror, I will delete photos from my phone!), but all the options I find required foreground interaction or full directory sync both ways, which requires all data to be on phone too
Or how about accessing your personal library of data while outside of the house (like while visiting family)?
Backups and professional media work from home? Those need upload too.
Consumers need symmetrical data connections, or at least something much closer to symmetrical, than any ISP (in my area at least) has been willing to provide.
> everyone in the community knows more or less everyone else in the community
Sometimes "impersonal" is a feature, not a bug. I really don't want community sysadmins with access to logs of information about other community members. That has much more potential for abuse than a more impersonal service with a stricter expectation of privacy.
Create a good community encryption policy that protects members from each other. The kind of thing a larger org would never do because it might be legally prevented, or wouldn't want to completely exclude the possibility of future monetization opportunities.
> That’s true, but consider the example of small towns: just like what you’re suggesting, there are no secrets.
Are you suggesting that small towns where everyone knows everyone and people gossip a lot (for better or for worse) are equivalent to communities where a sysadmin knows everything everyone does on the internet?
A lot of universities' computer science departments do something like this. They'll have a cluster of machines in a room somewhere for undergrads and grads to SSH into and use as they please. Those are usually run by IT, but grad students in ML fields will often have another set of machines with specific GPUs in their own offices that are completely student-run.
I've been into this for a long time. I think that 50-250 families can support their own sysadmin, someone who works directly for them and manages all of their tech and interactions with the internet.
true, but this would be ideological rather than economical, which is why it would never happen, unfortunately.
a sysadmin will commoditize the process, make it cheaper, faster, more convenient, transferable (a family moves to a different place), etc. This sysadmin's business would grow, kill the competition, and then we would be back at the current state where we've ceded computing/networking to corporations.
Reminds me of my time on TF2 servers when I was a kid. Everyone knew everyone on our main server and it was a community. It's something that discord doesn't a decent job at capturing today, but unfortunately not selfhosted. Matrix is interesting, but I'm waiting for their new Go implementation (dendrite?). The deployment for matrix feels like it's heavier than it should be. I feel like I should be able to spin up a process and point to the port and call it a day. Maybe the overhead is from the need for authentication for federation, but I personally don't care about federation for my purposes.
I don't know anything about ansible, or much about docker, or self hosting. And I was able to set it up and it's working quite well for my family and friends. You don't have to enable federation.
Set federation_domain_whitelist to an empty list, and poof, federation disabled.
DNS settings are pretty easy too - especially if you can allow your instance to take control over an entire domain (and don't have to host other web services other than what the playbook supports). Don't need the SRV stuff here:
https://github.com/spantaleev/matrix-docker-ansible-deploy/b...
If you just have a private server for < 100 users, 1 vCPU and 2GB RAM is enough. I also use it for bridging to IRC using heisenbridge (which the playbook supports) and it's no problem on the tiny server.
Updates are very easy, pull the latest playbook, and run setup again. Done.
> Matrix is interesting, but I'm waiting for their new Go implementation (dendrite?). The deployment for matrix feels like it's heavier than it should be. I feel like I should be able to spin up a process and point to the port and call it a day.
Sounds like you, like me, have had your brain broken by the ease of deploying Go programs :) The current Synapse server is written in Python, so it's a bit of a trial. That said, I run it on a tiny linode instance and it Just Works after maybe an hour of fiddling around (I seem to remember something about DNS records being the fiddliest part to get right).
There are repos for debian/ubuntu so its just matter of adding the repo and doing apt install. You have to make config for synapse but you have to do it everywhere. You can use sqlite instead of postgres. It will go surprisingly far and you don't have to install postgres :).
But also, I am not sure that "parity with Synapse" is necessarily required or desirable. Sure, it needs to be fully functional and handle all the basic federatable communications, but its not like most small scale self-hosters need/want all the enterprise features being packed into Synapse. I think it is better for the Matrix ecosystem to have various server implementation that target different segments but can all still communicate.
Dendrite shipped 0.4.0 2 weeks ago, and 0.4.1 ships today (with a 10x speed-up in state resolution performance). Meanwhile it passes 92% of the server-server matrix compliance test suite (sytest) and 61% of client-server.
I'd expect us to ship 1.0 once these numbers hit 100%, which at the current rate should be before end of the year.
Do you not think it's in the best interest of Element Matrix Services to ship Dendrite? Presumably it would reduce their cost of operations significantly ...
One fine line that the community will need to tread is that it needs to attract enough people with aligned interests to socialise the costs of paying people to do the sysadmin work (or find enough sufficiently-motivated volunteers to do so, and develop procedures allowing these people to hand over properly when they lose interest), but remain small enough to "know everyone" involved.
As a participant in a number of small, mostly-volunteer tech community groups, I think this might be a difficult endeavour.
Shared responsibility is no responsibility. The biggest challenge with community hosting is that there would be no accountability. In academia, many CS labs / universities have shared computing resources to run compute intensive jobs. But it doesn't really pan out, until there are few big incumbents doing most of the heavy lifting. I rather deal with AWS/GCP/Azure..etc than deal with community driven hosting.
This has been on my mind lately a lot with Nextdoor. I have really mixed feelings about Nextdoor which is a slightly separate issue, but it always seemed to me that something like Mastodon or maybe even SSB would be an ideal use case in the same space as Nextdoor. You could have local communities around local servers, that have some natural reason to organize about that (geography), but are still loosely federated.
I'm not sure where my thoughts are going, as I'm not exactly surprised Nextdoor has more use than a more decentralized system for this use, but it's salient to me as I'd think something like SSB or Mastodon would ideally occupy the space that Nextdoor is occupying. I'm not sure if it is highlighting the legwork that Nextdoor did to build up its userbase (physically mailing people in a community), or the lack of technical sophistication of users in general, or the relative infancy of Mastodon/SSB/etc, or something inherent about getting a foot in the door with decentralized stuff in terms of mindset, or some inherent limitations of decentralization (can you really just compel/convince people to use decentralized services? People just use them).
I'm trying to imagine, for example, local police posting to Mastodon about some local safety issue in the same way as on Nextdoor. With Nextdoor, it's something known nationally, the state probably gives them recommendations, they just post to Nextdoor. Nextdoor might have even reached out to them. With e.g., Mastodon, I suppose I could see it being recognized as a thing if use got up, but where are they posting? The local popular servers? Do they run their own police server? Some kind of city government server?
This isn't a criticism of decentralization -- I'd like to see everything more decentralized. I just think something like Nextdoor is an interesting case to me to think through these issues because Nextdoor is so localized, and it seems like that's kind of the ideal use case for decentralized services.
> Nextdoor is so localized, and it seems like that's kind of the ideal use case for decentralized services.
Ideal use cases for decentralized services are also ideal business opportunities. You want to find collective action problems, charge rent for solving them, then manipulate your users to make you even more money from whatever resource is being collectively managed.
edit: I can easily imagine an app started to organize and coordinate people who wanted to volunteer to pick-up and clean public parks 10 years later becoming a app that was de facto required in order to visit a public park.
> walk a developer through building a rough clone of Mastadon
There are already full featured clones that federate. Pleroma and Pixelfed for starters. There are also multiple alternative frontends for both Mastodon and Pleroma. Granted none of those are walkthroughs but they do provide full featured examples in multiple languages.
Indeed. But what do you think (or suspect, or hope) would be different about a Mastodon instance playing the same role?
My hunch is that the characteristics of ND are caused by its perceived role, not the technology, hosting arrangements or (lack of) federalization/locality.
What is Nextdoor? Google showed a .com address and it was asking for an address, but it didn’t seem to recognise any address of my country. App isn’t in App Store either. Is that the service you’ve mentioned?
Is it something US only? Is it a social network that starts with your address?
Yeah I'm super interested in that idea as well. I wonder if there are already some initiatives in the US doing stuff like it? Would it be as simple as setting up a server in someone's house and then splitting the cost of electricity and internet? Would a multiuser setup like that work under a personal internet line, or would most ISPs try to shut it down?
We used to do this in the late 90's (when there were less hosting options in general). A bunch of us at work wanted to run our own sites and experiments online, so we pooled our money and built a server that sat at one of our houses. At first we just got a static IP for the server, but eventually as more people at work joined, the guy who had the server in his basement got a T1 line installed.
We all just split the cost of internet and server upgrades, etc, which may have come out to like $40 a year or something on average. We probably did this for a decade or so until the hardware got too old and there wasn't as much interest in maintaining it all.
While I just have a VPS now, I do miss that old server and all of us working on it, and literally being able to do whatever we wanted with it. All it takes is a few buddies to get together and try it out. Experiment and see what happens, let it grow organically.
I do this for some friends and their contacts. And while making a small presentation about what we offer with pros and cons, I added one point to both
Pro: you know the sysadmin
Con: you know the sysadmin
So as some already mentioned, it also can be a problem to trust some random company or someone you know.
This sounds like an awesome idea, and for many communities it might be, but having to trust someone you know personally for things like these can also be a source of drama when your relationship becomes bad for unrelated reasons
I'd like to explore that. Specifically, the idea of small communities where a group of people maintains the underlying tech, and - kicker here - everyone in the community knows more or less everyone else in the community.
That offers a bit more security/safety/continuity than just self-hosting everything, while still not ceding control to a faceless corporation.
Granted, there will always be other reliances outside of the community - like internet and electricity providers - but a line has to be drawn somewhere.