Pieter Hintjens - Our decentralized future
EuroPython Conference 2014 keynote
Video: https://peertube.magicstone.dev/w/4JSGy66bu1THNrmMY2UZjt
ZeroMQ is an organization with 500k lines of code and 400 contributors that's been operating for 7 years. The core library alone is 33k lines of code.
The ZeroMQ community doesn't really argue or fight. They don't do design, don't do meetings or IRC meetups. They have beer, that's about it. They don't do wishlists. They don't have roadmaps and there's barely anyone in charge.
It's an unusual community but not accidental. They've deliberately developed a process to get away from certain problems.
All of our problems came from centralization which caters to those who love control, power, and coercion. Meanwhile most of the world we live in is really decentralized. Nobody runs the food supply system in a major city.
Europeans know that when anyone organizes a central planning committee the results will be disastrous, so why do we tolerate this in our software projects?
If you look at the ZeroMQ community what you will see is a decentralized, asynchronous, concurrent, message-driven, lock-free, process with no one really in charge.
What we came from before was conventional, single threaded, highly-locked, shared-state, serial, no message passing, process run by a few guys. You'll be familiar with this sort of hierarchy as the daily environment in companies.
Stepping back from controlling, allowing the future to develop organically is kind of difficult but if you can't do that, where's the future? The future of computing is distributed so if you want a job you'll be able to make distributed systems.
The thing is we can't really build successful decentralized systems. We can only grow them.
Nobody built the internet.
What is the internet? The internet is a stack of protocols. RFCs are the internet. One of the earliest RFCs (RFC 3) is about writing RFCs.
The internet wasn't the first try. MSN 1.0 didn't even talk to the internet. The internet was born in competition with the largest companies who brought billions of dollars and tons of engineering talent to bear.
This collection of RFCs became a superpower, outcompeting MSN and AOL and yet nobody really invested huge amounts of money backing it. That showed the power of the market when enabled by accurate contracts. Contracts let us compete honestly.
A big misunderstanding on software quality
The old model of collaboration in the ZeroMQ community was like this. There was a guy who was the main author. He would write code. It would be very clever. A genius man. There was a friend of his who was the main maintainer and he could understand git and branches and merges and packages and makefiles and autoconf and all this very complex stuff and he could talk to the first guy.
Their brains would synchronize with this amazing shared state which involved lots of beer and meetings in strange locations. These two were a very good team. They made amazing software which was very fast and didn't crash. It was amazing and good.
Then there were all these other guys who were using it and contributing to it, but the contributions had to go through a mailing list to one of these two guys here, who would then look at the mailing list patches and say, "Hmm we don't like your patch. We don't agree with your patch. We think your patch is.... We don't like it! You know it's basically our baby so if we don't like it... well we'll think about it. Your code is badly indented. Hmm, no."
So there was this very strange communication process based on a pure monopoly of power. It was a classic, very serial, single-threaded, highly-blocking, synchronous process based on this fuzzy communication that nobody could ever track. No one ever knew what the basis was for a decision. The users had no idea!
After I had the fourth or fifth patch rejected I was like, "Why is my stuff being rejected? I'm trying to contribute here." Of course I felt hurt but what I really felt was if I'm getting my stuff rejected what about other people? This sucks. And this was my money on the table so I was kind of annoyed as well.
That wasn't the worst part. The worst part was what came out of this process was really rubbish. It took us like six months to make a stable release. We would spend six months just removing bugs from raw code. We would throw this stuff at users who would say, "Uggh it's better than being stabbed in the eye with a fork but it's really, it's really, why is this stuff so raw? Why is it so immature? Why is there so much untested code in this codebase?"
As we went through this process of maturing the codebase we discovered that a lot of the code was never actually used. It was written for no reason except that the person who wrote it thought, in his opinion, that people might need it one day. Or it was fun to write it. Or he was convinced he had to make it but he was wrong.
So what came out of the end of the pipeline, and if you look at the first versions of ZeroMQ, was kind of embarrassing. I mean they're good in certain ways but they're obviously completely wrong. That's why nobody uses them today.
Over time we've gotten packages and designs that have emerged which are more and more and more accurate. That's why people are using them.
The current process is very strange. Basically you send a pull request to master and I'll merge it. That's it. There's a little bit of filtering of insanity but not much. Even insanity I tend to welcome and embrace. If you're insane enough to send a really crazy pull request I kind of like your character. Come and join us!
Good software is relevant to a broad market.
A small team can write very high quality software in terms of its technical characteristics but they are incompetent in writing software that is relevant to a broad market. In fact only the market can write that software. So to make successful large systems you need to bring in the large market to build those systems. Distributed systems require distributed communities to build them.
As an organization we build systems that mirror our communication patterns within the organization. - Conways Law
If we are a decentralized organization with nobody really in charge but with very strong contracts between different pieces and different layers, with a very smooth process for improving those contracts. where the contracts let us compete very honestly, then what we build in the end is a software system based on contracts with nobody really in charge with competition between pieces.
The template for the new process with ZeroMQ was Wikipedia. Easy edits, easy reversion. Not six month cycles but six hour cycles. Not afraid of mistakes, empowering people to learn and fix those mistakes.
One of the ways we got quality to a high level was through users contributing test cases. A test case is a form of contract. Contracts that are fast and automatically tested are best.
Controversy doesn't really exist in points of view like in journalism, at least the controversy can be documented there, but this is because people can fork projects and make competing projects and customers can make the choice.
Rather than benevolent dictator for life, I've been more of a lawyer writing RFCs and solving problems in the process.
I don't try to push through any kind of technical vision. When I have technical vision I try to let the market tell me how to develop the thing.
There is authority but anyone can choose the authority they want. Mobility between jurisdictions. If you're trying to work with contracts and base collaboration around contracts, who enforces the contracts? Someone has to stop people cheating. There's a small percentage of determined cheats.
In the anti-software patent fight, there were a few guys who were determined to cheat, so now I'm very careful about identifying sources of conflict and those who could cause real damage to other people's work. Authority is there to protect, not to attack.
Question: Behind power there are people. Women have to fight their way into IT and have to justify how they are doing it. How do you ensure diversity? Answer: I can't speak for women but I can talk about our work to remove preconceptions that might affect discrimination. Some of our communication patterns may feel kind of masculine. We make a market by specializing and trading. Diversity doesn't say we have to do the same thing. Difference makes us valuable.
Question: “The brighter the software, the dimmer the user.” - Nick Carr (The Shallows, What the Internet is Doing to Our Brains) Will the people always choose the best software for their own benefit? Answer: As far as the market's ability to make decisions, what's most valuable is choosing the right problems, and solving the biggest ones first. So we depend heavily on the community to identify those big problems.
Question: Is this about management styles and does that require scale to obtain key resources? Answer: Chris Argyris has these two models, model 1 and model 2. Model 1 is a top-down learning process. You learn in university, you learn in workshops, you apply in your job. Model 2 is a circular system where you learn as you work and you apply that all the way around. It's not about scale though. Those models scale whether you're a team of 5 or 5,000. For ZeroMQ the goal from the beginning was building by learning, rather than building by design.