+972 52-548-6969
Building a social media platform without going bankruptPart II–Accepting posts
This design deal with creating what is effectively a Twitter clone, seeing how we can do that efficiently. A really nice feature of Twitter is that it has just one type of interaction a tweet. The actual tweet may be a share, a reply, a mention or any number of other things, but those are properties on the post, not a different model entirely. Contrast that with Facebook, on the other hand, where you have Posts and Replies as very distinct items.
As it turns out, that can be utilized quite effectively to build the core foundation of the system with great efficiency. There are two separate sides for a social network, the write and read portions. And the read side is massively bigger than the write side. Twitter currently has about 6,000 tweets a second, for example, but it has 186 million daily users.
We are going to base the architecture on a few assumptions:
- We are going to favor reads over writes.
- Reads’ speed should be a priority at all times.
- It is fine to take some (finite, small) amount of time to show a post to followers.
- Favor the users’ experience over actual guarantees.
What this means is that when we write a new post, the process is going to be roughly so:
- Post the new message to a queue and send confirmation to the client.
- Add the new post to the user’s timeline on the client side directly.
- Done.
Really important detail here. The process of placing an item on the queue is simple, trivial to scale on an infinite basis and can easily handle huge spikes in load.
On the other hand, the fact that the client’s code will show the user that the message in their timeline is usually sufficient for good user experience.
There is the need to process that and send it to followers ASAP, but that is as soon as possible in people’s terms. In other words, if it takes 30 seconds or two minutes, it isn’t a big deal.
With just those details, we are pretty much done with the write side. We accepted the post, we pretend to the user that we are done processing it, but that is roughly about it. All the rest of the work that we need to do now is to see how we can most easily generate the read portion of things.
There are some other considerations to take into account. We need to deal not just with text but also images and videos. A pretty core part of the infrastructure is going to be an object storage with S3 compatible API. The S3 API has became an industry standard and is pretty widely supported. That help us reduce the dependency issue. If needed, we can MinIO, run on Backblaze, etc.
When a user send a new post, any media elements of the post are stored directly in the S3 storage and then the post itself is written to a queue. Workers will fetch items from the queue and process them. Such processing may entail things like:
- Stripping EXIF data from images.
- Re-encoding videos.
- Analyzing content for language / issues. For example, we never want to have posts about Broccoli, so we can remove / reject them at this stage.
This is where a lot of the business logic will reside, mind. During the write portion, we have time. This is an asynchronous process that we can afford to take some time. Scaling workers to read from a queue is cheap, simple an easy technique, after all. That means that we can afford to shift most of the work required to this part of the process.
For example, maybe a user posted a reply to a message that only allow replies from users mentioned on the post? That sort of thing.
Once processed, we end up with the following architecture:
The keys for each post are numeric (this will be important later). We can generate them using the Snowflake method:
In other words, we use 40 bits with 16 millisecond precision for the time, 10 bits (1,024) as the machine id and 14 bits (16,384) as the sequence number. The 16 ms precision is already the granularity that you can expect from most computer clocks, so we aren’t actually losing much by giving it up. It does means that we don’t really have to think about it. A single instance can generate 16K ids each 16 ms, or about a million ids per second. More than enough for our needs.
The key about those ids is that they are going to be roughly sorted. That will be very nice to use later on. When accepting a post, we’ll generate an id for that, and then place that in the key/value store using that id. All other work from that point of is about working with those ids, but we’ll discuss that with more details when we talk about timelines.
For now, I think that this post gives a good (and intentionally partial) view of how I expect to handle a new write:
- Upload any media to S3 compatible API.
- Generate a new ID for the post.
- Run whatever processing you need for the post and the media.
- Write the post to the key/value store under the relevant id.
- This include also the appropriate references for the parent post, any associated media, etc.
- Publish to the appropriate timelines. (I’ll discuss this in a future post)
I’m using the term key/value store here generically, because we’ll do a lookup per id and find the relevant JSON for the post. Such systems can scale pretty much linearly with very little work. Given the fact that we use roughly time based ids and the time base nature of most social interactions, we can usually move most posts to archive mode in a very natural way. But that would be a separate optimization step that I don’t think that would actually be relevant at this point. It is good to have such options, though.
And that is pretty much it for writes. There are probably pieces here that I’m missing, but I expect that they are related to the business processing that you’ll want to do on the posts, not the actual infrastructure. On my next post, I’ll deal with the other side. How do we actually read a post? Given the difference in scale, I think that this is a much more interesting scenario.
More posts in "Building a social media platform without going bankrupt" series:
- (05 Feb 2021) Part X–Optimizing for whales
- (04 Feb 2021) Part IX–Dealing with the past
- (03 Feb 2021) Part VIII–Tagging and searching
- (02 Feb 2021) Part VII–Counting views, replies and likes
- (01 Feb 2021) Part VI–Dealing with edits and deletions
- (29 Jan 2021) Part V–Handling the timeline
- (28 Jan 2021) Part IV–Caching and distribution
- (27 Jan 2021) Part III–Reading posts
- (26 Jan 2021) Part II–Accepting posts
- (25 Jan 2021) Part I–Laying the numbers
Comments
This pattern - put something on the queue and do the rest of processing later is applicable pretty much everywhere, and especially in transactional systems / relational dbs. The default is to do everything in one transaction but it can get big if the application logic grows and becomes complex - very often you dont even realize your button click triggers a chain of events and hundreds of database operations and finally some hard to debug performance problems. So it's really good if you're able to split the logic into the minimum amount of work that has to be done synchronously and the rest - later (like in interrupt handling, you do the bare minimum and put the rest of the logic into a 'bottom half' procedure that runs later) To make this easy every database should offer some basic message queuing function, so that you can access the queue in a transactional way and without resorting to distributed transactions - just like another table insert or whatever typical SQL operation.
Rafal,
I agree. In most DBs, you can just define a collection / table of commands, and insert to that and have readers process that.
I understand the need to have independent social media platforms but I think the cloud-based approach you're taking in dead born. The issue with social platforms is no longer technical -- as you mention, it has become pretty easy to build and scale with the cloud. The real issue is strategical and political. Who controls social media controls opinions, who controls opinions control societies. The struggle to control social platforms is huge. Look at what's happened to Parler, a platform that dared not to be aligned with the GAFAM political line. The effort to put TikTok under US sovereignty instead of Chinese has been huge. As are the efforts of China, Russia and even lately Poland to control the social media operating on their territory (for very good reasons I think, since these platforms have been and will be weaponized for social engineering).
So, the main feature of an alternative social platform is: how to work avoid interdiction (in the military meaning). What if your opponents are 90% of all world governments plus 90% of blue chip companies? You clearly cannot go for a centralized, cloud-based design.
Let's explore the decentralized design (I've been thinking about that as soon as I started to understand politics around 2001-2003). We can take inspiration from Bittorrent where every node caches a part of the internet. As soon as I publish something, my followers would download my content and would cache it locally. There would be no central node to attack. This would make interdiction impossible or much more difficult.
The next problem that comes is trust. Is everyone hosts everything, how do you know who wrote what? How do you trust an article that comes from a source that you don't know certainly. This is the problem the trust. Solved problem, you may way, we have done it with certificate chains. Except that the current CA-based trust system is still a centralized one. If you are building a social network for political opponents, you are pretty sure that your opponents with implement censorship at the level of CA if they can. Who should be the root of the trust network? Microsoft? Google? It does not work in this scenario. The only way to guarantee freedom of speech is that _you_, the user, is the root of the certification authority, and that "friends of your friends are your friends". You get a trust network, which can be resolved pretty elegantly by matrix theory with an algorithm that's close to Google's PR one. Except it has to be solved for each individual.
But we are not done.
The nodes can lie. They can cache, not the "whole" internet (or a random fraction of it) as they claim, but just the opinions they like. An attacker could saturate the network with "friendly" nodes which would favor "correct" opinions, which is basically what GAFA is already doing now.
Another problem is that you actually must design a censorship system. An attacker could force a node to cache content that is forbidden for good reasons, for instance child pornography (all laws against privacy always start with child pornography). An attacker would then sue a group of nodes (because nodes are hosted by citizens who are still identifiable physical person, subject to local law and to physical/financial coercion) and the rest of the crowd would get afraid and leave the platform. Therefore, the system must also implement a distributed, trust-base system of content withdrawal -- in other words, you must implement censorship in your distributed social platform anyway. So if my trusted friend marks some content as illegal, my node would remove that content from the cache so I could not be sued for serving illegal content. But what if there is a massive attack on the network to upload tens of thousands of child pornography? Defending against this is difficult without AI (that must runs locally) and an army of "community stewards", the guys who review the content (and then so often commit suicide). So, that sucks. Implementing censorship is necessary for an open platform, but it is very difficult. Established platform have vested interest in requiring a high level of censorship from new comers to the market - this is what they are doing with Parler.
Bottom line: I haven't found a workable solution to this problem. Technology is not the problem -- it's rather a solved problem -- and a cloud-based architecture is not a possible solution. The real problem comes when you take into account the strategic and military aspects of social platforms.
Gael,
I think that you are essentially correct, and I wrote a bit about that in the past, see the posts immediately following Parler's shutdown.
The key difference here is to change the conversation from having a Node to Identity. If you have a server for a person, or a group of well known people, that is a different game entirely.
Trust work, because I know who I'm talking to. Can enforce that cryptographically or not. But attacks such as forcing you to host illegal content will not work. If you want to add additional content to the network, you have to do that under your own identity. Basically, your own server. You could enable annotations from other locations on your messages, or you could use what is effectively a "retweet" to handle that.
That system is called blogs, where you may have a personal blog or a group blog. Comments are handled directly or through a separate service. For example, "disquss" or similar are probably big enough to handle AI agents for filtering, etc.
Note that blog spam is still a problem today. Censorship can be handled by "reposting" content with attribution.
@Gael Fraiteur You should really take a look at Solid which is an MIT project started by Tim Berners Lee the inventor of the internet. It's an interesting project where each individual owns their own data and can move it from place to place. I looked into implementing something last year, it's in it's infancy but I think something like it could really work.
@Oren Thanks for writing this. I have often thought about what I would use to build a Twitter clone. Learning about Snowflakes has been particularly interesting. What do you think about using an actor model based system like Microsoft Orleans backed by Azure Table storage or some such cheap storage instead of using a queue?
Yes indeed I have read your article after Parler and started thinking about that. But I am a very slow thinker and I'm only pinging now.
I think blogs are still centralized because they rely on DNS, which is centralized. Platforms like Blogger, Wordpress, Medium are centralized too. So if you push paranoia further, RSS is not a sufficient platforms. (Btw I'm a big nostalgic of blogs too, back in the day you could be esteemed for your technical contributions and not for your online social skills or political opinions. I miss that. RSS is now my only source of news - I've never had TV, and quitted Facebook and Twitter last year)
So, a server can be disabled through DNS or attacked through DDos or malwares. Now Cloudflare accepts to protect politically sensitive sites. They may also be put under pressure and stop doing that. A peer-to-peer network like Bittorrent, Tor or Bitcoin is more difficult to attack except if governments end up delivering authorizations to open a server port (which the panaroid me assumes is the predictable future).
That's why I think a peer-to-peer network with cryptography-based trust networks (and not hierarchies) is the way to go for politically sensitive content.
@Muhammad Thank you for the tip. It looks to go in the direction I'm dreaming of.
Muhammad,
A queue is better, period. Once you have a queue, you can do whatever modeling you want on top of that, but a queue is the first thing that you want.
And queues can be implemented using table storage, but there are dedicated options for that, which are probably better all around
Gael,
DNS is NOT centralized. That is pretty much the core feature there. Platforms may be, sure, but you don't need a platform to host a blog, see this one, for example :-)
Why isn't RSS a good enough platform? There are technical issues, sure, with many people reading the feed all the time, but beyond that?
Note that at some level, if you fear censorship, domain names are the easy target. Subject to regulation, etc. However, you can get domain names from different jurisdictions. And there are onion addresses, too.
DDoS is a separate issue, and yes, that is a concern. But the question is what are you trying to protect yourself from. If this is from a nation state, they have such things call cops that can come to your house and arrest you.
Whatever protocol you are using isn't going to help you there.
@Oren DNS is distributed by design but has a central authority and other weaknesses. It has been already used by China to implement censorship. Also, Google is pushing towards DNS over HTTPS, which is centralized because of the default configuration of Chrome, and Google has already been active with censorship. I think we agree that DNS is a potential weak point.
Regarding the use of violence by the state: I agree, the state can exert this power , but currently only on _publishers_. What we want is to protect _readers_. Some journalists have already emigrated to countries of another geopolitical block to escape the power of police and secret services. What we want is that readers can still have access to their articles even if they are published on a part of the internet that is in another juridiction. I think this will be at risk in a few years even in the West.
Gael,
The recent issues have all been targets on the publishers. The readers haven't been touched, because it is so much more effective to get shut down the publishers.
Also note that if you are thinking about the nation state, they have far better tools to get things shut down. The easiest way is to simply block IPs / IP ranges. See Brazil shutting down Whatsapp as a good example.
Comment preview