From 5871baf46bccfd25b62fbeaeac1c206cce813e5a Mon Sep 17 00:00:00 2001 From: Luna Mendes Date: Thu, 15 Nov 2018 16:56:39 -0300 Subject: [PATCH] Add internals.md --- internals.md | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 internals.md diff --git a/internals.md b/internals.md new file mode 100644 index 0000000..c7a7533 --- /dev/null +++ b/internals.md @@ -0,0 +1,57 @@ +# Discord's "stack" + +Technologies Discord uses to provide the service, gathered from +[Discord's engineering articles](https://blog.discordapp.com/tagged/engineering) and some other texts: + + +[Cassandra]: https://cassandra.apache.org +[MongoDB]: https://www.mongodb.com +[Flask]: http://flask.pocoo.org +[Punt]: https://github.com/discordapp/punt +[Loqui]: https://github.com/discordapp/punt + + +## Storage: + - [Cassandra](http://cassandra.apache.org/) for storage + - They used [MongoDB] as primary storage, but moved to Cassandra, [source](https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7)). + - It is still unknown what role MongoDB has on the stack, but it is still + a primary part of the service: [source](https://status.discordapp.com/incidents/rkj7rx8f4865) + and [source](https://status.discordapp.com/incidents/2cb5jc53jq28) + + +## Programming languages: + - [Elixir](https://elixir-lang.org/) for any real time communication, + [source](https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b). + - [Python](https://www.python.org/) for HTTP/REST API, + The best guesses to which framework are [Flask]. + - [Go](https://golang.org/) for the embed servers and one element of their logging(see [Punt]) + - [Rust](https://www.rust-lang.org/) for many parts of the Discord Store, most commonly: + - [Game SDK](https://discordapp.com/developers/docs/game-sdk/sdk-starter-guide), + to make Discord integrations for any game. + - [Dispatch](https://discordapp.com/developers/docs/dispatch/dispatch-and-you), + which is a tool to send assets to Discord's servers. + + +## Distribution: + - [Loqui] for node communication. + + +## Logging: + - [Punt] in favour of [Logstash](https://github.com/elastic/logstash) for logging. + - [Elasticsearch](https://github.com/elastic/elasticsearch) that powers the search feature for users and powers logging. + - Sources about logging: Punt and [this issue](https://github.com/elastic/elasticsearch/issues/20354) + - [Source about message search](https://blog.discordapp.com/how-discord-indexes-billions-of-messages-e3d5e9be866f) + + +## External services + - [Google's Cloud Platform](https://cloud.google.com/) for their infrastructure, + [source](https://status.discordapp.com/incidents/rhvp2tn7g0zc) + - [Cloudflare](https://www.cloudflare.com/) as a proxy to almost all services, + voice servers need to be direct connections, so they don't pass through CF. + + +## Many other sources + - [Description of a full outage where they had to reboot everything](https://status.discordapp.com/incidents/dj3l6lw926kl) + - [`sessions` and `presence` clusters get rebooted due to a host error in a `guild` node](https://status.discordapp.com/incidents/ywdwttd6b0hg) + - [Repeating message sends due to errors in the `push` cluster](https://status.discordapp.com/incidents/93kyyctg0wf3) + - ["furiously" spinning an nginx cluster due to an error in GCP's load balancer](https://status.discordapp.com/incidents/rhvp2tn7g0zc)