Add internals.md

This commit is contained in:
Luna Mendes 2018-11-15 16:56:39 -03:00
parent 7d14ab7d36
commit 5871baf46b

57
internals.md Normal file
View file

@ -0,0 +1,57 @@
# Discord's "stack"
Technologies Discord uses to provide the service, gathered from
[Discord's engineering articles](https://blog.discordapp.com/tagged/engineering) and some other texts:
[Cassandra]: https://cassandra.apache.org
[MongoDB]: https://www.mongodb.com
[Flask]: http://flask.pocoo.org
[Punt]: https://github.com/discordapp/punt
[Loqui]: https://github.com/discordapp/punt
## Storage:
- [Cassandra](http://cassandra.apache.org/) for storage
- They used [MongoDB] as primary storage, but moved to Cassandra, [source](https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7)).
- It is still unknown what role MongoDB has on the stack, but it is still
a primary part of the service: [source](https://status.discordapp.com/incidents/rkj7rx8f4865)
and [source](https://status.discordapp.com/incidents/2cb5jc53jq28)
## Programming languages:
- [Elixir](https://elixir-lang.org/) for any real time communication,
[source](https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b).
- [Python](https://www.python.org/) for HTTP/REST API,
The best guesses to which framework are [Flask].
- [Go](https://golang.org/) for the embed servers and one element of their logging(see [Punt])
- [Rust](https://www.rust-lang.org/) for many parts of the Discord Store, most commonly:
- [Game SDK](https://discordapp.com/developers/docs/game-sdk/sdk-starter-guide),
to make Discord integrations for any game.
- [Dispatch](https://discordapp.com/developers/docs/dispatch/dispatch-and-you),
which is a tool to send assets to Discord's servers.
## Distribution:
- [Loqui] for node communication.
## Logging:
- [Punt] in favour of [Logstash](https://github.com/elastic/logstash) for logging.
- [Elasticsearch](https://github.com/elastic/elasticsearch) that powers the search feature for users and powers logging.
- Sources about logging: Punt and [this issue](https://github.com/elastic/elasticsearch/issues/20354)
- [Source about message search](https://blog.discordapp.com/how-discord-indexes-billions-of-messages-e3d5e9be866f)
## External services
- [Google's Cloud Platform](https://cloud.google.com/) for their infrastructure,
[source](https://status.discordapp.com/incidents/rhvp2tn7g0zc)
- [Cloudflare](https://www.cloudflare.com/) as a proxy to almost all services,
voice servers need to be direct connections, so they don't pass through CF.
## Many other sources
- [Description of a full outage where they had to reboot everything](https://status.discordapp.com/incidents/dj3l6lw926kl)
- [`sessions` and `presence` clusters get rebooted due to a host error in a `guild` node](https://status.discordapp.com/incidents/ywdwttd6b0hg)
- [Repeating message sends due to errors in the `push` cluster](https://status.discordapp.com/incidents/93kyyctg0wf3)
- ["furiously" spinning an nginx cluster due to an error in GCP's load balancer](https://status.discordapp.com/incidents/rhvp2tn7g0zc)