Update discord-markdown

Interpret channel URLs the same as a channel #mention

docs/api.md

@@ -0,0 +1,52 @@
+# API
+
+There is a web API for getting information about things that are bridged with Out Of Your Element.
+
+The base URL is the URL of the particular OOYE instance, for example, https://bridge.cadence.moe.
+
+No authentication is required.
+
+I'm happy to add more endpoints, just ask for them.
+
+## Endpoint: GET /api/message
+
+|Query parameter|Type|Description|
+|---------------|----|-----------|
+|`message_id`|regexp `/^[0-9]+$/`|Discord message ID to look up information for|
+
+Response:
+
+```typescript
+{
+  source: "matrix" | "discord"                // Which platform the message originated on
+  matrix_author?: {                             // Only for Matrix messages; should be up-to-date rather than historical data
+    displayname: string,                        // Matrix user's current display name
+    avatar_url: string | null,                  // Absolute HTTP(S) URL to download the Matrix user's current avatar
+    mxid: string                                // Matrix user ID, can never change
+  },
+  events: [                                   // Data about each individual event
+    {
+      metadata: {                               // Data from OOYE's database about how bridging was performed
+        sender: string,                           // Same as matrix user ID
+        event_id: string,                         // Unique ID of the event on Matrix, can never change
+        event_type: "m.room.message" | string,    // Event type
+        event_subtype: "m.text" | string | null,  // For m.room.message events, this is the msgtype property
+        part: 0 | 1,                              // For multi-event messages, 0 if this is the first part
+        reaction_part: 0 | 1,                     // For multi-event messages, 0 if this is the last part
+        room_id: string,                          // Room ID that the event was sent in, linked to the Discord channel
+        source: number
+      },
+      raw: {                                    // Raw historical event data from the Matrix API. Contains at least these properties:
+        content: any,                             // The only non-metadata property, entirely client-generated
+        type: string,
+        room_id: string,
+        sender: string,
+        origin_server_ts: number,
+        unsigned?: any,
+        event_id: string,
+        user_id: string
+      }
+    }
+  ]
+}
+```

docs/developer-orientation.md

@@ -0,0 +1,129 @@
+# Development setup
+
+* Install development dependencies with `npm install --save-dev` so you can run the tests.
+* Most files you change, such as actions, converters, and web, will automatically be reloaded.
+* If developing on a different computer to the one running the homeserver, use SSH port forwarding so that Synapse can connect on its `localhost:6693` to reach the running bridge on your computer. Example: `ssh -T -v -R 6693:localhost:6693 me@matrix.cadence.moe`
+* I recommend developing in Visual Studio Code so that the JSDoc x TypeScript annotation comments just work automatically. I don't know which other editors or language servers support annotations and type inference.
+
+# Efficiency details
+
+Using WeatherStack as a thin layer between the bridge application and the Discord API lets us control exactly what data is cached in memory. Only necessary information is cached. For example, member data, user data, message content, and past edits are never stored in memory. This keeps the memory usage low and also prevents it ballooning in size over the bridge's runtime.
+
+The bridge uses a small SQLite database to store relationships like which Discord messages correspond to which Matrix messages. This is so the bridge knows what to edit when some message is edited on Discord. Using `without rowid` on the database tables stores the index and the data in the same B-tree. Since Matrix and Discord's internal IDs are quite long, this vastly reduces storage space because those IDs do not have to be stored twice separately. Some event IDs and URLs are actually stored as xxhash integers to reduce storage requirements even more. On my personal instance of OOYE, every 300,000 messages (representing a year of conversations) requires 47.3 MB of storage space in the SQLite database.
+
+Only necessary data and columns are queried from the database. We only contact the homeserver API if the database doesn't contain what we need.
+
+File uploads (like avatars from bridged members) are checked locally and deduplicated. Only brand new files are uploaded to the homeserver. This saves loads of space in the homeserver's media repo, especially for Synapse.
+
+Switching to [WAL mode](https://www.sqlite.org/wal.html) could improve your database access speed even more. Run `node scripts/wal.js` if you want to switch to WAL mode. (This will also enable `synchronous = NORMAL`.)
+
+
+# Repository structure
+
+    .
+    * Runtime configuration, like tokens and user info:
+    ├── registration.yaml
+    * You are here! :)
+    ├── readme.md
+    * The bridge's SQLite database is stored here:
+    ├── ooye.db*
+    * Source code
+    └── src
+        * Database schema:
+        ├── db
+        │   ├── orm.js, orm-defs.d.ts
+        │   * Migrations change the database schema when you update to a newer version of OOYE:
+        │   ├── migrate.js
+        │   └── migrations
+        │       └── *.sql, *.js
+        * Discord-to-Matrix bridging:
+        ├── d2m
+        │   * Execute actions through the whole flow, like sending a Discord message to Matrix:
+        │   ├── actions
+        │   │   └── *.js
+        │   * Convert data from one form to another without depending on bridge state. Called by actions:
+        │   ├── converters
+        │   │   └── *.js
+        │   * Making Discord work:
+        │   ├── discord-*.js
+        │   * Listening to events from Discord and dispatching them to the correct `action`:
+        │   └── event-dispatcher.js
+        * Discord bot commands and menus:
+        ├── discord
+        │   ├── interactions
+        │   │   └── *.js
+        │   └── discord-command-handler.js
+        * Matrix-to-Discord bridging:
+        ├── m2d
+        │   * Execute actions through the whole flow, like sending a Matrix message to Discord:
+        │   ├── actions
+        │   │   └── *.js
+        │   * Convert data from one form to another without depending on bridge state. Called by actions:
+        │   ├── converters
+        │   │   └── *.js
+        │   * Listening to events from Matrix and dispatching them to the correct `action`:
+        │   └── event-dispatcher.js
+        * We aren't using the matrix-js-sdk, so here are all the functions for the Matrix C-S and Appservice APIs:
+        ├── matrix
+        │   └── *.js
+        * Various files you can run once if you need them.
+        └── scripts
+            * First time running a new bridge? Run this file to set up prerequisites on the Matrix server:
+            ├── setup.js
+            * Hopefully you won't need the rest of these. Code quality varies wildly.
+            └── *.js
+
+# Read next
+
+If you haven't set up Out Of Your Element yet, you might find [Simplified homeserver setup](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/simplified-homeserver-setup.md) helpful.
+
+If you don't know what the Matrix event JSON generally looks like, turn on developer tools in your client (Element has pretty good ones). Right click a couple of messages and see what they look like on the inside.
+
+I recommend first reading [How to add a new event type](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/how-to-add-a-new-event-type.md) as this will fill you in on key information in how the codebase is organised, which data structures are important, and what level of abstraction we're working on.
+
+If you haven't seen the [Discord API documentation](https://discord.com/developers/docs/) before, have a quick look at one of the pages on there. Same with the [Matrix Client-Server APIs](https://spec.matrix.org/latest/client-server-api/). You don't need to know these inside out, they're primarily references, not stories. But it is useful to have an idea of what a couple of the API endpoints look like, the kind of data they tend to accept, and the kind of data they tend to return.
+
+Then you might like to peruse the other files in the docs folder. Most of these were written stream-of-thought style as I try to work through a problem and find the best way to implement it. You might enjoy getting inside my head and seeing me invent and evaluate ways to solve the problem.
+
+Whether you read those or not, I'm more than happy to help you 1-on-1 with coding your dream feature. Join the chatroom [#out-of-your-element:cadence.moe](https://matrix.to/#/#out-of-your-element:cadence.moe) or PM me [@cadence:cadence.moe](https://matrix.to/#/@cadence:cadence.moe) and ask away.
+
+# Dependency justification
+
+Total transitive production dependencies: 137
+
+### <font size="+2">🦕</font>
+
+* (31) better-sqlite3: SQLite is the best database, and this is the best library for it.
+* (27) @cloudrac3r/pug: Language for dynamic web pages. This is my fork. (I released code that hadn't made it to npm, and removed the heavy pug-filters feature.)
+* (16) stream-mime-type@1: This seems like the best option. Version 1 is used because version 2 is ESM-only.
+* (9) h3: Web server. OOYE needs this for the appservice listener, authmedia proxy, self-service, and more.
+* (11) sharp: Image resizing and compositing. OOYE needs this for the emoji sprite sheets.
+
+### <font size="-1">🪱</font>
+
+* (0) @chriscdn/promise-semaphore: It does what I want.
+* (1) @cloudrac3r/discord-markdown: This is my fork.
+* (0) @cloudrac3r/giframe: This is my fork.
+* (1) @cloudrac3r/html-template-tag: This is my fork.
+* (0) @cloudrac3r/in-your-element: This is my Matrix Appservice API library. It depends on h3 and zod, which are already pulled in by OOYE.
+* (0) @cloudrac3r/mixin-deep: This is my fork. (It fixes a bug in regular mixin-deep.)
+* (0) @cloudrac3r/pngjs: Lottie stickers are converted to bitmaps with the vendored Rlottie WASM build, then the bitmaps are converted to PNG with pngjs.
+* (0) @cloudrac3r/turndown: This HTML-to-Markdown converter looked the most suitable. I forked it to change the escaping logic to match the way Discord works.
+* (3) @stackoverflow/stacks: Stack Overflow design language and icons.
+* (0) ansi-colors: Helps with interactive prompting for the initial setup, and it's already pulled in by enquirer.
+* (1) chunk-text: It does what I want.
+* (0) cloudstorm: Discord gateway library with bring-your-own-caching that I trust.
+* (0) discord-api-types: Bitfields needed at runtime and types needed for development.
+* (0) domino: DOM implementation that's already pulled in by turndown.
+* (1) enquirer: Interactive prompting for the initial setup rather than forcing users to edit YAML non-interactively.
+* (0) entities: Looks fine. No dependencies.
+* (0) get-relative-path: Looks fine. No dependencies.
+* (1) heatsync: Module hot-reloader that I trust.
+* (1) js-yaml: Will be removed in the future after registration.yaml is converted to JSON.
+* (0) lru-cache: For holding unused nonce in memory and letting them be overwritten later if never used.
+* (0) prettier-bytes: It does what I want and has no dependencies.
+* (0) snowtransfer: Discord API library with bring-your-own-caching that I trust.
+* (0) try-to-catch: Not strictly necessary, but it's already pulled in by supertape, so I may as well.
+* (0) uqr: QR code SVG generator. Used on the website to scan in an invite link.
+* (0) xxhash-wasm: Used where cryptographically secure hashing is not required.
+* (0) zod: Input validation for the web server. It's popular and easy to use.

docs/foreign-keys.md

@@ -0,0 +1,98 @@
+# Foreign keys in the Out Of Your Element database
+
+Historically, Out Of Your Element did not use foreign keys in the database, but since I found a need for them, I have decided to add them. Referential integrity is probably valuable as well.
+
+The need is that unlinking a channel and room using the web interface should clear up all related entries from `message_channel`, `event_message`, `reaction`, etc. Without foreign keys, this requires multiple DELETEs with tricky queries. With foreign keys and ON DELETE CASCADE, this just works.
+
+## Quirks
+
+* **REPLACE INTO** internally causes a DELETE followed by an INSERT, and the DELETE part **will trigger any ON DELETE CASCADE** foreign key conditions on the table, even when the primary key being replaced is the same.
+	* ```sql
+		CREATE TABLE discord_channel (channel_id TEXT NOT NULL, name TEXT NOT NULL, PRIMARY KEY (channel_id));
+		CREATE TABLE discord_message (message_id TEXT NOT NULL, channel_id TEXT NOT NULL, PRIMARY KEY (message_id),
+		  FOREIGN KEY (channel_id) REFERENCES discord_channel (channel_id) ON DELETE CASCADE);
+		INSERT INTO discord_channel (channel_id, name) VALUES ("c_1", "place");
+		INSERT INTO discord_message (message_id, channel_id) VALUES ("m_2", "c_1"); -- i love my message
+		REPLACE INTO discord_channel (channel_id, name) VALUES ("c_1", "new place"); -- replace into time
+		-- i love my message
+		SELECT * FROM discord_message; -- where is my message
+		```
+* In SQLite, `pragma foreign_keys = on` must be set **for each connection** after it's established. I've added this at the start of `migrate.js`, which is called by all database connections.
+  * Pragma? Pragma keys
+* Whenever a child row is inserted, SQLite will look up a row from the parent table to ensure referential integrity. This means **the parent table should be sufficiently keyed or indexed on columns referenced by foreign keys**, or SQLite won't let you do it, with a cryptic error message later on during DML. Due to normal forms, foreign keys naturally tend to reference the parent table's primary key, which is indexed, so that's okay. But still keep this in mind, since many of OOYE's tables effectively have two primary keys, for the Discord and Matrix IDs. A composite primary key doesn't count, even when it's the first column. A unique index counts.
+
+## Where keys
+
+Here are some tables that could potentially have foreign keys added between them, and my thought process of whether foreign keys would be a good idea:
+
+* `guild_active` <--(PK guild_id FK)-- `channel_room` ✅
+	* Could be good for referential integrity.
+	* Linking to guild_space would be pretty scary in case the guild was being relinked to a different space - since rooms aren't tied to a space, this wouldn't actually disturb anything. So I pick guild_active instead.
+* `channel_room` <--(PK channel_id FK)-- `message_channel` ✅
+	* Seems useful as we want message records to be deleted when a channel is unlinked.
+* `message_channel` <--(PK message_id PK)-- `event_message` ✅
+	* Seems useful as we want event information to be deleted when a channel is unlinked.
+* `guild_active` <--(PK guild_id PK)-- `guild_space` ✅
+	* All bridged guilds should have a corresponding guild_active entry, so referential integrity would be useful here to make sure we haven't got any weird states.
+* `channel_room` <--(**C** room_id PK)-- `member_cache` ✅
+	* Seems useful as we want to clear the member cache when a channel is unlinked.
+	* There is no index on `channel_room.room_id` right now. It would be good to create this index. Will just make it UNIQUE in the table definition.
+* `message_channel` <--(PK message_id FK)-- `reaction` ✅
+	* Seems useful as we want to clear the reactions cache when a channel is unlinked.
+* `sim` <--(**C** mxid FK)-- `sim_member`
+	* OOYE inner joins on this.
+	* Sims are never deleted so if this was added it would only be used for enforcing referential integrity.
+	* The storage cost of the additional index on `sim` would not be worth the benefits.
+* `channel_room` <--(**C** room_id PK)-- `sim_member`
+	* If a room is being permanently unlinked, it may be useful to see a populated member list. If it's about to be relinked to another channel, we want to keep the sims in the room for more speed and to avoid spamming state events into the timeline.
+	* Either way, the sims could remain in the room even after it's been unlinked. So no referential integrity is desirable here.
+* `sim` <--(PK user_id PK)-- `sim_proxy`
+	* OOYE left joins on this. In normal operation, this relationship might not exist.
+* `channel_room` <--(PK channel_id PK)-- `webhook` ✅
+	* Seems useful. Webhooks should be deleted from Discord just before the channel is unlinked. That should be mirrored in the database too.
+
+## Occurrences of REPLACE INTO/DELETE FROM
+
+* `edit-message.js` — `REPLACE INTO message_channel`
+	* Scary! Changed to INSERT OR IGNORE
+* `send-message.js` — `REPLACE INTO message_channel`
+	* Changed to INSERT OR IGNORE
+* `add-reaction.js` — `REPLACE INTO reaction`
+* `channel-webhook.js` — `REPLACE INTO webhook`
+* `send-event.js` — `REPLACE INTO message_channel`
+	* Seems incorrect? Maybe?? Originally added in fcbb045. Changed to INSERT
+* `event-to-message.js` — `REPLACE INTO member_cache`
+* `oauth.js` — `REPLACE INTO guild_active`
+	* Very scary!! Changed to INSERT .. ON CONFLICT DO UPDATE
+* `create-room.js` — `DELETE FROM channel_room`
+	* Please cascade
+* `delete-message.js`
+	* Removed redundant DELETEs
+* `edit-message.js` — `DELETE FROM event_message`
+* `register-pk-user.js` — `DELETE FROM sim`
+	* It's a failsafe during creation
+* `register-user.js` — `DELETE FROM sim`
+	* It's a failsafe during creation
+* `remove-reaction.js` — `DELETE FROM reaction`
+* `event-dispatcher.js` — `DELETE FROM member_cache`
+* `redact.js` — `DELETE FROM event_message`
+	* Removed this redundant DELETE
+* `send-event.js` — `DELETE FROM event_message`
+	* Removed this redundant DELETE
+
+## How keys
+
+SQLite does not have a complete ALTER TABLE command, so I have to DROP and CREATE. According to [the docs](https://www.sqlite.org/lang_altertable.html), the correct strategy is:
+
+1. (Not applicable) *If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF.*
+2. Start a transaction.
+3. (Not applicable) *Remember the format of all indexes, triggers, and views associated with table X. This information will be needed in step 8 below. One way to do this is to run a query like the following: SELECT type, sql FROM sqlite_schema WHERE tbl_name='X'.*
+4. Use CREATE TABLE to construct a new table "new_X" that is in the desired revised format of table X. Make sure that the name "new_X" does not collide with any existing table name, of course.
+5. Transfer content from X into new_X using a statement like: INSERT INTO new_X SELECT ... FROM X.
+6. Drop the old table X: DROP TABLE X.
+7. Change the name of new_X to X using: ALTER TABLE new_X RENAME TO X.
+8. (Not applicable) *Use CREATE INDEX, CREATE TRIGGER, and CREATE VIEW to reconstruct indexes, triggers, and views associated with table X. Perhaps use the old format of the triggers, indexes, and views saved from step 3 above as a guide, making changes as appropriate for the alteration.*
+9. (Not applicable) *If any views refer to table X in a way that is affected by the schema change, then drop those views using DROP VIEW and recreate them with whatever changes are necessary to accommodate the schema change using CREATE VIEW.*
+10. If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.
+11. Commit the transaction started in step 2.
+12. (Not applicable) *If foreign keys constraints were originally enabled, reenable them now.*

docs/get-started.md

@@ -0,0 +1,108 @@
+# Setup
+
+If you get stuck, you're welcome to message [#out-of-your-element:cadence.moe](https://matrix.to/#/#out-of-your-element:cadence.moe) or [@cadence:cadence.moe](https://matrix.to/#/@cadence:cadence.moe) to ask for help setting up OOYE!
+
+You'll need:
+
+* Administrative access to a homeserver
+* Discord bot
+* Domain name for the bridge's website - [more info](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/why-does-the-bridge-have-a-website.md)
+* Reverse proxy for that domain - an interactive process will help you set this up in step 5!
+
+Follow these steps:
+
+1. [Get Node.js version 22 or later](https://nodejs.org/en/download/prebuilt-installer). If you're on Linux, you may prefer to install through system's package manager, though Debian and Ubuntu have hopelessly out of date packages.
+
+1. Switch to a normal user account. (i.e. do not run any of the following commands as root or sudo.)
+
+1. Clone this repo and checkout a specific tag. (Development happens on main. Stable versions are tagged.)
+	* The latest release tag is ![](https://img.shields.io/gitea/v/release/cadence/out-of-your-element?gitea_url=https%3A%2F%2Fgitdab.com&style=flat-square&label=%20&color=black).
+
+1. Install dependencies: `npm install`
+
+1. Run `npm run setup` to check your setup and set the bot's initial state. You only need to run this once ever. This command will guide you precisely through the following steps:
+
+	* First, you'll be asked for information like your homeserver URL.
+
+	* Then you'll be prompted to set up a reverse proxy pointing from your domain to the bridge's web server. Sample configurations can be found at the end of this guide. It will check that the reverse proxy works before you continue.
+
+	* Then you'll need to provide information about your Discord bot, and you'll be asked to change some of its settings.
+
+	* Finally, a registration.yaml file will be generated, which you need to give to your homeserver. You'll be told how to do this. It will check that it's done properly.
+
+1. Start the bridge: `npm run start`
+
+## Update
+
+New versions are announced in [#updates](https://matrix.to/#/#ooye-updates:cadence.moe) and listed on [releases](https://gitdab.com/cadence/out-of-your-element/releases). Here's how to update:
+
+1. Fetch the repo and checkout the latest release tag.
+
+1. Install dependencies: `npm install`
+
+1. Restart the bridge: Stop the currently running process, and then start the new one with `npm run start`
+
+# Get Started
+
+Visit the website on the domain name you set up, and click the button to add the bot to your Discord server.
+
+* If you click the Easy Mode button, it will automatically create a Matrix room corresponding to each Discord channel. This happens next time a message is sent on Discord (so that your Matrix-side isn't immediately cluttered with lots of inactive rooms).
+
+* If you click the Self Service button, it won't create anything for you. You'll have to provide your own Matrix space and rooms. After you click, you'll be prompted through the process. Use this if you're migrating from another bridge!
+
+After that, to get into the rooms on your Matrix account, use the invite form on the website, or the `/invite [your mxid here]` command on Discord.
+
+I hope you enjoy Out Of Your Element!
+
+----
+<br><br><br><br><br>
+
+# Appendix
+
+## Example reverse proxy for nginx, dedicated domain name
+
+Replace `bridge.cadence.moe` with the hostname you're using.
+
+```nix
+server {
+	listen 80;
+	listen [::]:80;
+	server_name bridge.cadence.moe;
+
+	return 301 https://bridge.cadence.moe$request_uri;
+}
+
+server {
+	listen 443 ssl http2;
+	listen [::]:443 ssl http2;
+	server_name bridge.cadence.moe;
+
+	# ssl parameters here...
+	client_max_body_size 5M;
+
+	location / {
+		add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
+		proxy_pass http://127.0.0.1:6693;
+	}
+}
+```
+
+## Example reverse proxy for nginx, sharing a domain name
+
+Same as above, but change the following:
+
+- `location / {` -> `location /ooye/ {` (any sub-path you want; you MUST use a trailing slash or it won't work)
+- `proxy_pass http://127.0.0.1:6693;` -> `proxy_pass http://127.0.0.1:6693/;` (you MUST use a trailing slash on this too or it won't work)
+
+## Example reverse proxy for Caddy, dedicated domain name
+
+```nix
+bridge.cadence.moe {
+	log {
+		output file /var/log/caddy/access.log
+		format console
+	}
+	encode gzip
+	reverse_proxy 127.0.0.1:6693
+}
+```

docs/self-service-room-creation-rules.md

@@ -69,3 +69,9 @@ So here's all the technical changes needed to support self-service in v3:
 - When bot is added through "self-service" web button, REPLACE INTO state 0.
 - Event dispatcher will only ensureRoom if the guild_active state is 1.
 - createRoom will only create other dependencies if the guild is autocreate.
+
+## Enough with your theory. How do rooms actually get bridged now?
+
+After clicking the easy mode button on web and adding the bot to a server, it will create new Matrix rooms on-demand when any invite features are used (web or command) OR just when any message is sent on Discord.
+
+Alternatively, pressing the self-service mode button and adding the bot to a server will prompt the web user to link it with a space. After doing so, they'll be on the standard guild management page where they can invite to the space and manually link rooms. Nothing will be autocreated.

docs/why-does-the-bridge-have-a-website.md

@@ -50,54 +50,6 @@ When you use `npm run setup` as part of OOYE's initial setup, it will guide you
 
 When OOYE is running, the web server runs on port 6693. (To use a different port or a UNIX socket, edit registration.yaml's `socket` setting and restart.)
 
-It doesn't have to have its own dedicated domain name, you can also use a sub-path on an existing domain, like the domain of your Matrix homeserver. See the end of this document for more information.
+It doesn't have to have its own dedicated domain name, you can also use a sub-path on an existing domain, like the domain of your Matrix homeserver. You are likely already using a reverse proxy to run your homeserver, so this should just be a configuration change.
 
-You are likely already using a reverse proxy for running your homeserver, so this should just be a configuration change.
-
-## Example configuration for nginx, dedicated domain name
-
-Replace `bridge.cadence.moe` with the hostname you're using.
-
-```nix
-server {
-	listen 80;
-	listen [::]:80;
-	server_name bridge.cadence.moe;
-
-	return 301 https://bridge.cadence.moe$request_uri;
-}
-
-server {
-	listen 443 ssl http2;
-	listen [::]:443 ssl http2;
-	server_name bridge.cadence.moe;
-
-	# ssl parameters here...
-	client_max_body_size 5M;
-
-	location / {
-		add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
-		proxy_pass http://127.0.0.1:6693;
-	}
-}
-```
-
-## Example configuration for nginx, sharing a domain name
-
-Same as above, but change the following:
-
-- `location / {` -> `location /ooye/ {` (any sub-path you want; you MUST use a trailing slash or it won't work)
-- `proxy_pass http://127.0.0.1:6693;` -> `proxy_pass http://127.0.0.1:6693/;` (you MUST use a trailing slash on this too or it won't work)
-
-## Example configuration for Caddy, dedicated domain name
-
-```nix
-bridge.cadence.moe {
-	log {
-		output file /var/log/caddy/access.log
-		format console
-	}
-	encode gzip
-	reverse_proxy 127.0.0.1:6693
-}
-```
+[See here for sample configurations!](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/get-started.md#appendix)

package.json

@@ -1,6 +1,6 @@
 {
   "name": "out-of-your-element",
-  "version": "1.1.1",
+  "version": "3.1.0",
   "description": "A bridge between Matrix and Discord",
   "main": "index.js",
   "repository": {
@@ -18,55 +18,52 @@
     "node": ">=20"
   },
   "dependencies": {
-    "@chriscdn/promise-semaphore": "^2.0.1",
-    "@cloudrac3r/discord-markdown": "^2.6.3",
+    "@chriscdn/promise-semaphore": "^3.0.1",
+    "@cloudrac3r/discord-markdown": "^2.6.6",
     "@cloudrac3r/giframe": "^0.4.3",
     "@cloudrac3r/html-template-tag": "^5.0.1",
-    "@cloudrac3r/in-your-element": "^1.0.0",
-    "@cloudrac3r/mixin-deep": "^3.0.0",
+    "@cloudrac3r/in-your-element": "^1.1.1",
+    "@cloudrac3r/mixin-deep": "^3.0.1",
     "@cloudrac3r/pngjs": "^7.0.3",
     "@cloudrac3r/pug": "^4.0.4",
     "@cloudrac3r/turndown": "^7.1.4",
-    "@stackoverflow/stacks": "^2.5.7",
+    "@stackoverflow/stacks": "^2.5.4",
     "@stackoverflow/stacks-icons": "^6.0.2",
     "ansi-colors": "^4.1.3",
-    "better-sqlite3": "^11.1.2",
+    "better-sqlite3": "^12.2.0",
     "chunk-text": "^2.0.1",
-    "cloudstorm": "^0.10.10",
+    "cloudstorm": "^0.14.0",
+    "discord-api-types": "^0.38.19",
     "domino": "^2.1.6",
     "enquirer": "^2.4.1",
     "entities": "^5.0.0",
     "get-relative-path": "^1.0.2",
-    "get-stream": "^6.0.1",
-    "h3": "^1.12.0",
-    "heatsync": "^2.5.5",
-    "lru-cache": "^10.4.3",
-    "minimist": "^1.2.8",
-    "node-fetch": "^2.6.7",
+    "h3": "^1.15.1",
+    "heatsync": "^2.7.2",
+    "htmx.org": "^2.0.4",
+    "lru-cache": "^11.0.2",
     "prettier-bytes": "^1.0.4",
     "sharp": "^0.33.4",
-    "snowtransfer": "^0.10.5",
+    "snowtransfer": "^0.15.0",
     "stream-mime-type": "^1.0.2",
     "try-to-catch": "^3.0.1",
     "uqr": "^0.1.2",
     "xxhash-wasm": "^1.0.2",
-    "zod": "^3.23.8"
+    "zod": "^4.0.17"
   },
   "devDependencies": {
     "@cloudrac3r/tap-dot": "^2.0.3",
-    "@types/node": "^18.16.0",
-    "@types/node-fetch": "^2.6.3",
+    "@types/node": "^22.17.1",
     "c8": "^10.1.2",
     "cross-env": "^7.0.3",
-    "discord-api-types": "^0.37.60",
-    "supertape": "^10.4.0"
+    "supertape": "^11.3.0"
   },
   "scripts": {
-    "start": "node start.js",
-    "setup": "node scripts/setup.js",
+    "start": "node --enable-source-maps start.js",
+    "setup": "node --enable-source-maps scripts/setup.js",
     "addbot": "node addbot.js",
     "test": "cross-env FORCE_COLOR=true supertape --no-check-assertions-count --format tap --no-worker test/test.js | tap-dot",
     "test-slow": "cross-env FORCE_COLOR=true supertape --no-check-assertions-count --format tap --no-worker test/test.js -- --slow | tap-dot",
-    "cover": "c8 -o test/coverage --skip-full -x db/migrations -x src/matrix/file.js -x src/matrix/api.js -x src/matrix/mreq.js -x src/d2m/converters/rlottie-wasm.js -r html -r text supertape --no-check-assertions-count --format fail --no-worker test/test.js -- --slow"
+    "cover": "c8 -o test/coverage --skip-full -x db/migrations -x src/matrix/file.js -x src/matrix/api.js -x src/d2m/converters/rlottie-wasm.js -r html -r text supertape --no-check-assertions-count --format fail --no-worker test/test.js -- --slow"
   }
 }

readme.md

@@ -6,15 +6,13 @@ Modern Matrix-to-Discord appservice bridge, created by [@cadence:cadence.moe](ht
 
 [![Releases](https://img.shields.io/gitea/v/release/cadence/out-of-your-element?gitea_url=https%3A%2F%2Fgitdab.com&style=plastic&color=green)](https://gitdab.com/cadence/out-of-your-element/releases) [![Discuss on Matrix](https://img.shields.io/badge/discuss-%23out--of--your--element-white?style=plastic)](https://matrix.to/#/#out-of-your-element:cadence.moe)
 
-## Docs
-
-This readme has the most important info. The rest is [in the docs folder.](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs)
+![](https://cadence.moe/i/f42a3f)
 
 ## Why a new bridge?
 
 * Modern: Supports new Discord features like replies, threads and stickers, and new Matrix features like edits, spaces and space membership.
-* Efficient: Special attention has been given to memory usage, database indexes, disk footprint, runtime algorithms, and queries to the homeserver.
-* Reliable: Any errors on either side are notified on Matrix and can be retried.
+* Efficient: Special attention has been given to memory usage, database indexes, disk footprint, runtime algorithms, and queries to the homeserver. [Efficiency details.](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/developer-orientation.md)
+* Reliable: Any errors on either side are notified on Matrix and can be retried. Messages sent during bridge downtime will still be bridged after it comes back up.
 * Tested: A test suite and code coverage make sure all the logic and special cases work.
 * Simple development: No build step (it's JavaScript, not TypeScript), minimal/lightweight dependencies, and abstraction only where necessary so that less background knowledge is required. No need to learn about Intents or library functions.
 * No locking algorithm: Other bridges use a locking algorithm which is a source of frequent bugs. This bridge avoids the need for one.
@@ -22,28 +20,17 @@ This readme has the most important info. The rest is [in the docs folder.](https
 
 ## What works?
 
-Most features you'd expect in both directions, plus a little extra spice:
+Most features you'd expect in both directions: messages, edits, deletions, formatting (including spoilers), reactions, custom emojis, custom emoji reactions, mentions, channel mentions, replies, threads, stickers (all formats: PNG, APNG, GIF, Lottie), attachments, spoiler attachments (compatible with most clients), embeds, URL previews, presence, discord.com hyperlinks, and more.
 
-* Messages
-* Edits
-* Deletions
-* Text formatting, including spoilers
-* Reactions
-* Mentions
-* Replies
-* Threads
-* Stickers (all formats: PNG, APNG, GIF, and Lottie)
-* Attachments
-* Spoiler attachments
-* Embeds
-* Guild-Space details syncing
-* Channel-Room details syncing
-* Custom emoji list syncing
-* Custom emojis in messages
-* Custom room names/avatars can be applied on Matrix-side
-* PluralKit members have persistent user accounts
-* Larger files from Discord are linked instead of reuploaded to Matrix (links don't expire)
+Metadata is also synced: people's names, avatars, usernames; channel names, icons, topics; spaces containing rooms; custom emoji lists. Syncing Matrix rooms, room icons, and topics is optional: you can keep them different from the Discord ones if you prefer.
+
+I've also added some interesting features that I haven't seen in any other bridge:
+
+* Members using the PluralKit bot each get their own persistent accounts
+* Replies from PluralKit members are restyled into native Matrix replies
 * Simulated user accounts are named @the_persons_username rather than @112233445566778899
+* Matrix custom emojis from private rooms are still visible on Discord as a sprite sheet
+* To save space, larger files from Discord are linked instead of reuploaded to Matrix (links don't expire)
 
 For more information about features, [see the user guide.](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/user-guide.md)
 
@@ -52,152 +39,6 @@ For more information about features, [see the user guide.](https://gitdab.com/ca
 * This bridge is not designed for puppetting.
 * Direct Messaging is not supported until I figure out a good way of doing it.
 
-## Efficiency details
+## Get started!
 
-Using WeatherStack as a thin layer between the bridge application and the Discord API lets us control exactly what data is cached in memory. Only necessary information is cached. For example, member data, user data, message content, and past edits are never stored in memory. This keeps the memory usage low and also prevents it ballooning in size over the bridge's runtime.
-
-The bridge uses a small SQLite database to store relationships like which Discord messages correspond to which Matrix messages. This is so the bridge knows what to edit when some message is edited on Discord. Using `without rowid` on the database tables stores the index and the data in the same B-tree. Since Matrix and Discord's internal IDs are quite long, this vastly reduces storage space because those IDs do not have to be stored twice separately. Some event IDs and URLs are actually stored as xxhash integers to reduce storage requirements even more. On my personal instance of OOYE, every 300,000 messages (representing a year of conversations) requires 47.3 MB of storage space in the SQLite database.
-
-Only necessary data and columns are queried from the database. We only contact the homeserver API if the database doesn't contain what we need.
-
-File uploads (like avatars from bridged members) are checked locally and deduplicated. Only brand new files are uploaded to the homeserver. This saves loads of space in the homeserver's media repo, especially for Synapse.
-
-Switching to [WAL mode](https://www.sqlite.org/wal.html) could improve your database access speed even more. Run `node scripts/wal.js` if you want to switch to WAL mode. This will also enable `synchronous = NORMAL`.
-
-# Setup
-
-If you get stuck, you're welcome to message [#out-of-your-element:cadence.moe](https://matrix.to/#/#out-of-your-element:cadence.moe) or [@cadence:cadence.moe](https://matrix.to/#/@cadence:cadence.moe) to ask for help setting up OOYE!
-
-You'll need:
-
-* Administrative access to a homeserver
-* Discord bot
-* Domain name for the bridge's website ([more info](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/why-does-the-bridge-have-a-website.md)
-
-Follow these steps:
-
-1. [Get Node.js version 20 or later](https://nodejs.org/en/download/prebuilt-installer)
-
-1. Clone this repo and checkout a specific tag. (Development happens on main. Stable versions are tagged.)
-	* The latest release tag is ![](https://img.shields.io/gitea/v/release/cadence/out-of-your-element?gitea_url=https%3A%2F%2Fgitdab.com&style=flat-square&label=%20&color=black).
-
-1. Install dependencies: `npm install`
-
-1. Run `npm run setup` to check your setup and set the bot's initial state. It will prompt you for information. You only need to run this once ever.
-
-1. Start the bridge: `npm run start`
-
-1. Add the bot to a server - use any *one* of the following commands for an invite link:
-	* (in the REPL) `addbot`
-	* $ `node addbot.js`
-	* $ `npm run addbot`
-	* $ `./addbot.sh`
-
-Now any message on Discord will create the corresponding rooms on Matrix-side. After the rooms have been created, Matrix and Discord users can chat back and forth.
-
-To get into the rooms on your Matrix account, use the `/invite [your mxid here]` command on Discord.
-
-# Development setup
-
-* Install development dependencies with `npm install --save-dev` so you can run the tests.
-* Most files you change, such as actions, converters, and web, will automatically be reloaded.
-* If developing on a different computer to the one running the homeserver, use SSH port forwarding so that Synapse can connect on its `localhost:6693` to reach the running bridge on your computer. Example: `ssh -T -v -R 6693:localhost:6693 me@matrix.cadence.moe`
-* I recommend developing in Visual Studio Code so that the JSDoc x TypeScript annotation comments work. I don't know which other editors or language servers support annotations and type inference.
-
-## Repository structure
-
-    .
-    * Runtime configuration, like tokens and user info:
-    ├── registration.yaml
-    * You are here! :)
-    ├── readme.md
-	 * The bridge's SQLite database is stored here:
-	 ├── ooye.db*
-    * Source code
-    └── src
-        * Database schema:
-        ├── db
-		  │   ├── orm.js, orm-defs.d.ts
-		  │   * Migrations change the database schema when you update to a newer version of OOYE:
-        │   ├── migrate.js
-        │   └── migrations
-        │       └── *.sql, *.js
-        * Discord-to-Matrix bridging:
-        ├── d2m
-        │   * Execute actions through the whole flow, like sending a Discord message to Matrix:
-        │   ├── actions
-        │   │   └── *.js
-        │   * Convert data from one form to another without depending on bridge state. Called by actions:
-        │   ├── converters
-        │   │   └── *.js
-        │   * Making Discord work:
-        │   ├── discord-*.js
-        │   * Listening to events from Discord and dispatching them to the correct `action`:
-        │   └── event-dispatcher.js
-        * Discord bot commands and menus:
-        ├── discord
-        │   ├── interactions
-        │   │   └── *.js
-        │   └── discord-command-handler.js
-        * Matrix-to-Discord bridging:
-        ├── m2d
-        │   * Execute actions through the whole flow, like sending a Matrix message to Discord:
-        │   ├── actions
-        │   │   └── *.js
-        │   * Convert data from one form to another without depending on bridge state. Called by actions:
-        │   ├── converters
-        │   │   └── *.js
-        │   * Listening to events from Matrix and dispatching them to the correct `action`:
-        │   └── event-dispatcher.js
-        * We aren't using the matrix-js-sdk, so here are all the functions for the Matrix C-S and Appservice APIs:
-        ├── matrix
-        │   └── *.js
-        * Various files you can run once if you need them.
-        └── scripts
-            * First time running a new bridge? Run this file to set up prerequisites on the Matrix server:
-            ├── setup.js
-            * Hopefully you won't need the rest of these. Code quality varies wildly.
-            └── *.js
-
-## Dependency justification
-
-Total transitive production dependencies: 148
-
-### <font size="+2">🦕</font>
-
-* (31) better-sqlite3: SQLite3 is the best database, and this is the best library for it.
-* (27) @cloudrac3r/pug: Language for dynamic web pages. This is my fork. (I released code that hadn't made it to npm, and removed the heavy pug-filters feature.)
-* (16) stream-mime-type@1: This seems like the best option. Version 1 is used because version 2 is ESM-only.
-* (14) h3: Web server. OOYE needs this for the appservice listener, authmedia proxy, and more. 14 transitive dependencies is on the low end for a web server.
-* (11) sharp: Image resizing and compositing. OOYE needs this for the emoji sprite sheets.
-
-### <font size="-1">🪱</font>
-
-* (0) @chriscdn/promise-semaphore: It does what I want.
-* (1) @cloudrac3r/discord-markdown: This is my fork.
-* (0) @cloudrac3r/giframe: This is my fork.
-* (1) @cloudrac3r/html-template-tag: This is my fork.
-* (0) @cloudrac3r/in-your-element: This is my Matrix Appservice API library. It depends on h3 and zod, which are already pulled in by OOYE.
-* (0) @cloudrac3r/mixin-deep: This is my fork. (It fixes a bug in regular mixin-deep.)
-* (0) @cloudrac3r/pngjs: Lottie stickers are converted to bitmaps with the vendored Rlottie WASM build, then the bitmaps are converted to PNG with pngjs.
-* (0) @cloudrac3r/turndown: This HTML-to-Markdown converter looked the most suitable. I forked it to change the escaping logic to match the way Discord works.
-* (3) @stackoverflow/stacks: Stack Overflow design language and icons.
-* (0) ansi-colors: Helps with interactive prompting for the initial setup, and it's already pulled in by enquirer.
-* (1) chunk-text: It does what I want.
-* (0) cloudstorm: Discord gateway library with bring-your-own-caching that I trust.
-* (0) domino: DOM implementation that's already pulled in by turndown.
-* (1) enquirer: Interactive prompting for the initial setup rather than forcing users to edit YAML non-interactively.
-* (0) entities: Looks fine. No dependencies.
-* (0) get-relative-path: Looks fine. No dependencies.
-* (0) get-stream: Only needed if content_length_workaround is true.
-* (1) heatsync: Module hot-reloader that I trust.
-* (1) js-yaml: Will be removed in the future after registration.yaml is converted to JSON.
-* (0) lru-cache: For holding unused nonce in memory and letting them be overwritten later if never used.
-* (0) minimist: It's already pulled in by better-sqlite3->prebuild-install.
-* (3) node-fetch@2: I like it and it does what I want. Version 2 is used because version 3 is ESM-only.
-* (0) prettier-bytes: It does what I want and has no dependencies.
-* (2) snowtransfer: Discord API library with bring-your-own-caching that I trust.
-* (0) try-to-catch: Not strictly necessary, but it's already pulled in by supertape, so I may as well.
-* (0) uqr: QR code SVG generator. Used on the website to scan in an invite link.
-* (0) xxhash-wasm: Used where cryptographically secure hashing is not required.
-* (0) zod: Input validation for the web server. It's popular and easy to use.
+[Read the installation instructions →](https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/get-started.md)

scripts/backfill.js

@@ -0,0 +1,79 @@
+#!/usr/bin/env node
+// @ts-check
+
+console.log("-=- This script is experimental. It WILL mess up the room history on Matrix. -=-")
+console.log()
+
+const {channel: channelID} = require("minimist")(process.argv.slice(2), {string: ["channel"]})
+if (!channelID) {
+	console.error("Usage: ./scripts/backfill.js --channel=<channel id here>")
+	process.exit(1)
+}
+
+const assert = require("assert/strict")
+const sqlite = require("better-sqlite3")
+const backfill = new sqlite("scripts/backfill.db")
+backfill.prepare("CREATE TABLE IF NOT EXISTS backfill (channel_id TEXT NOT NULL, message_id INTEGER NOT NULL, PRIMARY KEY (channel_id, message_id))").run()
+
+const HeatSync = require("heatsync")
+
+const {reg} = require("../src/matrix/read-registration")
+const passthrough = require("../src/passthrough")
+
+const sync = new HeatSync({watchFS: false})
+const db = new sqlite("ooye.db")
+Object.assign(passthrough, {sync, db})
+
+const DiscordClient = require("../src/d2m/discord-client")
+
+const discord = new DiscordClient(reg.ooye.discord_token, "half")
+passthrough.discord = discord
+
+const orm = sync.require("../src/db/orm")
+passthrough.from = orm.from
+passthrough.select = orm.select
+
+/** @type {import("../src/d2m/event-dispatcher")}*/
+const eventDispatcher = sync.require("../src/d2m/event-dispatcher")
+
+const roomID = passthrough.select("channel_room", "room_id", {channel_id: channelID}).pluck().get()
+if (!roomID) {
+	console.error("Please choose a channel that's already bridged.")
+	process.exit(1)
+}
+
+;(async () => {
+	await discord.cloud.connect()
+	console.log("Connected, waiting for data about requested channel...")
+
+	discord.cloud.on("event", event)
+})()
+
+const preparedInsert = backfill.prepare("INSERT INTO backfill (channel_id, message_id) VALUES (?, ?)")
+
+async function event(event) {
+	if (event.t !== "GUILD_CREATE") return
+	const channel = event.d.channels.find(c => c.id === channelID)
+	if (!channel) return
+	const guild_id = event.d.id
+
+	let last = backfill.prepare("SELECT cast(max(message_id) as TEXT) FROM backfill WHERE channel_id = ?").pluck().get(channelID) || "0"
+	console.log(`OK, processing messages for #${channel.name}, continuing from ${last}`)
+
+	while (last) {
+		const messages = await discord.snow.channel.getChannelMessages(channelID, {limit: 50, after: String(last)})
+		messages.reverse() // More recent messages come first -> More recent messages come last
+		for (const message of messages) {
+			const simulatedGatewayDispatchData = {
+				guild_id,
+				backfill: true,
+				...message
+			}
+			await eventDispatcher.onMessageCreate(discord, simulatedGatewayDispatchData)
+			preparedInsert.run(channelID, message.id)
+		}
+		last = messages.at(-1)?.id
+	}
+
+	process.exit()
+}

scripts/emoji-surrogates-statistics.js

@@ -0,0 +1,77 @@
+// @ts-check
+
+const fs = require("fs")
+const {join} = require("path")
+const s = fs.readFileSync(join(__dirname, "..", "src", "m2d", "converters", "emojis.txt"), "utf8").split("\n").map(x => encodeURIComponent(x))
+const searchPattern = "%EF%B8%8F"
+
+/**
+ * adapted from es.map.group-by.js in core-js
+ * @template K,V
+ * @param {V[]} items
+ * @param {(item: V) => K} fn
+ * @returns {Map<K, V[]>}
+ */
+function groupBy(items, fn) {
+	var map = new Map();
+	for (const value of items) {
+		var key = fn(value);
+		if (!map.has(key)) map.set(key, [value]);
+		else map.get(key).push(value);
+	}
+	return map;
+}
+
+/**
+ * @param {number[]} items
+ * @param {number} width
+ */
+function xhistogram(items, width) {
+	const chars = " ▏▎▍▌▋▊▉"
+	const max = items.reduce((a, c) => c > a ? c : a, 0)
+	return items.map(v => {
+		const p = v / max * (width-1)
+		return (
+			Array(Math.floor(p)).fill("█").join("") /* whole part */
+			+ chars[Math.ceil((p % 1) * (chars.length-1))] /* decimal part */
+		).padEnd(width)
+	})
+}
+
+/**
+ * @param {number[]} items
+ * @param {[number, number]} xrange
+ */
+function yhistogram(items, xrange, printHeader = false) {
+	const chars = "░▁_▂▃▄▅▆▇█"
+	const ones = "₀₁₂₃₄₅₆₇₈₉"
+	const tens = "0123456789"
+	const xy = []
+	let max = 0
+	/** value (x) -> frequency (y) */
+	const grouped = groupBy(items, x => x)
+	for (let i = xrange[0]; i <= xrange[1]; i++) {
+		if (printHeader) {
+			if (i === -1) process.stdout.write("-")
+			else if (i.toString().at(-1) === "0") process.stdout.write(tens[i/10])
+			else process.stdout.write(ones[i%10])
+		}
+		const y = grouped.get(i)?.length ?? 0
+		if (y > max) max = y
+		xy.push(y)
+	}
+	if (printHeader) console.log()
+	return xy.map(y => chars[Math.ceil(y / max * (chars.length-1))]).join("")
+}
+
+const grouped = groupBy(s, x => x.length)
+const sortedGroups = [...grouped.entries()].sort((a, b) => b[0] - a[0])
+let length = 0
+const lengthHistogram = xhistogram(sortedGroups.map(v => v[1].length), 10)
+for (let i = 0; i < sortedGroups.length; i++) {
+	const [k, v] = sortedGroups[i]
+	const l = lengthHistogram[i]
+	const h = yhistogram(v.map(x => x.indexOf(searchPattern)), [-1, k - searchPattern.length], i === 0)
+	if (i === 0) length = h.length + 1
+	console.log(`${h.padEnd(length, i % 2 === 0 ? "⸱" : " ")}length ${k.toString().padEnd(3)} ${l} ${v.length}`)
+}

scripts/migrate-from-old-bridge.js

@@ -2,8 +2,7 @@
 // @ts-check
 
 const assert = require("assert").strict
-/** @type {any} */ // @ts-ignore bad types from semaphore
-const Semaphore = require("@chriscdn/promise-semaphore")
+const {Semaphore} = require("@chriscdn/promise-semaphore")
 const sqlite = require("better-sqlite3")
 const HeatSync = require("heatsync")
 
@@ -115,7 +114,7 @@ async function migrateGuild(guild) {
 			// (By the way, thread_parent is always null here because thread rooms would never be migrated because they are not in the old bridge.)
 			db.transaction(() => {
 				db.prepare("DELETE FROM channel_room WHERE channel_id = ?").run(channel.id)
-				db.prepare("INSERT INTO channel_room (channel_id, room_id, name, nick, custom_avatar) VALUES (?, ?, ?, ?, ?)").run(channel.id, row.matrix_id, channel.name, preMigrationRow.nick, preMigrationRow.custom_avatar)
+				db.prepare("INSERT INTO channel_room (channel_id, room_id, name, nick, custom_avatar, guild_id) VALUES (?, ?, ?, ?, ?, ?)").run(channel.id, row.matrix_id, channel.name, preMigrationRow.nick, preMigrationRow.custom_avatar, guild.id)
 				console.log(`-- -- Added to database (transferred properties from previous OOYE room)`)
 			})()
 		} else {

scripts/setup.js

@@ -11,15 +11,12 @@ const {join} = require("path")
 
 const {prompt} = require("enquirer")
 const Input = require("enquirer/lib/prompts/input")
-const fetch = require("node-fetch").default
 const {magenta, bold, cyan} = require("ansi-colors")
 const HeatSync = require("heatsync")
 const {SnowTransfer} = require("snowtransfer")
 const DiscordTypes = require("discord-api-types/v10")
 const {createApp, defineEventHandler, toNodeListener} = require("h3")
 
-const args = require("minimist")(process.argv.slice(2), {string: ["emoji-guild"]})
-
 // Move database file if it's still in the old location
 if (fs.existsSync("db")) {
 	if (fs.existsSync("db/ooye.db")) {
@@ -51,24 +48,13 @@ passthrough.select = orm.select
 let registration = require("../src/matrix/read-registration")
 let {reg, getTemplateRegistration, writeRegistration, readRegistration, checkRegistration, registrationFilePath} = registration
 
+const {setupEmojis} = require("../src/m2d/actions/setup-emojis")
+
 function die(message) {
 	console.error(message)
 	process.exit(1)
 }
 
-async function uploadAutoEmoji(snow, guild, name, filename) {
-	let emoji = guild.emojis.find(e => e.name === name)
-	if (!emoji) {
-		console.log(`   Uploading ${name}...`)
-		const data = fs.readFileSync(filename, null)
-		emoji = await snow.guildAssets.createEmoji(guild.id, {name, image: "data:image/png;base64," + data.toString("base64")})
-	} else {
-		console.log(`   Reusing ${name}...`)
-	}
-	db.prepare("REPLACE INTO auto_emoji (name, emoji_id, guild_id) VALUES (?, ?, ?)").run(emoji.name, emoji.id, guild.id)
-	return emoji
-}
-
 async function suggestWellKnown(serverUrlPrompt, url, otherwise) {
 	try {
 		var json = await fetch(`${url}/.well-known/matrix/client`).then(res => res.json())
@@ -142,6 +128,7 @@ function defineEchoHandler() {
 		console.log("OOYE has its own web server. It needs to be accessible on the public internet.")
 		console.log("You need to enter a public URL where you will be able to host this web server.")
 		console.log("OOYE listens on localhost:6693, so you will probably have to set up a reverse proxy.")
+		console.log("Examples: https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/get-started.md#appendix")
 		console.log("Now listening on port 6693. Feel free to send some test requests.")
 		/** @type {{bridge_origin: string}} */
 		const bridgeOriginResponse = await prompt({
@@ -170,7 +157,7 @@ function defineEchoHandler() {
 		console.log("Go to https://discord.com/developers, create or pick an app, go to the Bot section, and reset the token.")
 		/** @type {SnowTransfer} */ // @ts-ignore
 		let snow = null
-		/** @type {{id: string, flags: number, redirect_uris: string[]}} */ // @ts-ignore
+		/** @type {{id: string, flags: number, redirect_uris: string[], description: string}} */ // @ts-ignore
 		let client = null
 		/** @type {{discord_token: string}} */
 		const discordTokenResponse = await prompt({
@@ -181,7 +168,7 @@ function defineEchoHandler() {
 				process.stdout.write(magenta(" checking, please wait..."))
 				try {
 					snow = new SnowTransfer(token)
-					client = await snow.requestHandler.request(`/applications/@me`, {}, "get")
+					client = await snow.requestHandler.request(`/applications/@me`, {}, "get", "json")
 					return true
 				} catch (e) {
 					return e.message
@@ -198,7 +185,7 @@ function defineEchoHandler() {
 				message: "Press Enter when you've enabled them",
 				validate: async token => {
 					process.stdout.write(magenta("checking, please wait..."))
-					client = await snow.requestHandler.request(`/applications/@me`, {}, "get")
+					client = await snow.requestHandler.request(`/applications/@me`, {}, "get", "json")
 					if (client.flags & mandatoryIntentFlags) {
 						return true
 					} else {
@@ -208,6 +195,34 @@ function defineEchoHandler() {
 			})
 		}
 
+		console.log("Would you like to require a password to add your bot to servers? This will discourage others from using your bridge.")
+		console.log("Important: To make it truly private, you MUST ALSO disable Public Bot in the Discord bot configuration page.")
+		/** @type {{web_password: string}} */
+		const passwordResponse = await prompt({
+			type: "text",
+			name: "web_password",
+			message: "Choose a simple password (optional)"
+		})
+
+		console.log("To fulfill license obligations, I recommend mentioning Out Of Your Element in your Discord bot's profile.")
+		console.log("On the Discord bot configuration page, go to General and add something like this to the description:")
+		console.log(cyan("Powered by **Out Of Your Element**"))
+		console.log(cyan("https://gitdab.com/cadence/out-of-your-element"))
+		await prompt({
+			type: "invisible",
+			name: "description",
+			message: "Press Enter to acknowledge",
+			validate: async token => {
+				process.stdout.write(magenta("checking, please wait..."))
+				client = await snow.requestHandler.request(`/applications/@me`, {}, "get", "json")
+				if (client.description?.match(/out.of.your.element/i)) {
+					return true
+				} else {
+					return "Description must name or link Out Of Your Element"
+				}
+			}
+		})
+
 		console.log("What is your Discord client secret?")
 		console.log(`You can find it in the application's OAuth2 section: https://discord.com/developers/applications/${client.id}/oauth2`)
 		/** @type {{discord_client_secret: string}} */
@@ -226,7 +241,7 @@ function defineEchoHandler() {
 				message: "Press Enter when you've added it",
 				validate: async token => {
 					process.stdout.write(magenta("checking, please wait..."))
-					client = await snow.requestHandler.request(`/applications/@me`, {}, "get")
+					client = await snow.requestHandler.request(`/applications/@me`, {}, "get", "json")
 					if (client.redirect_uris.includes(expectedUri)) {
 						return true
 					} else {
@@ -245,7 +260,8 @@ function defineEchoHandler() {
 				...bridgeOriginResponse,
 				server_origin: serverOrigin,
 				...discordTokenResponse,
-				...clientSecretResponse
+				...clientSecretResponse,
+				...passwordResponse
 			}
 		}
 		registration.reg = reg
@@ -319,8 +335,8 @@ function defineEchoHandler() {
 	await migrate.migrate(db)
 
 	// add initial rows to database, like adding the bot to sim...
-	const botID = Buffer.from(reg.ooye.discord_token.split(".")[0], "base64").toString()
-	db.prepare("INSERT OR IGNORE INTO sim (user_id, sim_name, localpart, mxid) VALUES (?, ?, ?, ?)").run(botID, reg.sender_localpart.slice(reg.ooye.namespace_prefix.length), reg.sender_localpart, mxid)
+	const client = await discord.snow.user.getSelf()
+	db.prepare("INSERT INTO sim (user_id, username, sim_name, mxid) VALUES (?, ?, ?, ?) ON CONFLICT DO NOTHING").run(client.id, client.username, reg.sender_localpart.slice(reg.ooye.namespace_prefix.length), mxid)
 
 	console.log("✅ Database is ready...")
 
@@ -332,47 +348,13 @@ function defineEchoHandler() {
 
 	console.log("✅ Matrix appservice login works...")
 
-	// upload the L1 L2 emojis to some guild
-	const emojis = db.prepare("SELECT name FROM auto_emoji WHERE name = 'L1' OR name = 'L2'").pluck().all()
-	if (emojis.length !== 2) {
-		// If an argument was supplied, always use that one
-		let guild = null
-		if (args["emoji-guild"]) {
-			if (typeof args["emoji-guild"] === "string") {
-				guild = await discord.snow.guild.getGuild(args["emoji-guild"])
-			}
-			if (!guild) return die(`Error: You asked emojis to be uploaded to guild ID ${args["emoji-guild"]}, but the bot isn't in that guild.`)
-		}
-		// Otherwise, check if we have already registered an auto emoji guild
-		if (!guild) {
-			const guildID = passthrough.select("auto_emoji", "guild_id", {name: "_"}).pluck().get()
-			if (guildID) {
-				guild = await discord.snow.guild.getGuild(guildID, false)
-			}
-		}
-		// Otherwise, check if we should create a new guild
-		if (!guild) {
-			const guilds = await discord.snow.user.getGuilds({limit: 11, with_counts: false})
-			if (guilds.length < 10) {
-				console.log("   Creating a guild for emojis...")
-				guild = await discord.snow.guild.createGuild({name: "OOYE Emojis"})
-			}
-		}
-		// Otherwise, it's the user's problem
-		if (!guild) {
-			return die(`Error: The bot needs to upload some emojis. Please say where to upload them to. Run setup again with --emoji-guild=GUILD_ID`)
-		}
-		// Upload those emojis to the chosen location
-		db.prepare("REPLACE INTO auto_emoji (name, emoji_id, guild_id) VALUES ('_', '_', ?)").run(guild.id)
-		await uploadAutoEmoji(discord.snow, guild, "L1", join(__dirname, "../docs/img/L1.png"))
-		await uploadAutoEmoji(discord.snow, guild, "L2", join(__dirname, "../docs/img/L2.png"))
-	}
+	// upload the L1 L2 emojis to user emojis
+	await setupEmojis()
 	console.log("✅ Emojis are ready...")
 
 	// set profile data on discord...
 	const avatarImageBuffer = await fetch("https://cadence.moe/friends/out_of_your_element.png").then(res => res.arrayBuffer())
 	await discord.snow.user.updateSelf({avatar: "data:image/png;base64," + Buffer.from(avatarImageBuffer).toString("base64")})
-	await discord.snow.requestHandler.request(`/applications/@me`, {}, "patch", "json", {description: "Powered by **Out Of Your Element**\nhttps://gitdab.com/cadence/out-of-your-element"})
 	console.log("✅ Discord profile updated...")
 
 	// set profile data on homeserver...

scripts/start-server.js

@@ -21,12 +21,7 @@ const DiscordClient = require("../src/d2m/discord-client")
 const discord = new DiscordClient(reg.ooye.discord_token, "half")
 passthrough.discord = discord
 
-const app = createApp()
-const router = createRouter()
-app.use(router)
-const server = createServer(toNodeListener(app))
-server.listen(reg.socket || new URL(reg.url).port)
-const as = Object.assign(new EventEmitter(), {app, router, server}) // @ts-ignore
+const {as} = require("../src/matrix/appservice")
 passthrough.as = as
 
 const orm = sync.require("../src/db/orm")
@@ -38,4 +33,6 @@ passthrough.select = orm.select
 	await discord.cloud.connect()
 	console.log("Discord gateway started")
 	sync.require("../src/web/server")
+
+	require("../src/stdin")
 })()

src/d2m/actions/add-reaction.js

@@ -25,7 +25,7 @@ async function addReaction(data) {
 	if (!parentID) return // Nothing can be done if the parent message was never bridged.
 	assert.equal(typeof parentID, "string")
 
-	const key = await emojiToKey.emojiToKey(data.emoji)
+	const key = await emojiToKey.emojiToKey(data.emoji, data.message_id)
 	const shortcode = key.startsWith("mxc://") ? `:${data.emoji.name}:` : undefined
 
 	const roomID = await createRoom.ensureRoom(data.channel_id)

src/d2m/actions/create-room.js

@@ -6,15 +6,19 @@ const Ty = require("../../types")
 const {reg} = require("../../matrix/read-registration")
 
 const passthrough = require("../../passthrough")
-const {discord, sync, db, select} = passthrough
+const {discord, sync, db, select, from} = passthrough
 /** @type {import("../../matrix/file")} */
 const file = sync.require("../../matrix/file")
 /** @type {import("../../matrix/api")} */
 const api = sync.require("../../matrix/api")
+/** @type {import("../../matrix/mreq")} */
+const mreq = sync.require("../../matrix/mreq")
 /** @type {import("../../matrix/kstate")} */
 const ks = sync.require("../../matrix/kstate")
 /** @type {import("../../discord/utils")} */
-const utils = sync.require("../../discord/utils")
+const dUtils = sync.require("../../discord/utils")
+/** @type {import("../../m2d/converters/utils")} */
+const mUtils = sync.require("../../m2d/converters/utils")
 /** @type {import("./create-space")} */
 const createSpace = sync.require("./create-space")
 
@@ -36,6 +40,8 @@ const PRIVACY_ENUMS = {
 
 const DEFAULT_PRIVACY_LEVEL = 0
 
+const READ_ONLY_ROOM_EVENTS_DEFAULT_POWER = 50
+
 /** @type {Map<string, Promise<string>>} channel ID -> Promise<room ID> */
 const inflightRoomCreate = new Map()
 
@@ -50,6 +56,7 @@ function convertNameAndTopic(channel, guild, customName) {
 	let channelPrefix =
 		( parentChannel?.type === DiscordTypes.ChannelType.GuildForum ? ""
 		: channel.type === DiscordTypes.ChannelType.PublicThread ? "[⛓️] "
+		: channel.type === DiscordTypes.ChannelType.AnnouncementThread ? "[⛓️] "
 		: channel.type === DiscordTypes.ChannelType.PrivateThread ? "[🔒⛓️] "
 		: channel.type === DiscordTypes.ChannelType.GuildVoice ? "[🔊] "
 		: "")
@@ -85,9 +92,10 @@ async function channelToKState(channel, guild, di) {
 		assert(typeof parentSpaceID === "string")
 	}
 
-	const channelRow = select("channel_room", ["nick", "custom_avatar"], {channel_id: channel.id}).get()
+	const channelRow = select("channel_room", ["nick", "custom_avatar", "custom_topic"], {channel_id: channel.id}).get()
 	const customName = channelRow?.nick
 	const customAvatar = channelRow?.custom_avatar
+	const hasCustomTopic = channelRow?.custom_topic
 	const [convertedName, convertedTopic] = convertNameAndTopic(channel, guild, customName)
 
 	const avatarEventContent = {}
@@ -114,8 +122,9 @@ async function channelToKState(channel, guild, di) {
 		join_rules = {join_rule: PRIVACY_ENUMS.ROOM_JOIN_RULES[privacyLevel]}
 	}
 
-	const everyonePermissions = utils.getPermissions([], guild.roles, undefined, channel.permission_overwrites)
-	const everyoneCanMentionEveryone = utils.hasAllPermissions(everyonePermissions, ["MentionEveryone"])
+	const everyonePermissions = dUtils.getPermissions([], guild.roles, undefined, channel.permission_overwrites)
+	const everyoneCanSend = dUtils.hasPermission(everyonePermissions, DiscordTypes.PermissionFlagsBits.SendMessages)
+	const everyoneCanMentionEveryone = dUtils.hasAllPermissions(everyonePermissions, ["MentionEveryone"])
 
 	const globalAdmins = select("member_power", ["mxid", "power_level"], {room_id: "*"}).all()
 	const globalAdminPower = globalAdmins.reduce((a, c) => (a[c.mxid] = c.power_level, a), {})
@@ -124,6 +133,7 @@ async function channelToKState(channel, guild, di) {
 	const spacePowerEvent = await di.api.getStateEvent(guildSpaceID, "m.room.power_levels", "")
 	const spacePower = spacePowerEvent.users
 
+	/** @type {any} */
 	const channelKState = {
 		"m.room.name/": {name: convertedName},
 		"m.room.topic/": {topic: convertedTopic},
@@ -136,7 +146,13 @@ async function channelToKState(channel, guild, di) {
 		},
 		/** @type {{join_rule: string, [x: string]: any}} */
 		"m.room.join_rules/": join_rules,
+		/** @type {Ty.Event.M_Power_Levels} */
 		"m.room.power_levels/": {
+			events_default: everyoneCanSend ? 0 : READ_ONLY_ROOM_EVENTS_DEFAULT_POWER,
+			events: {
+				"m.reaction": 0,
+				"m.room.redaction": 0 // only affects redactions of own events, required to be able to un-react
+			},
 			notifications: {
 				room: everyoneCanMentionEveryone ? 0 : 20
 			},
@@ -163,6 +179,16 @@ async function channelToKState(channel, guild, di) {
 		}
 	}
 
+	// Don't overwrite room topic if the topic has been customised
+	if (hasCustomTopic) delete channelKState["m.room.topic/"]
+
+	// Don't add a space parent if it's self service
+	// (The person setting up self-service has already put it in their preferred space to be able to get this far.)
+	const autocreate = select("guild_active", "autocreate", {guild_id: guild.id}).pluck().get()
+	if (autocreate === 0 && ![DiscordTypes.ChannelType.PrivateThread, DiscordTypes.ChannelType.PublicThread, DiscordTypes.ChannelType.AnnouncementThread].includes(channel.type)) {
+		delete channelKState[`m.space.parent/${parentSpaceID}`]
+	}
+
 	return {spaceID: parentSpaceID, privacyLevel, channelKState}
 }
 
@@ -207,8 +233,8 @@ async function createRoom(channel, guild, spaceID, kstate, privacyLevel) {
 		return roomID
 	})
 
-	// Put the newly created child into the space, no need to await this
-	_syncSpaceMember(channel, spaceID, roomID)
+	// Put the newly created child into the space
+	await _syncSpaceMember(channel, spaceID, roomID, guild.id)
 
 	return roomID
 }
@@ -227,6 +253,7 @@ async function postApplyPowerLevels(kstate, callback) {
 	const powerLevelContent = kstate["m.room.power_levels/"]
 	const kstateWithoutPowerLevels = {...kstate}
 	delete kstateWithoutPowerLevels["m.room.power_levels/"]
+	delete kstateWithoutPowerLevels["chat.schildi.hide_ui/read_receipts"]
 
 	/** @type {string} */
 	const roomID = await callback(kstateWithoutPowerLevels)
@@ -376,7 +403,7 @@ async function _syncRoom(channelID, shouldActuallySync) {
 	db.prepare("UPDATE channel_room SET name = ? WHERE room_id = ?").run(channel.name, roomID)
 
 	// sync room as space member
-	const spaceApply = _syncSpaceMember(channel, spaceID, roomID)
+	const spaceApply = _syncSpaceMember(channel, spaceID, roomID, guild.id)
 	await Promise.all([roomApply, spaceApply])
 
 	return roomID
@@ -392,7 +419,7 @@ function syncRoom(channelID) {
 	return _syncRoom(channelID, true)
 }
 
-async function _unbridgeRoom(channelID) {
+async function unbridgeChannel(channelID) {
 	/** @ts-ignore @type {DiscordTypes.APIGuildChannel} */
 	const channel = discord.channels.get(channelID)
 	assert.ok(channel)
@@ -401,37 +428,86 @@ async function _unbridgeRoom(channelID) {
 }
 
 /**
- * @param {{id: string, topic?: string?}} channel
+ * @param {{id: string, topic?: string?}} channel channel-ish (just needs an id, topic is optional)
  * @param {string} guildID
  */
 async function unbridgeDeletedChannel(channel, guildID) {
 	const roomID = select("channel_room", "room_id", {channel_id: channel.id}).pluck().get()
 	assert.ok(roomID)
-	const spaceID = select("guild_space", "space_id", {guild_id: guildID}).pluck().get()
-	assert.ok(spaceID)
+	const row = from("guild_space").join("guild_active", "guild_id").select("space_id", "autocreate").get()
+	assert.ok(row)
 
-	// remove room from being a space member
-	await api.sendState(roomID, "m.space.parent", spaceID, {})
-	await api.sendState(spaceID, "m.space.child", roomID, {})
+	let botInRoom = true
 
 	// remove declaration that the room is bridged
-	await api.sendState(roomID, "uk.half-shot.bridge", `moe.cadence.ooye://discord/${guildID}/${channel.id}`, {})
-	if ("topic" in channel) {
+	try {
+		await api.sendState(roomID, "uk.half-shot.bridge", `moe.cadence.ooye://discord/${guildID}/${channel.id}`, {})
+	} catch (e) {
+		if (String(e).includes("not in room")) {
+			botInRoom = false
+		} else {
+			throw e
+		}
+	}
+
+	if (botInRoom && "topic" in channel) {
 		// previously the Matrix topic would say the channel ID. we should remove that
 		await api.sendState(roomID, "m.room.topic", "", {topic: channel.topic || ""})
 	}
 
+	// delete webhook on discord
+	const webhook = select("webhook", ["webhook_id", "webhook_token"], {channel_id: channel.id}).get()
+	if (webhook) {
+		await discord.snow.webhook.deleteWebhook(webhook.webhook_id, webhook.webhook_token)
+		db.prepare("DELETE FROM webhook WHERE channel_id = ?").run(channel.id)
+	}
+
+	// delete room from database
+	db.prepare("DELETE FROM member_cache WHERE room_id = ?").run(roomID)
+	db.prepare("DELETE FROM channel_room WHERE room_id = ? AND channel_id = ?").run(roomID, channel.id) // cascades to most other tables, like messages
+
+	if (!botInRoom) return
+
+	// demote admins in room
+	/** @type {Ty.Event.M_Power_Levels} */
+	const powerLevelContent = await api.getStateEvent(roomID, "m.room.power_levels", "")
+	powerLevelContent.users ??= {}
+	const bot = `@${reg.sender_localpart}:${reg.ooye.server_name}`
+	for (const mxid of Object.keys(powerLevelContent.users)) {
+		if (powerLevelContent.users[mxid] >= 100 && mUtils.eventSenderIsFromDiscord(mxid) && mxid !== bot) {
+			delete powerLevelContent.users[mxid]
+			await api.sendState(roomID, "m.room.power_levels", "", powerLevelContent, mxid)
+		}
+	}
+
 	// send a notification in the room
 	await api.sendEvent(roomID, "m.room.message", {
 		msgtype: "m.notice",
 		body: "⚠️ This room was removed from the bridge."
 	})
 
+	// if it is an easy mode room, clean up the room from the managed space and make it clear it's not being bridged
+	// (don't do this for self-service rooms, because they might continue to be used on Matrix or linked somewhere else later)
+	if (row.autocreate === 1) {
+		// remove room from being a space member
+		await api.sendState(roomID, "m.space.parent", row.space_id, {})
+		await api.sendState(row.space_id, "m.space.child", roomID, {})
+	}
+
+	// if it is a self-service room, remove sim members
+	// (the room can be used with less clutter and the member list makes sense if it's bridged somewhere else)
+	if (row.autocreate === 0) {
+		// remove sim members
+		const members = db.prepare("SELECT mxid FROM sim_member WHERE room_id = ? AND mxid <> ?").pluck().all(roomID, bot)
+		const preparedDelete = db.prepare("DELETE FROM sim_member WHERE room_id = ? AND mxid = ?")
+		for (const mxid of members) {
+			await api.leaveRoom(roomID, mxid)
+			preparedDelete.run(roomID, mxid)
+		}
+	}
+
 	// leave room
 	await api.leaveRoom(roomID)
-
-	// delete room from database
-	db.prepare("DELETE FROM channel_room WHERE room_id = ? AND channel_id = ?").run(roomID, channel.id)
 }
 
 /**
@@ -439,14 +515,25 @@ async function unbridgeDeletedChannel(channel, guildID) {
  * @param {DiscordTypes.APIGuildTextChannel} channel
  * @param {string} spaceID
  * @param {string} roomID
+ * @param {string} guild_id
  * @returns {Promise<string[]>}
  */
-async function _syncSpaceMember(channel, spaceID, roomID) {
+async function _syncSpaceMember(channel, spaceID, roomID, guild_id) {
+	// If space is self-service then only permit changes to space parenting for threads
+	// (The person setting up self-service has already put it in their preferred space to be able to get this far.)
+	const autocreate = select("guild_active", "autocreate", {guild_id}).pluck().get()
+	if (autocreate === 0 && ![DiscordTypes.ChannelType.PrivateThread, DiscordTypes.ChannelType.PublicThread, DiscordTypes.ChannelType.AnnouncementThread].includes(channel.type)) {
+		return []
+	}
+
 	const spaceKState = await ks.roomToKState(spaceID)
 	let spaceEventContent = {}
 	if (
 		channel.type !== DiscordTypes.ChannelType.PrivateThread // private threads do not belong in the space (don't offer people something they can't join)
-		&& !channel["thread_metadata"]?.archived // archived threads do not belong in the space (don't offer people conversations that are no longer relevant)
+		&& (
+			!channel["thread_metadata"]?.archived // archived threads do not belong in the space (don't offer people conversations that are no longer relevant)
+			|| discord.channels.get(channel.parent_id || "")?.type === DiscordTypes.ChannelType.GuildForum
+		)
 	) {
 		spaceEventContent = {
 			via: [reg.ooye.server_name]
@@ -472,6 +559,7 @@ async function createAllForGuild(guildID) {
 }
 
 module.exports.DEFAULT_PRIVACY_LEVEL = DEFAULT_PRIVACY_LEVEL
+module.exports.READ_ONLY_ROOM_EVENTS_DEFAULT_POWER = READ_ONLY_ROOM_EVENTS_DEFAULT_POWER
 module.exports.PRIVACY_ENUMS = PRIVACY_ENUMS
 module.exports.createRoom = createRoom
 module.exports.ensureRoom = ensureRoom
@@ -480,7 +568,7 @@ module.exports.createAllForGuild = createAllForGuild
 module.exports.channelToKState = channelToKState
 module.exports.postApplyPowerLevels = postApplyPowerLevels
 module.exports._convertNameAndTopic = convertNameAndTopic
-module.exports._unbridgeRoom = _unbridgeRoom
+module.exports.unbridgeChannel = unbridgeChannel
 module.exports.unbridgeDeletedChannel = unbridgeDeletedChannel
 module.exports.existsOrAutocreatable = existsOrAutocreatable
 module.exports.assertExistsOrAutocreatable = assertExistsOrAutocreatable

src/d2m/actions/create-room.test.js

@@ -14,7 +14,7 @@ test("channel2room: discoverable privacy room", async t => {
 	let called = 0
 	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
 		called++
-		t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 		t.equal(type, "m.room.power_levels")
 		t.equal(key, "")
 		return {users: {"@example:matrix.org": 50}}
@@ -36,7 +36,7 @@ test("channel2room: linkable privacy room", async t => {
 	let called = 0
 	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
 		called++
-		t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 		t.equal(type, "m.room.power_levels")
 		t.equal(key, "")
 		return {users: {"@example:matrix.org": 50}}
@@ -57,7 +57,7 @@ test("channel2room: invite-only privacy room", async t => {
 	let called = 0
 	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
 		called++
-		t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 		t.equal(type, "m.room.power_levels")
 		t.equal(key, "")
 		return {users: {"@example:matrix.org": 50}}
@@ -76,7 +76,7 @@ test("channel2room: room where limited people can mention everyone", async t =>
 	let called = 0
 	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
 		called++
-		t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 		t.equal(type, "m.room.power_levels")
 		t.equal(key, "")
 		return {users: {"@example:matrix.org": 50}}
@@ -94,6 +94,109 @@ test("channel2room: room where limited people can mention everyone", async t =>
 	t.equal(called, 1)
 })
 
+test("channel2room: matrix room that already has a custom topic set", async t => {
+	let called = 0
+	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
+		called++
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
+		t.equal(type, "m.room.power_levels")
+		t.equal(key, "")
+		return {}
+	}
+	db.prepare("UPDATE channel_room SET custom_topic = 1 WHERE channel_id = ?").run(testData.channel.general.id)
+	const expected = mixin({}, testData.room.general, {"m.room.power_levels/": {notifications: {room: 20}}})
+	// @ts-ignore
+	delete expected["m.room.topic/"]
+	t.deepEqual(
+		kstateStripConditionals(await channelToKState(testData.channel.general, testData.guild.general, {api: {getStateEvent}}).then(x => x.channelKState)),
+		expected
+	)
+	t.equal(called, 1)
+})
+
+test("channel2room: read-only discord channel", async t => {
+	let called = 0
+	async function getStateEvent(roomID, type, key) { // getting power levels from space to apply to room
+		called++
+		t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
+		t.equal(type, "m.room.power_levels")
+		t.equal(key, "")
+		return {}
+	}
+	const expected = {
+		"chat.schildi.hide_ui/read_receipts": {},
+		"m.room.avatar/": {
+			url: {
+				$url: "/icons/112760669178241024/a_f83622e09ead74f0c5c527fe241f8f8c.png?size=1024",
+			},
+		},
+		"m.room.guest_access/": {
+			guest_access: "can_join",
+		},
+		"m.room.history_visibility/": {
+			history_visibility: "shared",
+		},
+		"m.room.join_rules/": {
+			allow: [
+				{
+					room_id: "!jjmvBegULiLucuWEHU:cadence.moe",
+					type: "m.room_membership",
+				},
+			],
+			join_rule: "restricted",
+		},
+		"m.room.name/": {
+			name: "updates",
+		},
+		"m.room.topic/": {
+      	topic: "Updates and release announcements for Out Of Your Element.\n\nChannel ID: 1161864271370666075\nGuild ID: 112760669178241024"
+		},
+		"m.room.power_levels/": {
+			events_default: 50, // <-- it should be read-only!
+			events: {
+				"m.reaction": 0,
+				"m.room.redaction": 0
+			},
+			notifications: {
+				room: 20,
+			},
+			users: {
+				"@test_auto_invite:example.org": 100,
+			},
+		},
+		"m.space.parent/!jjmvBegULiLucuWEHU:cadence.moe": {
+			canonical: true,
+			via: [
+				"cadence.moe",
+			],
+		},
+		"uk.half-shot.bridge/moe.cadence.ooye://discord/112760669178241024/1161864271370666075": {
+			bridgebot: "@_ooye_bot:cadence.moe",
+			channel: {
+				displayname: "updates",
+				external_url: "https://discord.com/channels/112760669178241024/1161864271370666075",
+				id: "1161864271370666075",
+			},
+			network: {
+				avatar_url: {
+					"$url": "/icons/112760669178241024/a_f83622e09ead74f0c5c527fe241f8f8c.png?size=1024",
+				},
+				displayname: "Psychonauts 3",
+				id: "112760669178241024",
+			},
+			protocol: {
+				displayname: "Discord",
+				id: "discord",
+			}
+		}
+	}
+	t.deepEqual(
+		kstateStripConditionals(await channelToKState(testData.channel.updates, testData.guild.general, {api: {getStateEvent}}).then(x => x.channelKState)),
+		expected
+	)
+	t.equal(called, 1)
+})
+
 test("convertNameAndTopic: custom name and topic", t => {
 	t.deepEqual(
 		_convertNameAndTopic({id: "123", name: "the-twilight-zone", topic: "Spooky stuff here. :ghost:", type: 0}, {id: "456"}, "hauntings"),

src/d2m/actions/create-space.js

@@ -31,6 +31,8 @@ async function createSpace(guild, kstate) {
 	const topic = kstate["m.room.topic/"]?.topic || undefined
 	assert(name)
 
+	const memberCount = guild["member_count"] ?? guild.approximate_member_count ?? 0
+	const enablePresenceByDefault = +(memberCount < 50) // scary! all active users in a presence-enabled guild will be pinging the server every <30 seconds to stay online
 	const globalAdmins = select("member_power", "mxid", {room_id: "*"}).pluck().all()
 
 	const roomID = await createRoom.postApplyPowerLevels(kstate, async kstate => {
@@ -50,7 +52,7 @@ async function createSpace(guild, kstate) {
 			initial_state: await ks.kstateToState(kstate)
 		})
 	})
-	db.prepare("INSERT INTO guild_space (guild_id, space_id) VALUES (?, ?)").run(guild.id, roomID)
+	db.prepare("INSERT INTO guild_space (guild_id, space_id, presence) VALUES (?, ?, ?)").run(guild.id, roomID, enablePresenceByDefault)
 	return roomID
 }
 
@@ -127,16 +129,10 @@ async function _syncSpace(guild, shouldActuallySync) {
 		// don't try to update rooms with custom avatars though
 		const roomsWithCustomAvatars = select("channel_room", "room_id", {}, "WHERE custom_avatar IS NOT NULL").pluck().all()
 
-		const state = await ks.kstateToState(spaceKState)
-		const childRooms = state.filter(({type, state_key, content}) => {
-			return type === "m.space.child" && "via" in content && !roomsWithCustomAvatars.includes(state_key)
-		}).map(({state_key}) => state_key)
-
-		for (const roomID of childRooms) {
-			const avatarEventContent = await api.getStateEvent(roomID, "m.room.avatar", "")
-			if (avatarEventContent.url !== newAvatarState.url) {
-				await api.sendState(roomID, "m.room.avatar", "", newAvatarState)
-			}
+		for await (const room of api.generateFullHierarchy(spaceID)) {
+			if (room.avatar_url === newAvatarState.url) continue
+			if (roomsWithCustomAvatars.includes(room.room_id)) continue
+			await api.sendState(room.room_id, "m.room.avatar", "", newAvatarState)
 		}
 	}
 

src/d2m/actions/delete-message.js

@@ -16,7 +16,6 @@ async function deleteMessage(data) {
 
 	const eventsToRedact = select("event_message", "event_id", {message_id: data.id}).pluck().all()
 	db.prepare("DELETE FROM message_channel WHERE message_id = ?").run(data.id)
-	db.prepare("DELETE FROM event_message WHERE message_id = ?").run(data.id)
 	for (const eventID of eventsToRedact) {
 		// Unfortunately, we can't specify a sender to do the redaction as, unless we find out that info via the audit logs
 		await api.redactEvent(row.room_id, eventID)
@@ -35,7 +34,6 @@ async function deleteMessageBulk(data) {
 	const sids = JSON.stringify(data.ids)
 	const eventsToRedact = from("event_message").pluck("event_id").and("WHERE message_id IN (SELECT value FROM json_each(?))").all(sids)
 	db.prepare("DELETE FROM message_channel WHERE message_id IN (SELECT value FROM json_each(?))").run(sids)
-	db.prepare("DELETE FROM event_message WHERE message_id IN (SELECT value FROM json_each(?))").run(sids)
 	for (const eventID of eventsToRedact) {
 		// Awaiting will make it go slower, but since this could be a long-running operation either way, we want to leave rate limit capacity for other operations
 		await api.redactEvent(roomID, eventID)

src/d2m/actions/edit-message.js

@@ -22,9 +22,7 @@ async function editMessage(message, guild, row) {
 	if (row && row.speedbump_webhook_id === message.webhook_id) {
 		// Handle the PluralKit public instance
 		if (row.speedbump_id === "466378653216014359") {
-			const root = await registerPkUser.fetchMessage(message.id)
-			assert(root.member)
-			senderMxid = await registerPkUser.ensureSimJoined(root, roomID)
+			senderMxid = await registerPkUser.syncUser(message.id, message.author, roomID, false)
 		}
 	}
 
@@ -61,7 +59,7 @@ async function editMessage(message, guild, row) {
 
 	// 4. Send all the things.
 	if (eventsToSend.length) {
-		db.prepare("REPLACE INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(message.id, message.channel_id)
+		db.prepare("INSERT OR IGNORE INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(message.id, message.channel_id)
 	}
 	for (const content of eventsToSend) {
 		const eventType = content.$type

src/d2m/actions/lottie.js

@@ -33,7 +33,7 @@ async function convert(stickerItem) {
 	if (res.status !== 200) throw new Error("Sticker data file not found.")
 	const text = await res.text()
 
-	// Convert to PNG (readable stream)
+	// Convert to PNG (stream.Readable)
 	const readablePng = await convertLottie.convert(text)
 
 	// Upload to MXC

src/d2m/actions/register-pk-user.js

@@ -3,10 +3,9 @@
 const assert = require("assert")
 const {reg} = require("../../matrix/read-registration")
 const Ty = require("../../types")
-const fetch = require("node-fetch").default
 
 const passthrough = require("../../passthrough")
-const {discord, sync, db, select} = passthrough
+const {sync, db, select, from} = passthrough
 /** @type {import("../../matrix/api")} */
 const api = sync.require("../../matrix/api")
 /** @type {import("../../matrix/file")} */
@@ -21,6 +20,20 @@ const registerUser = sync.require("./register-user")
  * @prop {string} id
  */
 
+/** @returns {Promise<Ty.PkMessage>} */
+async function fetchMessage(messageID) {
+	try {
+		var res = await fetch(`https://api.pluralkit.me/v2/messages/${messageID}`)
+	} catch (networkError) {
+		// Network issue, raise a more readable message
+		throw new Error(`Failed to connect to PK API: ${networkError.toString()}`)
+	}
+	if (!res.ok) throw new Error(`PK API returned an error: ${await res.text()}`)
+	const root = await res.json()
+	if (!root.member) throw new Error(`PK API didn't return member data: ${JSON.stringify(root)}`)
+	return root
+}
+
 /**
  * A sim is an account that is being simulated by the bridge to copy events from the other side.
  * @param {Ty.PkMessage} pkMessage
@@ -33,7 +46,7 @@ async function createSim(pkMessage) {
 	const mxid = `@${localpart}:${reg.ooye.server_name}`
 
 	// Save chosen name in the database forever
-	db.prepare("INSERT INTO sim (user_id, sim_name, localpart, mxid) VALUES (?, ?, ?, ?)").run(pkMessage.member.uuid, simName, localpart, mxid)
+	db.prepare("INSERT INTO sim (user_id, username, sim_name, mxid) VALUES (?, ?, ?, ?)").run(pkMessage.member.uuid, simName, simName, mxid)
 
 	// Register matrix user with that name
 	try {
@@ -96,6 +109,7 @@ async function ensureSimJoined(pkMessage, roomID) {
 }
 
 /**
+ * Generate profile data based on webhook displayname and configured avatar.
  * @param {Ty.PkMessage} pkMessage
  * @param {WebhookAuthor} author
  */
@@ -116,49 +130,47 @@ async function memberToStateContent(pkMessage, author) {
 
 /**
  * Sync profile data for a sim user. This function follows the following process:
- * 1. Join the sim to the room if needed
- * 2. Make an object of what the new room member state content would be, including uploading the profile picture if it hasn't been done before
- * 3. Compare against the previously known state content, which is helpfully stored in the database
- * 4. If the state content has changed, send it to Matrix and update it in the database for next time
- * @param {WebhookAuthor} author
- * @param {Ty.PkMessage} pkMessage
- * @param {string} roomID
+ * 1. Look up data about proxy user from API
+ * 2. If this fails, try to use previously cached data (won't sync)
+ * 3. Create and join the sim to the room if needed
+ * 4. Make an object of what the new room member state content would be, including uploading the profile picture if it hasn't been done before
+ * 5. Compare against the previously known state content, which is helpfully stored in the database
+ * 6. If the state content has changed, send it to Matrix and update it in the database for next time
+ * @param {string} messageID to call API with
+ * @param {WebhookAuthor} author for profile data
+ * @param {string} roomID room to join member to
+ * @param {boolean} shouldActuallySync whether to actually sync updated user data or just ensure it's joined
  * @returns {Promise<string>} mxid of the updated sim
  */
-async function syncUser(author, pkMessage, roomID) {
-	const mxid = await ensureSimJoined(pkMessage, roomID)
-	// Update the sim_proxy table, so mentions can look up the original sender later
-	db.prepare("INSERT OR IGNORE INTO sim_proxy (user_id, proxy_owner_id, displayname) VALUES (?, ?, ?)").run(pkMessage.member.uuid, pkMessage.sender, author.username)
-	// Sync the member state
-	const content = await memberToStateContent(pkMessage, author)
-	const currentHash = registerUser._hashProfileContent(content, 0)
-	const existingHash = select("sim_member", "hashed_profile_content", {room_id: roomID, mxid}).safeIntegers().pluck().get()
-	// only do the actual sync if the hash has changed since we last looked
-	if (existingHash !== currentHash) {
-		await api.sendState(roomID, "m.room.member", mxid, content, mxid)
-		db.prepare("UPDATE sim_member SET hashed_profile_content = ? WHERE room_id = ? AND mxid = ?").run(currentHash, roomID, mxid)
+async function syncUser(messageID, author, roomID, shouldActuallySync) {
+	try {
+		// API lookup
+		var pkMessage = await fetchMessage(messageID)
+		db.prepare("INSERT OR IGNORE INTO sim_proxy (user_id, proxy_owner_id, displayname) VALUES (?, ?, ?)").run(pkMessage.member.uuid, pkMessage.sender, author.username)
+	} catch (e) {
+		// Fall back to offline cache
+		const senderMxid = from("sim_proxy").join("sim", "user_id").join("sim_member", "mxid").where({displayname: author.username, room_id: roomID}).pluck("mxid").get()
+		if (!senderMxid) throw e
+		return senderMxid
 	}
+
+	// Create and join the sim to the room if needed
+	const mxid = await ensureSimJoined(pkMessage, roomID)
+
+	if (shouldActuallySync) {
+		// Build current profile data
+		const content = await memberToStateContent(pkMessage, author)
+		const currentHash = registerUser._hashProfileContent(content, 0)
+		const existingHash = select("sim_member", "hashed_profile_content", {room_id: roomID, mxid}).safeIntegers().pluck().get()
+
+		// Only do the actual sync if the hash has changed since we last looked
+		if (existingHash !== currentHash) {
+			await api.sendState(roomID, "m.room.member", mxid, content, mxid)
+			db.prepare("UPDATE sim_member SET hashed_profile_content = ? WHERE room_id = ? AND mxid = ?").run(currentHash, roomID, mxid)
+		}
+	}
+
 	return mxid
 }
 
-/** @returns {Promise<Ty.PkMessage>} */
-async function fetchMessage(messageID) {
-	// Their backend is weird. Sometimes it says "message not found" (code 20006) on the first try, so we make multiple attempts.
-	let attempts = 0
-	do {
-		var res = await fetch(`https://api.pluralkit.me/v2/messages/${messageID}`)
-		if (res.ok) return res.json()
-
-		// I think the backend needs some time to update.
-		await new Promise(resolve => setTimeout(resolve, 2000))
-	} while (++attempts < 3)
-
-	const errorMessage = await res.json()
-	throw new Error(`PK API returned an error after ${attempts} tries: ${JSON.stringify(errorMessage)}`)
-}
-
-module.exports._memberToStateContent = memberToStateContent
-module.exports.ensureSim = ensureSim
-module.exports.ensureSimJoined = ensureSimJoined
 module.exports.syncUser = syncUser
-module.exports.fetchMessage = fetchMessage

src/d2m/actions/register-user.js

@@ -3,6 +3,7 @@
 const assert = require("assert").strict
 const {reg} = require("../../matrix/read-registration")
 const DiscordTypes = require("discord-api-types/v10")
+const Ty = require("../../types")
 const mixin = require("@cloudrac3r/mixin-deep")
 
 const passthrough = require("../../passthrough")
@@ -15,6 +16,8 @@ const file = sync.require("../../matrix/file")
 const utils = sync.require("../../discord/utils")
 /** @type {import("../converters/user-to-mxid")} */
 const userToMxid = sync.require("../converters/user-to-mxid")
+/** @type {import("./create-room")} */
+const createRoom = sync.require("./create-room")
 /** @type {import("xxhash-wasm").XXHashAPI} */ // @ts-ignore
 let hasher = null
 // @ts-ignore
@@ -33,7 +36,7 @@ async function createSim(user) {
 
 	// Save chosen name in the database forever
 	// Making this database change right away so that in a concurrent registration, the 2nd registration will already have generated a different localpart because it can see this row when it generates
-	db.prepare("INSERT INTO sim (user_id, sim_name, localpart, mxid) VALUES (?, ?, ?, ?)").run(user.id, simName, localpart, mxid)
+	db.prepare("INSERT INTO sim (user_id, username, sim_name, mxid) VALUES (?, ?, ?, ?)").run(user.id, user.username, simName, mxid)
 
 	// Register matrix user with that name
 	try {
@@ -97,12 +100,12 @@ async function ensureSimJoined(user, roomID) {
 
 /**
  * @param {DiscordTypes.APIUser} user
- * @param {Omit<DiscordTypes.APIGuildMember, "user">} member
+ * @param {Omit<DiscordTypes.APIGuildMember, "user"> | undefined} member
  */
 async function memberToStateContent(user, member, guildID) {
 	let displayname = user.username
 	if (user.global_name) displayname = user.global_name
-	if (member.nick) displayname = member.nick
+	if (member?.nick) displayname = member.nick
 
 	const content = {
 		displayname,
@@ -117,7 +120,7 @@ async function memberToStateContent(user, member, guildID) {
 		}
 	}
 
-	if (member.avatar || user.avatar) {
+	if (member?.avatar || user.avatar) {
 		// const avatarPath = file.userAvatar(user) // the user avatar only
 		const avatarPath = file.memberAvatar(guildID, user, member) // the member avatar or the user avatar
 		content["moe.cadence.ooye.member"].avatar = avatarPath
@@ -130,13 +133,16 @@ async function memberToStateContent(user, member, guildID) {
 /**
  * https://gitdab.com/cadence/out-of-your-element/issues/9
  * @param {DiscordTypes.APIUser} user
- * @param {Omit<DiscordTypes.APIGuildMember, "user">} member
+ * @param {Omit<DiscordTypes.APIGuildMember, "user"> | undefined} member
  * @param {DiscordTypes.APIGuild} guild
  * @param {DiscordTypes.APIGuildChannel} channel
  * @returns {number} 0 to 100
  */
 function memberToPowerLevel(user, member, guild, channel) {
+	if (!member) return 0
+
 	const permissions = utils.getPermissions(member.roles, guild.roles, user.id, channel.permission_overwrites)
+	const everyonePermissions = utils.getPermissions([], guild.roles, undefined, channel.permission_overwrites)
 	/*
 	 * PL 100 = Administrator = People who can brick the room. RATIONALE:
 	 * 	- Administrator.
@@ -156,8 +162,14 @@ function memberToPowerLevel(user, member, guild, channel) {
 	 * 	- Moderate Members.
 	 */
 	if (utils.hasSomePermissions(permissions, ["ManageMessages", "ManageNicknames", "ManageThreads", "KickMembers", "BanMembers", "MuteMembers", "DeafenMembers", "ModerateMembers"])) return 50
+	/* PL 50 = if room is read-only but the user has been specially allowed to send messages */
+	const everyoneCanSend = utils.hasPermission(everyonePermissions, DiscordTypes.PermissionFlagsBits.SendMessages)
+	const userCanSend = utils.hasPermission(permissions, DiscordTypes.PermissionFlagsBits.SendMessages)
+	if (!everyoneCanSend && userCanSend) return createRoom.READ_ONLY_ROOM_EVENTS_DEFAULT_POWER
 	/* PL 20 = Mention Everyone for technical reasons. */
-	if (utils.hasSomePermissions(permissions, ["MentionEveryone"])) return 20
+	const everyoneCanMentionEveryone = utils.hasPermission(everyonePermissions, DiscordTypes.PermissionFlagsBits.MentionEveryone)
+	const userCanMentionEveryone = utils.hasPermission(permissions, DiscordTypes.PermissionFlagsBits.MentionEveryone)
+	if (!everyoneCanMentionEveryone && userCanMentionEveryone) return 20
 	return 0
 }
 
@@ -179,7 +191,7 @@ function _hashProfileContent(content, powerLevel) {
  * 4. Compare against the previously known state content, which is helpfully stored in the database
  * 5. If the state content or power level have changed, send them to Matrix and update them in the database for next time
  * @param {DiscordTypes.APIUser} user
- * @param {Omit<DiscordTypes.APIGuildMember, "user">} member
+ * @param {Omit<DiscordTypes.APIGuildMember, "user"> | undefined} member
  * @param {DiscordTypes.APIGuildChannel} channel
  * @param {DiscordTypes.APIGuild} guild
  * @param {string} roomID
@@ -192,16 +204,14 @@ async function syncUser(user, member, channel, guild, roomID) {
 	const currentHash = _hashProfileContent(content, powerLevel)
 	const existingHash = select("sim_member", "hashed_profile_content", {room_id: roomID, mxid}).safeIntegers().pluck().get()
 	// only do the actual sync if the hash has changed since we last looked
-	if (existingHash !== currentHash) {
+	const hashHasChanged = existingHash !== currentHash
+	// however, do not overwrite pre-existing data if we already have data and `member` is not accessible, because this would replace good data with bad data
+	const wouldOverwritePreExisting = existingHash && !member
+	if (hashHasChanged && !wouldOverwritePreExisting) {
 		// Update room member state
 		await api.sendState(roomID, "m.room.member", mxid, content, mxid)
 		// Update power levels
-		const powerLevelsStateContent = await api.getStateEvent(roomID, "m.room.power_levels", "")
-		const oldPowerLevel = powerLevelsStateContent.users?.[mxid] || 0
-		mixin(powerLevelsStateContent, {users: {[mxid]: powerLevel}})
-		if (powerLevel === 0) delete powerLevelsStateContent.users[mxid] // keep the event compact
-		const sendPowerLevelAs = powerLevel < oldPowerLevel ? mxid : undefined // bridge bot won't not have permission to demote equal power users, so do this action as themselves
-		await api.sendState(roomID, "m.room.power_levels", "", powerLevelsStateContent, sendPowerLevelAs)
+		await api.setUserPower(roomID, mxid, powerLevel)
 		// Update cached hash
 		db.prepare("UPDATE sim_member SET hashed_profile_content = ? WHERE room_id = ? AND mxid = ?").run(currentHash, roomID, mxid)
 	}
@@ -245,3 +255,4 @@ module.exports.ensureSim = ensureSim
 module.exports.ensureSimJoined = ensureSimJoined
 module.exports.syncUser = syncUser
 module.exports.syncAllUsersInRoom = syncAllUsersInRoom
+module.exports._memberToPowerLevel = memberToPowerLevel

src/d2m/actions/register-user.test.js

@@ -1,10 +1,12 @@
-const {_memberToStateContent} = require("./register-user")
+const {_memberToStateContent, _memberToPowerLevel} = require("./register-user")
 const {test} = require("supertape")
-const testData = require("../../../test/data")
+const data = require("../../../test/data")
+const mixin = require("@cloudrac3r/mixin-deep")
+const DiscordTypes = require("discord-api-types/v10")
 
 test("member2state: without member nick or avatar", async t => {
 	t.deepEqual(
-		await _memberToStateContent(testData.member.kumaccino.user, testData.member.kumaccino, testData.guild.general.id),
+		await _memberToStateContent(data.member.kumaccino.user, data.member.kumaccino, data.guild.general.id),
 		{
 			avatar_url: "mxc://cadence.moe/UpAeIqeclhKfeiZNdIWNcXXL",
 			displayname: "kumaccino",
@@ -24,7 +26,7 @@ test("member2state: without member nick or avatar", async t => {
 
 test("member2state: with global name, without member nick or avatar", async t => {
 	t.deepEqual(
-		await _memberToStateContent(testData.member.papiophidian.user, testData.member.papiophidian, testData.guild.general.id),
+		await _memberToStateContent(data.member.papiophidian.user, data.member.papiophidian, data.guild.general.id),
 		{
 			avatar_url: "mxc://cadence.moe/JPzSmALLirnIprlSMKohSSoX",
 			displayname: "PapiOphidian",
@@ -44,7 +46,7 @@ test("member2state: with global name, without member nick or avatar", async t =>
 
 test("member2state: with member nick and avatar", async t => {
 	t.deepEqual(
-		await _memberToStateContent(testData.member.sheep.user, testData.member.sheep, testData.guild.general.id),
+		await _memberToStateContent(data.member.sheep.user, data.member.sheep, data.guild.general.id),
 		{
 			avatar_url: "mxc://cadence.moe/rfemHmAtcprjLEiPiEuzPhpl",
 			displayname: "The Expert's Submarine",
@@ -61,3 +63,64 @@ test("member2state: with member nick and avatar", async t => {
 		}
 	)
 })
+
+test("member2power: default to zero if member roles unknown", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, null, data.guild.data_horde, data.channel.saving_the_world)
+	t.equal(power, 0)
+})
+
+test("member2power: unremarkable = 0", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: []
+	}, data.guild.data_horde, data.channel.general)
+	t.equal(power, 0)
+})
+
+test("member2power: can mention everyone = 20", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: ["684524730274807911"]
+	}, data.guild.data_horde, data.channel.general)
+	t.equal(power, 20)
+})
+
+test("member2power: can send messages in protected channel due to role = 50", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: ["684524730274807911"]
+	}, data.guild.data_horde, data.channel.saving_the_world)
+	t.equal(power, 50)
+})
+
+test("member2power: can send messages in protected channel due to user override = 50", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: []
+	}, data.guild.data_horde, mixin({}, data.channel.saving_the_world, {
+		permission_overwrites: data.channel.saving_the_world.permission_overwrites.concat({
+			type: DiscordTypes.OverwriteType.member,
+			id: data.user.clyde_ai.id,
+			allow: String(DiscordTypes.PermissionFlagsBits.SendMessages),
+			deny: "0"
+		})
+	}))
+	t.equal(power, 50)
+})
+
+test("member2power: can kick users = 50", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: ["682789592390281245"]
+	}, data.guild.data_horde, data.channel.general)
+	t.equal(power, 50)
+})
+
+test("member2power: can manage channels = 100", async t => {
+	const power = _memberToPowerLevel(data.user.clyde_ai, {
+		roles: ["665290147377578005"]
+	}, data.guild.data_horde, data.channel.saving_the_world)
+	t.equal(power, 100)
+})
+
+test("member2power: pathfinder use case", async t => {
+	const power = _memberToPowerLevel(data.user.jerassicore, {
+		roles: ["1235396773510647810", "1359752622130593802", "1249165855632265267", "1380768596929806356", "1380756348190462015"]
+	}, data.guild.pathfinder, data.channel.character_art)
+	t.equal(power, 50)
+})

src/d2m/actions/remove-reaction.js

@@ -43,7 +43,7 @@ async function removeSomeReactions(data) {
  * @param {Ty.Event.Outer<Ty.Event.M_Reaction>[]} reactions
  */
 async function removeReaction(data, reactions) {
-	const key = await emojiToKey.emojiToKey(data.emoji)
+	const key = await emojiToKey.emojiToKey(data.emoji, data.message_id)
 	return converter.removeReaction(data, reactions, key)
 }
 
@@ -52,8 +52,8 @@ async function removeReaction(data, reactions) {
  * @param {Ty.Event.Outer<Ty.Event.M_Reaction>[]} reactions
  */
 async function removeEmojiReaction(data, reactions) {
-	const key = await emojiToKey.emojiToKey(data.emoji)
-	const discordPreferredEncoding = emoji.encodeEmoji(key, undefined)
+	const key = await emojiToKey.emojiToKey(data.emoji, data.message_id)
+	const discordPreferredEncoding = await emoji.encodeEmoji(key, undefined)
 	db.prepare("DELETE FROM reaction WHERE message_id = ? AND encoded_emoji = ?").run(data.message_id, discordPreferredEncoding)
 
 	return converter.removeEmojiReaction(data, reactions, key)

src/d2m/actions/send-message.js

@@ -31,23 +31,20 @@ async function sendMessage(message, channel, guild, row) {
 	if (!dUtils.isWebhookMessage(message)) {
 		if (message.author.id === discord.application.id) {
 			// no need to sync the bot's own user
-		} else if (message.member) { // available on a gateway message create event
+		} else {
 			senderMxid = await registerUser.syncUser(message.author, message.member, channel, guild, roomID)
-		} else { // well, good enough...
-			senderMxid = await registerUser.ensureSimJoined(message.author, roomID)
 		}
 	} else if (row && row.speedbump_webhook_id === message.webhook_id) {
 		// Handle the PluralKit public instance
 		if (row.speedbump_id === "466378653216014359") {
-			const pkMessage = await registerPkUser.fetchMessage(message.id)
-			senderMxid = await registerPkUser.syncUser(message.author, pkMessage, roomID)
+			senderMxid = await registerPkUser.syncUser(message.id, message.author, roomID, true)
 		}
 	}
 
-	const events = await messageToEvent.messageToEvent(message, guild, {}, {api})
+	const events = await messageToEvent.messageToEvent(message, guild, {}, {api, snow: discord.snow})
 	const eventIDs = []
 	if (events.length) {
-		db.prepare("REPLACE INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(message.id, message.channel_id)
+		db.prepare("INSERT OR IGNORE INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(message.id, message.channel_id)
 		if (senderMxid) api.sendTyping(roomID, false, senderMxid).catch(() => {})
 	}
 	for (const event of events) {

src/d2m/actions/set-presence.js

@@ -0,0 +1,114 @@
+// @ts-check
+
+const passthrough = require("../../passthrough")
+const {sync, select} = passthrough
+/** @type {import("../../matrix/api")} */
+const api = sync.require("../../matrix/api")
+
+/*
+	We do this in two phases for optimisation reasons.
+	Discord sends us an event when the presence *changes.*
+	We need to keep the event data in memory because we need to *repeatedly* send it to Matrix using a long-lived loop.
+
+	There are two phases to get it from Discord to Matrix.
+	The first phase stores Discord presence data in memory.
+	The second phase loops over the memory and sends it on to Matrix.
+
+	Optimisations:
+	* Presence can be deactivated per-guild in OOYE settings. If the user doesn't share any presence-enabled-guilds with us, we don't need to do anything.
+	* Presence can be sent for users without sims. In this case, they will be discarded from memory when the next loop begins.
+	* Matrix ID is cached in memory on the Presence class. The alternative to this is querying it every time we receive a presence change event in a valid guild.
+	* Presence can be sent multiple times in a row for the same user for each guild we share. The loop timer prevents these "changes" from individually reaching the homeserver.
+*/
+
+// Synapse expires each user's presence after 30 seconds and makes them offline, so we have to loop every 28 seconds and update each user again.
+const presenceLoopInterval = 28e3
+
+// Cache the list of enabled guilds rather than accessing it like multiple times per second when any user changes presence
+const guildPresenceSetting = new class {
+	/** @private @type {Set<string>} */ guilds
+	constructor() {
+		this.update()
+	}
+	update() {
+		this.guilds = new Set(select("guild_space", "guild_id", {presence: 1}).pluck().all())
+	}
+	isEnabled(guildID) {
+		return this.guilds.has(guildID)
+	}
+}
+
+class Presence extends sync.reloadClassMethods(() => Presence) {
+	/** @type {string} */ userID
+	/** @type {{presence: "online" | "offline" | "unavailable", status_msg?: string}} */ data
+	/** @private @type {?string | undefined} */ mxid
+	/** @private @type {number} */ delay = Math.random()
+
+	constructor(userID) {
+		super()
+		this.userID = userID
+	}
+
+	/**
+	 * @param {string} status status field from Discord's PRESENCE_UPDATE event
+	 */
+	setData(status) {
+		const presence =
+			( status === "online" ? "online"
+			: status === "offline" ? "offline"
+			: "unavailable")
+		this.data = {presence}
+	}
+
+	sync(presences) {
+		const mxid = this.mxid ??= select("sim", "mxid", {user_id: this.userID}).pluck().get()
+		if (!mxid) return presences.delete(this.userID)
+		// I haven't tried, but I assume Synapse explodes if you try to update too many presences at the same time.
+		// This random delay will space them out over the whole 28 second cycle.
+		setTimeout(() => {
+			api.setPresence(this.data, mxid).catch(() => {})
+		}, this.delay * presenceLoopInterval).unref()
+	}
+}
+
+const presenceTracker = new class {
+	/** @private @type {Map<string, Presence>} userID -> Presence */ presences = sync.remember(() => new Map())
+
+	constructor() {
+		sync.addTemporaryInterval(() => this.syncPresences(), presenceLoopInterval)
+	}
+
+	/**
+	 * This function is called for each Discord presence packet.
+	 * @param {string} userID Discord user ID
+	 * @param {string} guildID Discord guild ID that this presence applies to (really, the same presence applies to every single guild, but is delivered separately by Discord for some reason)
+	 * @param {string} status status field from Discord's PRESENCE_UPDATE event
+	 */
+	incomingPresence(userID, guildID, status) {
+		// stop tracking offline presence objects - they will naturally expire and fall offline on the homeserver
+		if (status === "offline") return this.presences.delete(userID)
+		// check if we care about this guild
+		if (!guildPresenceSetting.isEnabled(guildID)) return
+		// start tracking presence for user (we'll check if they have a sim in the next sync loop)
+		this.getOrCreatePresence(userID).setData(status)
+	}
+
+	/** @private */
+	getOrCreatePresence(userID) {
+		return this.presences.get(userID) || (() => {
+			const presence = new Presence(userID)
+			this.presences.set(userID, presence)
+			return presence
+		})()
+	}
+
+	/** @private */
+	syncPresences() {
+		for (const presence of this.presences.values()) {
+			presence.sync(this.presences)
+		}
+	}
+}
+
+module.exports.presenceTracker = presenceTracker
+module.exports.guildPresenceSetting = guildPresenceSetting

src/d2m/actions/update-pins.js

@@ -25,10 +25,18 @@ function convertTimestamp(timestamp) {
  * @param {number?} convertedTimestamp
  */
 async function updatePins(channelID, roomID, convertedTimestamp) {
-	const discordPins = await discord.snow.channel.getChannelPinnedMessages(channelID)
-	const pinned = pinsToList.pinsToList(discordPins)
+	try {
+		var discordPins = await discord.snow.channel.getChannelPinnedMessages(channelID)
+	} catch (e) {
+		if (e.message === `{"message": "Missing Access", "code": 50001}`) {
+			return // Discord sends channel pins update events even for channels that the bot can't view/get pins in, just ignore it
+		}
+		throw e
+	}
 
 	const kstate = await ks.roomToKState(roomID)
+	const pinned = pinsToList.pinsToList(discordPins, kstate)
+
 	const diff = ks.diffKState(kstate, {"m.room.pinned_events/": {pinned}})
 	await ks.applyKStateDiffToRoom(roomID, diff)
 

src/d2m/converters/edit-to-changes.js

@@ -22,6 +22,10 @@ function eventCanBeEdited(ev) {
 	return true
 }
 
+function eventIsText(ev) {
+	return ev.old.event_type === "m.room.message" && (ev.old.event_subtype === "m.text" || ev.old.event_subtype === "m.notice")
+}
+
 /**
  * @param {import("discord-api-types/v10").GatewayMessageCreateDispatchData} message
  * @param {import("discord-api-types/v10").APIGuild} guild
@@ -121,6 +125,20 @@ async function editToChanges(message, guild, api) {
 	unchangedEvents.push(...eventsToReplace.filter(ev => !eventCanBeEdited(ev))) // Move them from eventsToRedact to unchangedEvents.
 	eventsToReplace = eventsToReplace.filter(eventCanBeEdited)
 
+	// Now, everything in eventsToReplace has the potential to have changed, but did it actually?
+	// (Example: if a URL preview was generated or updated, the message text won't have changed.)
+	// Only way to detect this is by text content. So we'll remove text events from eventsToReplace that have the same new text as text currently in the event.
+	for (let i = eventsToReplace.length; i--;) { // move backwards through array
+		const event = eventsToReplace[i]
+		if (!eventIsText(event)) continue // not text, can't analyse
+		const oldEvent = await api.getEvent(roomID, eventsToReplace[i].old.event_id)
+		const oldEventBodyWithoutQuotedReply = oldEvent.content.body?.replace(/^(>.*\n)*\n*/sm, "")
+		if (oldEventBodyWithoutQuotedReply !== event.newInnerContent.body) continue // event changed, must replace it
+		// Move it from eventsToRedact to unchangedEvents.
+		unchangedEvents.push(...eventsToReplace.filter(ev => ev.old.event_id === event.old.event_id))
+		eventsToReplace = eventsToReplace.filter(ev => ev.old.event_id !== event.old.event_id)
+	}
+
 	// We want to maintain exactly one part = 0 and one reaction_part = 0 database row at all times.
 	// This would be disrupted if existing events that are (reaction_)part = 0 will be redacted.
 	// If that is the case, pick a different existing or newly sent event to be (reaction_)part = 0.
@@ -193,4 +211,3 @@ function makeReplacementEventContent(oldID, newFallbackContent, newInnerContent)
 }
 
 module.exports.editToChanges = editToChanges
-module.exports.makeReplacementEventContent = makeReplacementEventContent

src/d2m/converters/edit-to-changes.test.js

@@ -4,7 +4,14 @@ const data = require("../../../test/data")
 const Ty = require("../../types")
 
 test("edit2changes: edit by webhook", async t => {
-	const {senderMxid, eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.edit_by_webhook, data.guild.general, {})
+	let called = 0
+	const {senderMxid, eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.edit_by_webhook, data.guild.general, {
+		getEvent(roomID, eventID) {
+			called++
+			t.equal(eventID, "$zXSlyI78DQqQwwfPUSzZ1b-nXzbUrCDljJgnGDdoI10")
+			return {content: {body: "dummy"}}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
 	t.deepEqual(eventsToSend, [])
 	t.deepEqual(eventsToReplace, [{
@@ -28,10 +35,15 @@ test("edit2changes: edit by webhook", async t => {
 	}])
 	t.equal(senderMxid, null)
 	t.deepEqual(promotions, [])
+	t.equal(called, 1)
 })
 
 test("edit2changes: bot response", async t => {
 	const {senderMxid, eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.bot_response, data.guild.general, {
+		getEvent(roomID, eventID) {
+			t.equal(eventID, "$fdD9OZ55xg3EAsfvLZza5tMhtjUO91Wg3Otuo96TplY")
+			return {content: {body: "dummy"}}
+		},
 		async getJoinedMembers(roomID) {
 			t.equal(roomID, "!hYnGGlPHlbujVVfktC:cadence.moe")
 			return new Promise(resolve => {
@@ -123,7 +135,14 @@ test("edit2changes: add caption back to that image (due to it having a reaction,
 })
 
 test("edit2changes: stickers and attachments are not changed, only the content can be edited", async t => {
-	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments, data.guild.general, {})
+	let called = 0
+	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments, data.guild.general, {
+		getEvent(roomID, eventID) {
+			called++
+			t.equal(eventID, "$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qdFv4")
+			return {content: {body: "dummy"}}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
 	t.deepEqual(eventsToSend, [])
 	t.deepEqual(eventsToReplace, [{
@@ -145,10 +164,16 @@ test("edit2changes: stickers and attachments are not changed, only the content c
 			}
 		}
 	}])
+	t.equal(called, 1)
 })
 
 test("edit2changes: edit of reply to skull webp attachment with content", async t => {
-	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edit_of_reply_to_skull_webp_attachment_with_content, data.guild.general, {})
+	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edit_of_reply_to_skull_webp_attachment_with_content, data.guild.general, {
+		getEvent(roomID, eventID) {
+			t.equal(eventID, "$vgTKOR5ZTYNMKaS7XvgEIDaOWZtVCEyzLLi5Pc5Gz4M")
+			return {content: {body: "dummy"}}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
 	t.deepEqual(eventsToSend, [])
 	t.deepEqual(eventsToReplace, [{
@@ -177,7 +202,12 @@ test("edit2changes: edit of reply to skull webp attachment with content", async
 })
 
 test("edit2changes: edits the text event when multiple rows have part = 0 (should never happen in real life, but make sure the safety net works)", async t => {
-	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments_but_all_parts_equal_0, data.guild.general, {})
+	const {eventsToRedact, eventsToReplace, eventsToSend} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments_but_all_parts_equal_0, data.guild.general, {
+		getEvent(roomID, eventID) {
+			t.equal(eventID, "$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qd999")
+			return {content: {body: "dummy"}}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
 	t.deepEqual(eventsToSend, [])
 	t.deepEqual(eventsToReplace, [{
@@ -202,7 +232,12 @@ test("edit2changes: edits the text event when multiple rows have part = 0 (shoul
 })
 
 test("edit2changes: promotes the text event when multiple rows have part = 1 (should never happen in real life, but make sure the safety net works)", async t => {
-	const {eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments_but_all_parts_equal_1, data.guild.general, {})
+	const {eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.edited_content_with_sticker_and_attachments_but_all_parts_equal_1, data.guild.general, {
+		getEvent(roomID, eventID) {
+			t.equal(eventID, "$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qd111")
+			return {content: {body: "dummy"}}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
 	t.deepEqual(eventsToSend, [])
 	t.deepEqual(eventsToReplace, [{
@@ -279,32 +314,31 @@ test("edit2changes: generated embed", async t => {
 })
 
 test("edit2changes: generated embed on a reply", async t => {
-	const {senderMxid, eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.embed_generated_on_reply, data.guild.general, {})
+	let called = 0
+	const {senderMxid, eventsToRedact, eventsToReplace, eventsToSend, promotions} = await editToChanges(data.message_update.embed_generated_on_reply, data.guild.general, {
+		getEvent(roomID, eventID) {
+			called++
+			t.equal(eventID, "$UTqiL3Zj3FC4qldxRLggN1fhygpKl8sZ7XGY5f9MNbF")
+			return {
+				type: "m.room.message",
+				content: {
+					// Unfortunately the edited message doesn't include the message_reference field. Fine. Whatever. It looks normal if you're using a good client.
+					body: "> a Discord user: [Replied-to message content wasn't provided by Discord]"
+						+ "\n\nhttps://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM",
+					format: "org.matrix.custom.html",
+					formatted_body: "<mx-reply><blockquote><a href=\"https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM\">In reply to</a> a Discord user<br>[Replied-to message content wasn't provided by Discord]</blockquote></mx-reply><a href=\"https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM\">https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM</a>",
+					"m.mentions": {},
+					"m.relates_to": {
+						event_id: "$UTqiL3Zj3FC4qldxRLggN1fhygpKl8sZ7XGY5f9MNbF",
+						rel_type: "m.replace",
+					},
+					msgtype: "m.text",
+				}
+			}
+		}
+	})
 	t.deepEqual(eventsToRedact, [])
-	t.deepEqual(eventsToReplace, [{
-		oldID: "$UTqiL3Zj3FC4qldxRLggN1fhygpKl8sZ7XGY5f9MNbF",
-		newContent: {
-			$type: "m.room.message",
-			// Unfortunately the edited message doesn't include the message_reference field. Fine. Whatever. It looks normal if you're using a good client.
-			body: "> a Discord user: [Replied-to message content wasn't provided by Discord]"
-				+ "\n\n* https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM",
-			format: "org.matrix.custom.html",
-			formatted_body: "<mx-reply><blockquote><a href=\"https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM\">In reply to</a> a Discord user<br>[Replied-to message content wasn't provided by Discord]</blockquote></mx-reply>* <a href=\"https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM\">https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM</a>",
-			"m.mentions": {},
-			"m.new_content": {
-				body: "https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM",
-				format: "org.matrix.custom.html",
-				formatted_body: "<a href=\"https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM\">https://matrix.to/#/!BnKuBPCvyfOkhcUjEu:cadence.moe/$aLVZyiC3HlOu-prCSIaXlQl68I8leUdnPFiCwkgn6qM</a>",
-				"m.mentions": {},
-				msgtype: "m.text",
-			},
-			"m.relates_to": {
-				event_id: "$UTqiL3Zj3FC4qldxRLggN1fhygpKl8sZ7XGY5f9MNbF",
-				rel_type: "m.replace",
-			},
-			msgtype: "m.text",
-		},
-	}])
+	t.deepEqual(eventsToReplace, [])
 	t.deepEqual(eventsToSend, [{
 		$type: "m.room.message",
 		msgtype: "m.notice",
@@ -324,4 +358,5 @@ test("edit2changes: generated embed on a reply", async t => {
 		"nextEvent": true,
 	}])
 	t.equal(senderMxid, "@_ooye_cadence:cadence.moe")
+	t.equal(called, 1)
 })

src/d2m/converters/emoji-to-key.js

@@ -8,9 +8,10 @@ const file = sync.require("../../matrix/file")
 
 /**
  * @param {import("discord-api-types/v10").APIEmoji} emoji
+ * @param {string} message_id
  * @returns {Promise<string>}
  */
-async function emojiToKey(emoji) {
+async function emojiToKey(emoji, message_id) {
 	let key
 	if (emoji.id) {
 		// Custom emoji
@@ -30,7 +31,10 @@ async function emojiToKey(emoji) {
 		// Default emoji
 		const name = emoji.name
 		assert(name)
-		key = name
+		// If the reaction was used on Matrix already, it might be using a different arrangement of Variation Selector 16 characters.
+		// We'll use the same arrangement that was originally used, otherwise a duplicate of the emoji will appear as a separate reaction.
+		const originalEncoding = select("reaction", "original_encoding", {message_id, encoded_emoji: encodeURIComponent(name)}).pluck().get()
+		key = originalEncoding || name
 	}
 	return key
 }

src/d2m/converters/lottie.js

@@ -21,7 +21,7 @@ const Rlottie = (async () => {
 
 /**
  * @param {string} text
- * @returns {Promise<import("stream").Readable>}
+ * @returns {Promise<stream.Readable>}
  */
 async function convert(text) {
 	const r = await Rlottie
@@ -41,6 +41,7 @@ async function convert(text) {
 	png.data = Buffer.from(rendered)
 	// png.pack() is a bad stream and will throw away any data it sends if it's not connected to a destination straight away.
 	// We use Duplex.from to convert it into a good stream.
+	// @ts-ignore
 	return stream.Duplex.from(png.pack())
 }
 

src/d2m/converters/message-to-event.embeds.test.js

@@ -1,7 +1,7 @@
 const {test} = require("supertape")
 const {messageToEvent} = require("./message-to-event")
 const data = require("../../../test/data")
-const Ty = require("../../types")
+const {db} = require("../../passthrough")
 
 test("message2event embeds: nothing but a field", async t => {
 	const events = await messageToEvent(data.message_with_embeds.nothing_but_a_field, data.guild.general, {})
@@ -33,7 +33,9 @@ test("message2event embeds: reply with just an embed", async t => {
 		$type: "m.room.message",
 		msgtype: "m.notice",
 		"m.mentions": {},
-		body: "| ## ⏺️ dynastic (@dynastic) https://twitter.com/i/user/719631291747078145"
+		body: "> In reply to an unbridged message:"
+      	+ "\n> PokemonGod: https://twitter.com/dynastic/status/1707484191963648161"
+			+ "\n\n| ## ⏺️ dynastic (@dynastic) https://twitter.com/i/user/719631291747078145"
 			+ "\n| \n| does anyone know where to find that one video of the really mysterious yam-like object being held up to a bunch of random objects, like clocks, and they have unexplained impossible reactions to it?"
 			+ "\n| \n| ### Retweets"
 			+ "\n| 119"
@@ -41,7 +43,8 @@ test("message2event embeds: reply with just an embed", async t => {
 			+ "\n| 5581"
 			+ "\n| — Twitter",
 		format: "org.matrix.custom.html",
-		formatted_body: '<blockquote><p><strong><a href="https://twitter.com/i/user/719631291747078145">⏺️ dynastic (@dynastic)</a></strong>'
+		formatted_body: '<blockquote>In reply to an unbridged message from PokemonGod:<br><a href=\"https://twitter.com/dynastic/status/1707484191963648161\">https://twitter.com/dynastic/status/1707484191963648161</a></blockquote>'
+			+ '<blockquote><p><strong><a href="https://twitter.com/i/user/719631291747078145">⏺️ dynastic (@dynastic)</a></strong>'
 			+ '</p><p>does anyone know where to find that one video of the really mysterious yam-like object being held up to a bunch of random objects, like clocks, and they have unexplained impossible reactions to it?'
 			+ '</p><p><strong>Retweets</strong><br>119</p><p><strong>Likes</strong><br>5581</p>— Twitter</blockquote>'
 	}])
@@ -318,6 +321,25 @@ test("message2event embeds: youtube video", async t => {
 	}])
 })
 
+test("message2event embeds: tenor gif should show a video link without a provider", async t => {
+	const events = await messageToEvent(data.message_with_embeds.tenor_gif, data.guild.general, {}, {})
+	t.deepEqual(events, [{
+		$type: "m.room.message",
+		msgtype: "m.text",
+		body: "@Realdditors: get real https://tenor.com/view/get-real-gif-26176788",
+		format: "org.matrix.custom.html",
+		formatted_body: "<font color=\"#ff4500\">@Realdditors</font> get real <a href=\"https://tenor.com/view/get-real-gif-26176788\">https://tenor.com/view/get-real-gif-26176788</a>",
+		"m.mentions": {}
+	}, {
+		$type: "m.room.message",
+		msgtype: "m.notice",
+		body: "| 🎞️ https://media.tenor.com/Bz5pfRIu81oAAAPo/get-real.mp4",
+		format: "org.matrix.custom.html",
+		formatted_body: "<blockquote><p>🎞️ https://media.tenor.com/Bz5pfRIu81oAAAPo/get-real.mp4</p></blockquote>",
+		"m.mentions": {}
+	}])
+})
+
 test("message2event embeds: if discord creates an embed preview for a discord channel link, don't copy that embed", async t => {
 	const events = await messageToEvent(data.message_with_embeds.discord_server_included_punctuation_bad_discord, data.guild.general, {}, {
 		api: {
@@ -351,3 +373,16 @@ test("message2event embeds: if discord creates an embed preview for a discord ch
 		"m.mentions": {}
 	}])
 })
+
+test("message2event embeds: nothing generated if embeds are disabled in settings", async t => {
+	db.prepare("UPDATE guild_space SET url_preview = 0 WHERE guild_id = ?").run(data.guild.general.id)
+	const events = await messageToEvent(data.message_with_embeds.youtube_video, data.guild.general)
+	t.deepEqual(events, [{
+		$type: "m.room.message",
+		msgtype: "m.text",
+		body: "https://youtu.be/kDMHHw8JqLE?si=NaqNjVTtXugHeG_E\n\n\nJutomi I'm gonna make these sounds in your walls tonight",
+		format: "org.matrix.custom.html",
+		formatted_body: `<a href="https://youtu.be/kDMHHw8JqLE?si=NaqNjVTtXugHeG_E">https://youtu.be/kDMHHw8JqLE?si=NaqNjVTtXugHeG_E</a><br><br><br>Jutomi I'm gonna make these sounds in your walls tonight`,
+		"m.mentions": {}
+	}])
+})

src/d2m/converters/message-to-event.js

@@ -34,6 +34,7 @@ function getDiscordParseCallbacks(message, guild, useHTML) {
 			const mxid = select("sim", "mxid", {user_id: node.id}).pluck().get()
 			const interaction = message.interaction_metadata || message.interaction
 			const username = message.mentions.find(ment => ment.id === node.id)?.username
+				|| message.referenced_message?.mentions.find(ment => ment.id === node.id)?.username
 				|| (interaction?.user.id === node.id ? interaction.user.username : null)
 				|| node.id
 			if (mxid && useHTML) {
@@ -199,11 +200,12 @@ async function attachmentToEvent(mentions, attachment) {
 /**
  * @param {DiscordTypes.APIMessage} message
  * @param {DiscordTypes.APIGuild} guild
- * @param {{includeReplyFallback?: boolean, includeEditFallbackStar?: boolean, alwaysReturnFormattedBody?: boolean}} options default values:
+ * @param {{includeReplyFallback?: boolean, includeEditFallbackStar?: boolean, alwaysReturnFormattedBody?: boolean, scanTextForMentions?: boolean}} options default values:
  * - includeReplyFallback: true
  * - includeEditFallbackStar: false
  * - alwaysReturnFormattedBody: false - formatted_body will be skipped if it is the same as body because the message is plaintext. if you want the formatted_body to be returned anyway, for example to merge it with another message, then set this to true.
- * @param {{api: import("../../matrix/api")}} di simple-as-nails dependency injection for the matrix API
+ * - scanTextForMentions: true - needs to be set to false when converting forwarded messages etc which may be from a different channel that can't be scanned.
+ * @param {{api: import("../../matrix/api"), snow?: import("snowtransfer").SnowTransfer}} di simple-as-nails dependency injection for the matrix API
  */
 async function messageToEvent(message, guild, options = {}, di) {
 	const events = []
@@ -261,7 +263,9 @@ async function messageToEvent(message, guild, options = {}, di) {
 				- So make sure we don't do anything in this case.
 	*/
 	const mentions = {}
+	/** @type {{event_id: string, room_id: string, source: number}?} */
 	let repliedToEventRow = null
+	let repliedToUnknownEvent = false
 	let repliedToEventSenderMxid = null
 
 	if (message.mention_everyone) mentions.room = true
@@ -277,6 +281,8 @@ async function messageToEvent(message, guild, options = {}, di) {
 		const row = from("event_message").join("message_channel", "message_id").join("channel_room", "channel_id").select("event_id", "room_id", "source").and("WHERE message_id = ? AND part = 0").get(message.message_reference.message_id)
 		if (row) {
 			repliedToEventRow = row
+		} else if (message.referenced_message) {
+			repliedToUnknownEvent = true
 		}
 	} else if (dUtils.isWebhookMessage(message) && message.embeds[0]?.author?.name?.endsWith("↩️")) {
 		// It could be a PluralKit emulated reply, let's see if it has a message link
@@ -396,7 +402,7 @@ async function messageToEvent(message, guild, options = {}, di) {
 			const id = match[3]
 			const name = match[2]
 			const animated = !!match[1]
-			return emojiToKey.emojiToKey({id, name, animated}) // Register the custom emoji if needed
+			return emojiToKey.emojiToKey({id, name, animated}, message.id) // Register the custom emoji if needed
 		}))
 
 		async function transformParsedVia(parsed) {
@@ -407,8 +413,10 @@ async function messageToEvent(message, guild, options = {}, di) {
 						node.via = await getViaServersMemo(node.row.room_id)
 					}
 				}
-				if (Array.isArray(node.content)) {
-					await transformParsedVia(node.content)
+				;for (const maybeChildNodesArray of [node, node.content, node.items]) {
+					if (Array.isArray(maybeChildNodesArray)) {
+						await transformParsedVia(maybeChildNodesArray)
+					}
 				}
 			}
 			return parsed
@@ -450,7 +458,7 @@ async function messageToEvent(message, guild, options = {}, di) {
 
 		// Fallback body/formatted_body for replies
 		// This branch is optional - do NOT change anything apart from the reply fallback, since it may not be run
-		if (repliedToEventRow && options.includeReplyFallback !== false) {
+		if ((repliedToEventRow || repliedToUnknownEvent) && options.includeReplyFallback !== false) {
 			let repliedToDisplayName
 			let repliedToUserHtml
 			if (repliedToEventRow?.source === 0 && repliedToEventSenderMxid) {
@@ -472,20 +480,26 @@ async function messageToEvent(message, guild, options = {}, di) {
 			}
 			if (repliedToContent == "") repliedToContent = "[Media]"
 			else if (!repliedToContent) repliedToContent = "[Replied-to message content wasn't provided by Discord]"
-			const repliedToHtml = markdown.toHTML(repliedToContent, {
-				discordCallback: getDiscordParseCallbacks(message, guild, true)
-			})
-			const repliedToBody = markdown.toHTML(repliedToContent, {
-				discordCallback: getDiscordParseCallbacks(message, guild, false),
-				discordOnly: true,
-				escapeHTML: false,
-			})
-			html = `<mx-reply><blockquote><a href="https://matrix.to/#/${repliedToEventRow.room_id}/${repliedToEventRow.event_id}">In reply to</a> ${repliedToUserHtml}`
-				+ `<br>${repliedToHtml}</blockquote></mx-reply>`
-				+ html
-			body = (`${repliedToDisplayName}: ` // scenario 1 part B for mentions
-				+ repliedToBody).split("\n").map(line => "> " + line).join("\n")
-				+ "\n\n" + body
+			const {body: repliedToBody, html: repliedToHtml} = await transformContent(repliedToContent)
+			if (repliedToEventRow) {
+				// Generate a reply pointing to the Matrix event we found
+				html = `<mx-reply><blockquote><a href="https://matrix.to/#/${repliedToEventRow.room_id}/${repliedToEventRow.event_id}">In reply to</a> ${repliedToUserHtml}`
+					+ `<br>${repliedToHtml}</blockquote></mx-reply>`
+					+ html
+				body = (`${repliedToDisplayName}: ` // scenario 1 part B for mentions
+					+ repliedToBody).split("\n").map(line => "> " + line).join("\n")
+					+ "\n\n" + body
+			} else { // repliedToUnknownEvent
+				// This reply can't point to the Matrix event because it isn't bridged, we need to indicate this.
+				assert(message.referenced_message)
+				const dateDisplay = dUtils.howOldUnbridgedMessage(message.referenced_message.timestamp, message.timestamp)
+				html = `<blockquote>In reply to ${dateDisplay} from ${repliedToDisplayName}:`
+					+ `<br>${repliedToHtml}</blockquote>`
+					+ html
+				body = (`In reply to ${dateDisplay}:\n${repliedToDisplayName}: `
+					+ repliedToBody).split("\n").map(line => "> " + line).join("\n")
+					+ "\n\n" + body
+			}
 		}
 
 		const newTextMessageEvent = {
@@ -544,7 +558,7 @@ async function messageToEvent(message, guild, options = {}, di) {
 
 		// Forwarded content
 		// @ts-ignore
-		const forwardedEvents = await messageToEvent(message.message_snapshots[0].message, guild, {includeReplyFallback: false, includeEditFallbackStar: false, alwaysReturnFormattedBody: true}, di)
+		const forwardedEvents = await messageToEvent(message.message_snapshots[0].message, guild, {includeReplyFallback: false, includeEditFallbackStar: false, alwaysReturnFormattedBody: true, scanTextForMentions: false}, di)
 
 		// Indent
 		for (const event of forwardedEvents) {
@@ -570,7 +584,7 @@ async function messageToEvent(message, guild, options = {}, di) {
 	if (message.content) {
 		// Mentions scenario 3: scan the message content for written @mentions of matrix users. Allows for up to one space between @ and mention.
 		const matches = [...message.content.matchAll(/@ ?([a-z0-9._]+)\b/gi)]
-		if (matches.length && matches.some(m => m[1].match(/[a-z]/i) && m[1] !== "everyone" && m[1] !== "here")) {
+		if (options.scanTextForMentions !== false && matches.length && matches.some(m => m[1].match(/[a-z]/i) && m[1] !== "everyone" && m[1] !== "here")) {
 			const writtenMentionsText = matches.map(m => m[1].toLowerCase())
 			const roomID = select("channel_room", "room_id", {channel_id: message.channel_id}).pluck().get()
 			assert(roomID)
@@ -589,6 +603,49 @@ async function messageToEvent(message, guild, options = {}, di) {
 		await addTextEvent(body, html, msgtype)
 	}
 
+	// Then scheduled events
+	if (message.content && di?.snow) {
+		for (const match of [...message.content.matchAll(/discord\.gg\/([A-Za-z0-9]+)\?event=([0-9]{18,})/g)]) { // snowflake has minimum 18 because the events feature is at least that old
+			const invite = await di.snow.invite.getInvite(match[1], {guild_scheduled_event_id: match[2]})
+			const event = invite.guild_scheduled_event
+			if (!event) continue // the event ID provided was not valid
+
+			const formatter = new Intl.DateTimeFormat("en-NZ", {month: "long", day: "numeric", hour: "numeric", minute: "2-digit", timeZoneName: "shortGeneric"}) // 9 June at 3:00 pm NZT
+			const rep = new mxUtils.MatrixStringBuilder()
+
+			// Add time
+			if (event.scheduled_end_time) {
+				// @ts-ignore - no definition available for formatRange
+				rep.addParagraph(`Scheduled Event - ${formatter.formatRange(new Date(event.scheduled_start_time), new Date(event.scheduled_end_time))}`)
+			} else {
+				rep.addParagraph(`Scheduled Event - ${formatter.format(new Date(event.scheduled_start_time))}`)
+			}
+
+			// Add details
+			rep.addLine(`## ${event.name}`, tag`<strong>${event.name}</strong>`)
+			if (event.description) rep.addLine(event.description)
+
+			// Add location
+			if (event.entity_metadata?.location) {
+				rep.addParagraph(`📍 ${event.entity_metadata.location}`)
+			} else if (invite.channel?.name) {
+				const roomID = select("channel_room", "room_id", {channel_id: invite.channel.id}).pluck().get()
+				if (roomID) {
+					const via = await getViaServersMemo(roomID)
+					rep.addParagraph(`🔊 ${invite.channel.name} - https://matrix.to/#/${roomID}?${via}`, tag`🔊 ${invite.channel.name} - <a href="https://matrix.to/#/${roomID}?${via}">${invite.channel.name}</a>`)
+				} else {
+					rep.addParagraph(`🔊 ${invite.channel.name}`)
+				}
+			}
+
+			// Send like an embed
+			let {body, formatted_body: html} = rep.get()
+			body = body.split("\n").map(l => "| " + l).join("\n")
+			html = `<blockquote>${html}</blockquote>`
+			await addTextEvent(body, html, "m.notice")
+		}
+	}
+
 	// Then attachments
 	if (message.attachments) {
 		const attachmentEvents = await Promise.all(message.attachments.map(attachmentToEvent.bind(null, mentions)))
@@ -596,7 +653,12 @@ async function messageToEvent(message, guild, options = {}, di) {
 	}
 
 	// Then embeds
+	const urlPreviewEnabled = select("guild_space", "url_preview", {guild_id: guild?.id}).pluck().get() ?? 1
 	for (const embed of message.embeds || []) {
+		if (!urlPreviewEnabled && !message.author?.bot) {
+			continue // show embeds for everyone if enabled, or bot users only if disabled (bots often send content in embeds)
+		}
+
 		if (embed.type === "image") {
 			continue // Matrix's own URL previews are fine for images.
 		}
@@ -609,7 +671,7 @@ async function messageToEvent(message, guild, options = {}, di) {
 		const rep = new mxUtils.MatrixStringBuilder()
 
 		// Provider
-		if (embed.provider?.name) {
+		if (embed.provider?.name && embed.provider.name !== "Tenor") {
 			if (embed.provider.url) {
 				rep.addParagraph(`via ${embed.provider.name} ${embed.provider.url}`, tag`<sub><a href="${embed.provider.url}">${embed.provider.name}</a></sub>`)
 			} else {

src/d2m/converters/message-to-event.test.js

@@ -532,6 +532,43 @@ test("message2event: simple reply to matrix user, reply fallbacks disabled", asy
 	}])
 })
 
+test("message2event: reply to matrix user with mention", async t => {
+	const events = await messageToEvent(data.message.reply_to_matrix_user_mention, data.guild.general, {}, {
+		api: {
+			getEvent: mockGetEvent(t, "!kLRqKKUQXcibIMtOpl:cadence.moe", "$7P2O_VTQNHvavX5zNJ35DV-dbJB1Ag80tGQP_JzGdhk", {
+				type: "m.room.message",
+				content: {
+					msgtype: "m.text",
+					body: "@_ooye_extremity:cadence.moe you owe me $30",
+					format: "org.matrix.custom.html",
+					formatted_body: "<a href=\"https://matrix.to/#/@_ooye_extremity:cadence.moe\">@_ooye_extremity:cadence.moe</a> you owe me $30"
+				},
+				sender: "@cadence:cadence.moe"
+			})
+		}
+	})
+	t.deepEqual(events, [{
+		$type: "m.room.message",
+		"m.relates_to": {
+			"m.in_reply_to": {
+				event_id: "$7P2O_VTQNHvavX5zNJ35DV-dbJB1Ag80tGQP_JzGdhk"
+			}
+		},
+		"m.mentions": {
+			user_ids: [
+				"@cadence:cadence.moe"
+			]
+		},
+		msgtype: "m.text",
+		body: "> okay 🤍 yay 🤍: @extremity: you owe me $30\n\nkys",
+		format: "org.matrix.custom.html",
+		formatted_body:
+			'<mx-reply><blockquote><a href="https://matrix.to/#/!kLRqKKUQXcibIMtOpl:cadence.moe/$7P2O_VTQNHvavX5zNJ35DV-dbJB1Ag80tGQP_JzGdhk">In reply to</a> <a href="https://matrix.to/#/@cadence:cadence.moe">okay 🤍 yay 🤍</a>'
+			+ '<br><a href="https://matrix.to/#/@_ooye_extremity:cadence.moe">@extremity</a> you owe me $30</blockquote></mx-reply>'
+			+ 'kys'
+	}])
+})
+
 test("message2event: reply with a video", async t => {
 	const events = await messageToEvent(data.message.reply_with_video, data.guild.general, {
 		api: {
@@ -724,6 +761,20 @@ test("message2event: infinidoge's reply to ami's matrix smalltext singleline rep
 	}])
 })
 
+test("message2event: reply to a Discord message that wasn't bridged", async t => {
+	const events = await messageToEvent(data.message.reply_to_unknown_message, data.guild.general)
+	t.deepEqual(events, [{
+		$type: "m.room.message",
+		msgtype: "m.text",
+      body: `> In reply to a 1-day-old unbridged message:`
+			+ `\n> Occimyy: BILLY BOB THE GREAT`
+			+ `\n\nenigmatic`,
+      format: "org.matrix.custom.html",
+      formatted_body: `<blockquote>In reply to a 1-day-old unbridged message from Occimyy:<br>BILLY BOB THE GREAT</blockquote>enigmatic`,
+		"m.mentions": {}
+	}])
+})
+
 test("message2event: simple written @mention for matrix user", async t => {
 	const events = await messageToEvent(data.message.simple_written_at_mention_for_matrix, data.guild.general, {}, {
 		api: {
@@ -1134,3 +1185,191 @@ test("message2event: constructed forwarded text", async t => {
 		}
 	])
 })
+
+
+test("message2event: don't scan forwarded messages for mentions", async t => {
+	const events = await messageToEvent(data.message.forwarded_dont_scan_for_mentions, {}, {}, {})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "[🔀 Forwarded message]"
+				+ "\n» If some folks have spare bandwidth then helping out ArchiveTeam with archiving soon to be deleted research and government data might be worthwhile https://social.luca.run/@luca/113950834185678114",
+			format: "org.matrix.custom.html",
+			formatted_body: `🔀 <em>Forwarded message</em>`
+				+ `<br><blockquote>If some folks have spare bandwidth then helping out ArchiveTeam with archiving soon to be deleted research and government data might be worthwhile <a href="https://social.luca.run/@luca/113950834185678114">https://social.luca.run/@luca/113950834185678114</a></blockquote>`,
+			"m.mentions": {},
+			msgtype: "m.notice"
+		}
+	])
+})
+
+test("message2event: invite no details embed if no event", async t => {
+	const events = await messageToEvent({content: "https://discord.gg/placeholder?event=1381190945646710824"}, {}, {}, {
+		snow: {
+			invite: {
+				getInvite: async () => ({...data.invite.irl, guild_scheduled_event: null})
+			}
+		}
+	})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "https://discord.gg/placeholder?event=1381190945646710824",
+			format: "org.matrix.custom.html",
+			formatted_body: "<a href=\"https://discord.gg/placeholder?event=1381190945646710824\">https://discord.gg/placeholder?event=1381190945646710824</a>",
+			"m.mentions": {},
+			msgtype: "m.text",
+		}
+	])
+})
+
+test("message2event: irl invite event renders embed", async t => {
+	const events = await messageToEvent({content: "https://discord.gg/placeholder?event=1381190945646710824"}, {}, {}, {
+		snow: {
+			invite: {
+				getInvite: async () => data.invite.irl
+			}
+		}
+	})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "https://discord.gg/placeholder?event=1381190945646710824",
+			format: "org.matrix.custom.html",
+			formatted_body: "<a href=\"https://discord.gg/placeholder?event=1381190945646710824\">https://discord.gg/placeholder?event=1381190945646710824</a>",
+			"m.mentions": {},
+			msgtype: "m.text",
+		},
+		{
+			$type: "m.room.message",
+			msgtype: "m.notice",
+			body: `| Scheduled Event - 8 June at 10:00 pm NZT – 9 June at 12:00 am NZT`
+				+ `\n| ## forest exploration`
+				+ `\n| `
+				+ `\n| 📍 the dark forest`,
+			format: "org.matrix.custom.html",
+			formatted_body: `<blockquote><p>Scheduled Event - 8 June at 10:00 pm NZT – 9 June at 12:00 am NZT</p>`
+				+ `<strong>forest exploration</strong>`
+				+ `<p>📍 the dark forest</p></blockquote>`,
+			"m.mentions": {}
+		}
+	])
+})
+
+test("message2event: vc invite event renders embed", async t => {
+	const events = await messageToEvent({content: "https://discord.gg/placeholder?event=1381174024801095751"}, {}, {}, {
+		snow: {
+			invite: {
+				getInvite: async () => data.invite.vc
+			}
+		}
+	})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "https://discord.gg/placeholder?event=1381174024801095751",
+			format: "org.matrix.custom.html",
+			formatted_body: "<a href=\"https://discord.gg/placeholder?event=1381174024801095751\">https://discord.gg/placeholder?event=1381174024801095751</a>",
+			"m.mentions": {},
+			msgtype: "m.text",
+		},
+		{
+			$type: "m.room.message",
+			msgtype: "m.notice",
+			body: `| Scheduled Event - 9 June at 3:00 pm NZT`
+				+ `\n| ## Cooking (Netrunners)`
+				+ `\n| Short circuited brain interfaces actually just means your brain is medium rare, yum.`
+				+ `\n| `
+				+ `\n| 🔊 Cooking`,
+			format: "org.matrix.custom.html",
+			formatted_body: `<blockquote><p>Scheduled Event - 9 June at 3:00 pm NZT</p>`
+				+ `<strong>Cooking (Netrunners)</strong><br>Short circuited brain interfaces actually just means your brain is medium rare, yum.`
+				+ `<p>🔊 Cooking</p></blockquote>`,
+			"m.mentions": {}
+		}
+	])
+})
+
+test("message2event: vc invite event renders embed with room link", async t => {
+	const events = await messageToEvent({content: "https://discord.gg/placeholder?event=1381174024801095751"}, {}, {}, {
+		api: {
+			getJoinedMembers: async () => ({
+				joined: {
+					"@_ooye_bot:cadence.moe": {display_name: null, avatar_url: null},
+				}
+			})
+		},
+		snow: {
+			invite: {
+				getInvite: async () => data.invite.known_vc
+			}
+		}
+	})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "https://discord.gg/placeholder?event=1381174024801095751",
+			format: "org.matrix.custom.html",
+			formatted_body: "<a href=\"https://discord.gg/placeholder?event=1381174024801095751\">https://discord.gg/placeholder?event=1381174024801095751</a>",
+			"m.mentions": {},
+			msgtype: "m.text",
+		},
+		{
+			$type: "m.room.message",
+			msgtype: "m.notice",
+			body: `| Scheduled Event - 9 June at 3:00 pm NZT`
+				+ `\n| ## Cooking (Netrunners)`
+				+ `\n| Short circuited brain interfaces actually just means your brain is medium rare, yum.`
+				+ `\n| `
+				+ `\n| 🔊 Hey. - https://matrix.to/#/!FuDZhlOAtqswlyxzeR:cadence.moe?via=cadence.moe`,
+			format: "org.matrix.custom.html",
+			formatted_body: `<blockquote><p>Scheduled Event - 9 June at 3:00 pm NZT</p>`
+				+ `<strong>Cooking (Netrunners)</strong><br>Short circuited brain interfaces actually just means your brain is medium rare, yum.`
+				+ `<p>🔊 Hey. - <a href="https://matrix.to/#/!FuDZhlOAtqswlyxzeR:cadence.moe?via=cadence.moe">Hey.</a></p></blockquote>`,
+			"m.mentions": {}
+		}
+	])
+})
+
+test("message2event: channel links are converted even inside lists (parser post-processer descends into list items)", async t => {
+	let called = 0
+	const events = await messageToEvent({
+		content: "1. Don't be a dick"
+		+ "\n2. Follow rule number 1"
+		+ "\n3. Follow Discord TOS"
+		+ "\n4. Do **not** post NSFW content, shock content, suggestive content"
+		+ "\n5. Please keep <#176333891320283136> professional and helpful, no random off-topic joking"
+		+ "\nThis list will probably change in the future"
+	}, data.guild.general, {}, {
+		api: {
+			getJoinedMembers(roomID) {
+				called++
+				t.equal(roomID, "!qzDBLKlildpzrrOnFZ:cadence.moe")
+				return {
+					joined: {
+						"@quadradical:federated.nexus": {
+							membership: "join",
+							display_name: "quadradical"
+						}
+					}
+				}
+			}
+		}
+	})
+	t.deepEqual(events, [
+		{
+			$type: "m.room.message",
+			body: "1. Don't be a dick"
+			+ "\n2. Follow rule number 1"
+			+ "\n3. Follow Discord TOS"
+			+ "\n4. Do **not** post NSFW content, shock content, suggestive content"
+			+ "\n5. Please keep #wonderland professional and helpful, no random off-topic joking"
+			+ "\nThis list will probably change in the future",
+			format: "org.matrix.custom.html",
+			formatted_body: "<ol start=\"1\"><li>Don't be a dick</li><li>Follow rule number 1</li><li>Follow Discord TOS</li><li>Do <strong>not</strong> post NSFW content, shock content, suggestive content</li><li>Please keep <a href=\"https://matrix.to/#/!qzDBLKlildpzrrOnFZ:cadence.moe?via=cadence.moe&via=federated.nexus\">#wonderland</a> professional and helpful, no random off-topic joking</li></ol>This list will probably change in the future",
+			"m.mentions": {},
+			msgtype: "m.text"
+		}
+	])
+	t.equal(called, 1)
+})

src/d2m/converters/pins-to-list.js

@@ -4,16 +4,28 @@ const {select} = require("../../passthrough")
 
 /**
  * @param {import("discord-api-types/v10").RESTGetAPIChannelPinsResult} pins
+ * @param {{"m.room.pinned_events/"?: {pinned?: string[]}}} kstate
  */
-function pinsToList(pins) {
+function pinsToList(pins, kstate) {
+	let alreadyPinned = kstate["m.room.pinned_events/"]?.pinned || []
+
+	// If any of the already pinned messages are bridged messages then remove them from the already pinned list.
+	//   * If a bridged message is still pinned then it'll be added back in the next step.
+	//   * If a bridged message was unpinned from Discord-side then it'll be unpinned from our side due to this step.
+	//   * Matrix-only unbridged messages that are pinned will remain pinned.
+	alreadyPinned = alreadyPinned.filter(event_id => {
+		const messageID = select("event_message", "message_id", {event_id}).pluck().get()
+		return !messageID || pins.find(m => m.id === messageID) // if it is bridged then remove it from the filter
+	})
+
 	/** @type {string[]} */
 	const result = []
 	for (const message of pins) {
 		const eventID = select("event_message", "event_id", {message_id: message.id, part: 0}).pluck().get()
-		if (eventID) result.push(eventID)
+		if (eventID && !alreadyPinned.includes(eventID)) result.push(eventID)
 	}
 	result.reverse()
-	return result
+	return alreadyPinned.concat(result)
 }
 
 module.exports.pinsToList = pinsToList

src/d2m/converters/pins-to-list.test.js

@@ -3,10 +3,59 @@ const data = require("../../../test/data")
 const {pinsToList} = require("./pins-to-list")
 
 test("pins2list: converts known IDs, ignores unknown IDs", t => {
-	const result = pinsToList(data.pins.faked)
+	const result = pinsToList(data.pins.faked, {})
 	t.deepEqual(result, [
 		"$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qdFv4",
 		"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
 		"$X16nfVks1wsrhq4E9SSLiqrf2N8KD0erD0scZG7U5xg"
 	])
 })
+
+test("pins2list: already pinned duplicate items are not moved", t => {
+	const result = pinsToList(data.pins.faked, {
+		"m.room.pinned_events/": {
+			pinned: [
+				"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA"
+			]
+		}
+	})
+	t.deepEqual(result, [
+		"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
+		"$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qdFv4",
+		"$X16nfVks1wsrhq4E9SSLiqrf2N8KD0erD0scZG7U5xg"
+	])
+})
+
+test("pins2list: already pinned unknown items are not moved", t => {
+	const result = pinsToList(data.pins.faked, {
+		"m.room.pinned_events/": {
+			pinned: [
+				"$unknown1",
+				"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
+				"$unknown2"
+			]
+		}
+	})
+	t.deepEqual(result, [
+		"$unknown1",
+		"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
+		"$unknown2",
+		"$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qdFv4",
+		"$X16nfVks1wsrhq4E9SSLiqrf2N8KD0erD0scZG7U5xg"
+	])
+})
+
+test("pins2list: bridged messages can be unpinned", t => {
+	const result = pinsToList(data.pins.faked.slice(0, -2), {
+		"m.room.pinned_events/": {
+			pinned: [
+				"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
+				"$lnAF9IosAECTnlv9p2e18FG8rHn-JgYKHEHIh5qdFv4"
+			]
+		}
+	})
+	t.deepEqual(result, [
+		"$mtR8cJqM4fKno1bVsm8F4wUVqSntt2sq6jav1lyavuA",
+		"$X16nfVks1wsrhq4E9SSLiqrf2N8KD0erD0scZG7U5xg",
+	])
+})

src/d2m/converters/user-to-mxid.js

@@ -64,7 +64,7 @@ function userToSimName(user) {
 	}
 
 	// 1. Is sim user already registered?
-	const existing = select("sim", "sim_name", {user_id: user.id}).pluck().get()
+	const existing = select("sim", "user_id", {user_id: user.id}).pluck().get()
 	assert.equal(existing, null, "Shouldn't try to create a new name for an existing sim")
 
 	// 2. Register based on username (could be new or old format)

src/d2m/discord-client.js

@@ -1,10 +1,15 @@
 // @ts-check
 
-const { SnowTransfer } = require("snowtransfer")
-const { Client: CloudStorm } = require("cloudstorm")
+const DiscordTypes = require("discord-api-types/v10")
+const {Endpoints, SnowTransfer} = require("snowtransfer")
+const {reg} = require("../matrix/read-registration")
+const {Client: CloudStorm} = require("cloudstorm")
+
+// @ts-ignore
+Endpoints.BASE_HOST = reg.ooye.discord_origin || "https://discord.com"; Endpoints.CDN_URL = reg.ooye.discord_cdn_origin || "https://cdn.discordapp.com"
 
 const passthrough = require("../passthrough")
-const { sync } = passthrough
+const {sync} = passthrough
 
 /** @type {import("./discord-packets")} */
 const discordPackets = sync.require("./discord-packets")
@@ -24,7 +29,7 @@ class DiscordClient {
 			intents: [
 				"DIRECT_MESSAGES", "DIRECT_MESSAGE_REACTIONS", "DIRECT_MESSAGE_TYPING",
 				"GUILDS", "GUILD_EMOJIS_AND_STICKERS", "GUILD_MESSAGES", "GUILD_MESSAGE_REACTIONS", "GUILD_MESSAGE_TYPING", "GUILD_WEBHOOKS",
-				"MESSAGE_CONTENT"
+				"MESSAGE_CONTENT", "GUILD_PRESENCES"
 			],
 			ws: {
 				compress: false,
@@ -32,15 +37,15 @@ class DiscordClient {
 			}
 		})
 		this.ready = false
-		/** @type {import("discord-api-types/v10").APIUser} */
+		/** @type {DiscordTypes.APIUser} */
 		// @ts-ignore avoid setting as or null because we know we need to wait for ready anyways
 		this.user = null
-		/** @type {Pick<import("discord-api-types/v10").APIApplication, "id" | "flags">} */
+		/** @type {Pick<DiscordTypes.APIApplication, "id" | "flags">} */
 		// @ts-ignore
 		this.application = null
-		/** @type {Map<string, import("discord-api-types/v10").APIChannel>} */
+		/** @type {Map<string, DiscordTypes.APIChannel>} */
 		this.channels = new Map()
-		/** @type {Map<string, import("discord-api-types/v10").APIGuild>} */
+		/** @type {Map<string, DiscordTypes.APIGuild & {members: DiscordTypes.APIGuildMember[]}>} */ // we get members from the GUILD_CREATE and we do maintain it
 		this.guilds = new Map()
 		/** @type {Map<string, Array<string>>} */
 		this.guildChannelMap = new Map()
@@ -57,9 +62,6 @@ class DiscordClient {
 		addEventLogger("error", "Error")
 		addEventLogger("disconnected", "Disconnected")
 		addEventLogger("ready", "Ready")
-		this.snow.requestHandler.on("requestError", (requestID, error) => {
-			console.error("request error:", error)
-		})
 	}
 }
 

src/d2m/discord-packets.js

@@ -32,6 +32,7 @@ const utils = {
 			console.log(`Discord logged in as ${client.user.username}#${client.user.discriminator} (${client.user.id})`)
 
 		} else if (message.t === "GUILD_CREATE") {
+			message.d.members = message.d.members.filter(m => m.user.id === client.user.id) // only keep the bot's own member - it's needed to determine private channels on web
 			client.guilds.set(message.d.id, message.d)
 			const arr = []
 			client.guildChannelMap.set(message.d.id, arr)
@@ -101,6 +102,13 @@ const utils = {
 				}
 			}
 
+		} else if (message.t === "GUILD_MEMBER_UPDATE") {
+			const guild = client.guilds.get(message.d.guild_id)
+			const member = guild?.members.find(m => m.user.id === message.d.user.id)
+			if (member) { // only update existing members (i.e. the bot's own member) - don't want to inflate the cache with new irrelevant ones
+				Object.assign(member, message.d)
+			}
+
 		} else if (message.t === "THREAD_CREATE") {
 			client.channels.set(message.d.id, message.d)
 			if (message.d["guild_id"]) {
@@ -149,55 +157,17 @@ const utils = {
 		}
 
 		// Event dispatcher for OOYE bridge operations
-		if (listen === "full") {
+		if (listen === "full" && message.t) {
 			try {
-				if (message.t === "GUILD_UPDATE") {
-					await eventDispatcher.onGuildUpdate(client, message.d)
-
-				} else if (message.t === "GUILD_EMOJIS_UPDATE" || message.t === "GUILD_STICKERS_UPDATE") {
-					await eventDispatcher.onExpressionsUpdate(client, message.d)
-
-				} else if (message.t === "CHANNEL_UPDATE") {
-					await eventDispatcher.onChannelOrThreadUpdate(client, message.d, false)
-
-				} else if (message.t === "CHANNEL_PINS_UPDATE") {
-					await eventDispatcher.onChannelPinsUpdate(client, message.d)
-
-				} else if (message.t === "CHANNEL_DELETE") {
-					await eventDispatcher.onChannelDelete(client, message.d)
-
-				} else if (message.t === "THREAD_CREATE") {
-					// @ts-ignore
-					await eventDispatcher.onThreadCreate(client, message.d)
-
-				} else if (message.t === "THREAD_UPDATE") {
-					await eventDispatcher.onChannelOrThreadUpdate(client, message.d, true)
-
-				} else if (message.t === "MESSAGE_CREATE") {
-					await eventDispatcher.onMessageCreate(client, message.d)
-
-				} else if (message.t === "MESSAGE_UPDATE") {
-					await eventDispatcher.onMessageUpdate(client, message.d)
-
-				} else if (message.t === "MESSAGE_DELETE") {
-					await eventDispatcher.onMessageDelete(client, message.d)
-
-				} else if (message.t === "MESSAGE_DELETE_BULK") {
-					await eventDispatcher.onMessageDeleteBulk(client, message.d)
-
-				} else if (message.t === "TYPING_START") {
-					await eventDispatcher.onTypingStart(client, message.d)
-
-				} else if (message.t === "MESSAGE_REACTION_ADD") {
-					await eventDispatcher.onReactionAdd(client, message.d)
-
-				} else if (message.t === "MESSAGE_REACTION_REMOVE" || message.t === "MESSAGE_REACTION_REMOVE_EMOJI" || message.t === "MESSAGE_REACTION_REMOVE_ALL") {
+				if (message.t === "MESSAGE_REACTION_REMOVE" || message.t === "MESSAGE_REACTION_REMOVE_EMOJI" || message.t === "MESSAGE_REACTION_REMOVE_ALL") {
 					await eventDispatcher.onSomeReactionsRemoved(client, message.d)
 
 				} else if (message.t === "INTERACTION_CREATE") {
 					await interactions.dispatchInteraction(message.d)
-				}
 
+				} else if (message.t in eventDispatcher) {
+					await eventDispatcher[message.t](client, message.d)
+				}
 			} catch (e) {
 				// Let OOYE try to handle errors too
 				await eventDispatcher.onError(client, e, message)

src/d2m/event-dispatcher.js

@@ -2,7 +2,6 @@
 
 const assert = require("assert").strict
 const DiscordTypes = require("discord-api-types/v10")
-const util = require("util")
 const {sync, db, select, from} = require("../passthrough")
 
 /** @type {import("./actions/send-message")}) */
@@ -27,19 +26,18 @@ const updatePins = sync.require("./actions/update-pins")
 const api = sync.require("../matrix/api")
 /** @type {import("../discord/utils")} */
 const dUtils = sync.require("../discord/utils")
-/** @type {import("../m2d/converters/utils")} */
-const mxUtils = require("../m2d/converters/utils")
 /** @type {import("./actions/speedbump")} */
 const speedbump = sync.require("./actions/speedbump")
 /** @type {import("./actions/retrigger")} */
 const retrigger = sync.require("./actions/retrigger")
+/** @type {import("./actions/set-presence")} */
+const setPresence = sync.require("./actions/set-presence")
+/** @type {import("../m2d/event-dispatcher")} */
+const matrixEventDispatcher = sync.require("../m2d/event-dispatcher")
 
-/** @type {any} */ // @ts-ignore bad types from semaphore
-const Semaphore = require("@chriscdn/promise-semaphore")
+const {Semaphore} = require("@chriscdn/promise-semaphore")
 const checkMissedPinsSema = new Semaphore()
 
-let lastReportedEvent = 0
-
 // Grab Discord events we care about for the bridge, check them, and pass them on
 
 module.exports = {
@@ -49,48 +47,14 @@ module.exports = {
 	 * @param {import("cloudstorm").IGatewayMessage} gatewayMessage
 	 */
 	async onError(client, e, gatewayMessage) {
-		console.error("hit event-dispatcher's error handler with this exception:")
-		console.error(e) // TODO: also log errors into a file or into the database, maybe use a library for this? or just wing it? definitely need to be able to store the formatted event body to load back in later
-		console.error(`while handling this ${gatewayMessage.t} gateway event:`)
-		console.dir(gatewayMessage.d, {depth: null})
-
-		if (gatewayMessage.t === "TYPING_START") return
-
-		if (Date.now() - lastReportedEvent < 5000) return
-		lastReportedEvent = Date.now()
-
 		const channelID = gatewayMessage.d["channel_id"]
 		if (!channelID) return
 		const roomID = select("channel_room", "room_id", {channel_id: channelID}).pluck().get()
 		if (!roomID) return
 
-		let stackLines = null
-		if (e.stack) {
-			stackLines = e.stack.split("\n")
-			let cloudstormLine = stackLines.findIndex(l => l.includes("/node_modules/cloudstorm/"))
-			if (cloudstormLine !== -1) {
-				stackLines = stackLines.slice(0, cloudstormLine - 2)
-			}
-		}
+		if (gatewayMessage.t === "TYPING_START") return
 
-		const builder = new mxUtils.MatrixStringBuilder()
-		builder.addLine("\u26a0 Bridged event from Discord not delivered", "\u26a0 <strong>Bridged event from Discord not delivered</strong>")
-		builder.addLine(`Gateway event: ${gatewayMessage.t}`)
-		builder.addLine(e.toString())
-		if (stackLines) {
-			builder.addLine(`Error trace:\n${stackLines.join("\n")}`, `<details><summary>Error trace</summary><pre>${stackLines.join("\n")}</pre></details>`)
-		}
-		builder.addLine("", `<details><summary>Original payload</summary><pre>${util.inspect(gatewayMessage.d, false, 4, false)}</pre></details>`)
-		await api.sendEvent(roomID, "m.room.message", {
-			...builder.get(),
-			"moe.cadence.ooye.error": {
-				source: "discord",
-				payload: gatewayMessage
-			},
-			"m.mentions": {
-				user_ids: ["@cadence:cadence.moe"]
-			}
-		})
+		await matrixEventDispatcher.sendError(roomID, "Discord", gatewayMessage.t, e, gatewayMessage)
 	},
 
 	/**
@@ -105,7 +69,10 @@ module.exports = {
 		const bridgedChannels = select("channel_room", "channel_id").pluck().all()
 		const preparedExists = db.prepare("SELECT channel_id FROM message_channel WHERE channel_id = ? LIMIT 1")
 		const preparedGet = select("event_message", "event_id", {}, "WHERE message_id = ?").pluck()
-		for (const channel of guild.channels.concat(guild.threads)) {
+		/** @type {(DiscordTypes.APIChannel & {type: DiscordTypes.GuildChannelType})[]} */
+		let channels = []
+		channels = channels.concat(guild.channels, guild.threads)
+		for (const channel of channels) {
 			if (!bridgedChannels.includes(channel.id)) continue
 			if (!("last_message_id" in channel) || !channel.last_message_id) continue
 
@@ -141,13 +108,24 @@ module.exports = {
 			})
 			// console.log(`[check missed messages] got ${messages.length} messages; last message that IS bridged is at position ${latestBridgedMessageIndex} in the channel`)
 			if (latestBridgedMessageIndex === -1) latestBridgedMessageIndex = 1 // rather than crawling the ENTIRE channel history, let's just bridge the most recent 1 message to make it up to date.
+
+			// We get member data so that we can accurately update any changes to nickname or permissions that have occurred in the meantime
+			// The rate limit is lax enough that the backlog will still be pretty quick (at time of writing, 5 per 1 second per guild)
+			/** @type {Map<string, DiscordTypes.APIGuildMember | undefined>} id -> member: cache members for the run because people talk to each other */
+			const members = new Map()
+
+			// Send in order
 			for (let i = Math.min(messages.length, latestBridgedMessageIndex)-1; i >= 0; i--) {
-				const simulatedGatewayDispatchData = {
+				const message = messages[i]
+
+				if (!members.has(message.author.id)) members.set(message.author.id, await client.snow.guild.getGuildMember(guild.id, message.author.id).catch(() => undefined))
+				await module.exports.MESSAGE_CREATE(client, {
 					guild_id: guild.id,
+					member: members.get(message.author.id),
+					// @ts-ignore
 					backfill: true,
-					...messages[i]
-				}
-				await module.exports.onMessageCreate(client, simulatedGatewayDispatchData)
+					...message
+				})
 			}
 		}
 	},
@@ -194,7 +172,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.APIThreadChannel} thread
 	 */
-	async onThreadCreate(client, thread) {
+	async THREAD_CREATE(client, thread) {
 		const channelID = thread.parent_id || undefined
 		const parentRoomID = select("channel_room", "room_id", {channel_id: channelID}).pluck().get()
 		if (!parentRoomID) return // Not interested in a thread if we aren't interested in its wider channel (won't autocreate)
@@ -206,7 +184,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayGuildUpdateDispatchData} guild
 	 */
-	async onGuildUpdate(client, guild) {
+	async GUILD_UPDATE(client, guild) {
 		const spaceID = select("guild_space", "space_id", {guild_id: guild.id}).pluck().get()
 		if (!spaceID) return
 		await createSpace.syncSpace(guild)
@@ -215,19 +193,26 @@ module.exports = {
 	/**
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayChannelUpdateDispatchData} channelOrThread
-	 * @param {boolean} isThread
 	 */
-	async onChannelOrThreadUpdate(client, channelOrThread, isThread) {
+	async CHANNEL_UPDATE(client, channelOrThread) {
 		const roomID = select("channel_room", "room_id", {channel_id: channelOrThread.id}).pluck().get()
 		if (!roomID) return // No target room to update the data on
 		await createRoom.syncRoom(channelOrThread.id)
 	},
 
+	/**
+	 * @param {import("./discord-client")} client
+	 * @param {DiscordTypes.GatewayChannelUpdateDispatchData} thread
+	 */
+	async THREAD_UPDATE(client, thread) {
+		await module.exports.CHANNEL_UPDATE(client, thread)
+	},
+
 	/**
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayChannelPinsUpdateDispatchData} data
 	 */
-	async onChannelPinsUpdate(client, data) {
+	async CHANNEL_PINS_UPDATE(client, data) {
 		const roomID = select("channel_room", "room_id", {channel_id: data.channel_id}).pluck().get()
 		if (!roomID) return // No target room to update pins in
 		const convertedTimestamp = updatePins.convertTimestamp(data.last_pin_timestamp)
@@ -238,7 +223,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayChannelDeleteDispatchData} channel
 	 */
-	async onChannelDelete(client, channel) {
+	async CHANNEL_DELETE(client, channel) {
 		const guildID = channel["guild_id"]
 		if (!guildID) return // channel must have been a DM channel or something
 		const roomID = select("channel_room", "room_id", {channel_id: channel.id}).pluck().get()
@@ -251,7 +236,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayMessageCreateDispatchData} message
 	 */
-	async onMessageCreate(client, message) {
+	async MESSAGE_CREATE(client, message) {
 		if (message.author.username === "Deleted User") return // Nothing we can do for deleted users.
 		const channel = client.channels.get(message.channel_id)
 		if (!channel || !("guild_id" in channel) || !channel.guild_id) return // Nothing we can do in direct messages.
@@ -281,7 +266,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayMessageUpdateDispatchData} data
 	 */
-	async onMessageUpdate(client, data) {
+	async MESSAGE_UPDATE(client, data) {
 		// Based on looking at data they've sent me over the gateway, this is the best way to check for meaningful changes.
 		// If the message content is a string then it includes all interesting fields and is meaningful.
 		// Otherwise, if there are embeds, then the system generated URL preview embeds.
@@ -299,7 +284,7 @@ module.exports = {
 		if (affected) return
 
 		// Check that the sending-to room exists, and deal with Eventual Consistency(TM)
-		if (retrigger.eventNotFoundThenRetrigger(data.id, module.exports.onMessageUpdate, client, data)) return
+		if (retrigger.eventNotFoundThenRetrigger(data.id, module.exports.MESSAGE_UPDATE, client, data)) return
 
 		/** @type {DiscordTypes.GatewayMessageCreateDispatchData} */
 		// @ts-ignore
@@ -317,7 +302,7 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayMessageReactionAddDispatchData} data
 	 */
-	async onReactionAdd(client, data) {
+	async MESSAGE_REACTION_ADD(client, data) {
 		if (data.user_id === client.user.id) return // m2d reactions are added by the discord bot user - do not reflect them back to matrix.
 		await addReaction.addReaction(data)
 	},
@@ -334,25 +319,25 @@ module.exports = {
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayMessageDeleteDispatchData} data
 	 */
-	async onMessageDelete(client, data) {
+	async MESSAGE_DELETE(client, data) {
 		speedbump.onMessageDelete(data.id)
-		if (retrigger.eventNotFoundThenRetrigger(data.id, module.exports.onMessageDelete, client, data)) return
+		if (retrigger.eventNotFoundThenRetrigger(data.id, module.exports.MESSAGE_DELETE, client, data)) return
 		await deleteMessage.deleteMessage(data)
 	},
 
-		/**
+	/**
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayMessageDeleteBulkDispatchData} data
 	 */
-		async onMessageDeleteBulk(client, data) {
-			await deleteMessage.deleteMessageBulk(data)
-		},
+	async MESSAGE_DELETE_BULK(client, data) {
+		await deleteMessage.deleteMessageBulk(data)
+	},
 
 	/**
 	 * @param {import("./discord-client")} client
 	 * @param {DiscordTypes.GatewayTypingStartDispatchData} data
 	 */
-	async onTypingStart(client, data) {
+	async TYPING_START(client, data) {
 		const roomID = select("channel_room", "room_id", {channel_id: data.channel_id}).pluck().get()
 		if (!roomID) return
 		const mxid = from("sim").join("sim_member", "mxid").where({user_id: data.user_id, room_id: roomID}).pluck("mxid").get()
@@ -365,9 +350,27 @@ module.exports = {
 
 	/**
 	 * @param {import("./discord-client")} client
-	 * @param {DiscordTypes.GatewayGuildEmojisUpdateDispatchData | DiscordTypes.GatewayGuildStickersUpdateDispatchData} data
+	 * @param {DiscordTypes.GatewayGuildEmojisUpdateDispatchData} data
 	 */
-	async onExpressionsUpdate(client, data) {
+	async GUILD_EMOJIS_UPDATE(client, data) {
 		await createSpace.syncSpaceExpressions(data, false)
+	},
+
+	/**
+	 * @param {import("./discord-client")} client
+	 * @param {DiscordTypes.GatewayGuildStickersUpdateDispatchData} data
+	 */
+	async GUILD_STICKERS_UPDATE(client, data) {
+		await createSpace.syncSpaceExpressions(data, false)
+	},
+
+	/**
+	 * @param {import("./discord-client")} client
+	 * @param {DiscordTypes.GatewayPresenceUpdateDispatchData} data
+	 */
+	PRESENCE_UPDATE(client, data) {
+		const status = data.status
+		if (!status) return
+		setPresence.presenceTracker.incomingPresence(data.user.id, data.guild_id, status)
 	}
 }

src/db/migrate.js

@@ -6,7 +6,8 @@ const {join} = require("path")
 async function migrate(db) {
 	let files = fs.readdirSync(join(__dirname, "migrations"))
 	files = files.sort()
-	db.prepare("CREATE TABLE IF NOT EXISTS migration (filename TEXT NOT NULL)").run()
+	db.prepare("CREATE TABLE IF NOT EXISTS migration (filename TEXT NOT NULL, PRIMARY KEY (filename)) WITHOUT ROWID").run()
+	/** @type {string} */
 	let progress = db.prepare("SELECT * FROM migration").pluck().get()
 	if (!progress) {
 		progress = ""
@@ -37,6 +38,8 @@ async function migrate(db) {
 	if (migrationRan) {
 		console.log("Database migrations all done.")
 	}
+
+	db.pragma("foreign_keys = on")
 }
 
 module.exports.migrate = migrate

src/db/migrations/0016-foreign-keys.sql

@@ -0,0 +1,147 @@
+-- /docs/foreign-keys.md
+
+-- 2
+BEGIN TRANSACTION;
+
+-- *** channel_room ***
+
+-- 4
+-- adding UNIQUE to room_id here will auto-generate the usable index we wanted
+CREATE TABLE "new_channel_room" (
+	"channel_id"	TEXT NOT NULL,
+	"room_id"	TEXT NOT NULL UNIQUE,
+	"name"	TEXT NOT NULL,
+	"nick"	TEXT,
+	"thread_parent"	TEXT,
+	"custom_avatar"	TEXT,
+	"last_bridged_pin_timestamp"	INTEGER,
+	"speedbump_id"	TEXT,
+	"speedbump_checked"	INTEGER,
+	"speedbump_webhook_id"	TEXT,
+	"guild_id"	TEXT,
+	PRIMARY KEY("channel_id"),
+	FOREIGN KEY("guild_id") REFERENCES "guild_active"("guild_id") ON DELETE CASCADE
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_channel_room (channel_id, room_id, name, nick, thread_parent, custom_avatar, last_bridged_pin_timestamp, speedbump_id, speedbump_checked, speedbump_webhook_id, guild_id) SELECT channel_id, room_id, name, nick, thread_parent, custom_avatar, last_bridged_pin_timestamp, speedbump_id, speedbump_checked, speedbump_webhook_id, guild_id FROM channel_room;
+-- 6
+DROP TABLE channel_room;
+-- 7
+ALTER TABLE new_channel_room RENAME TO channel_room;
+
+-- *** message_channel ***
+
+-- 4
+CREATE TABLE "new_message_channel" (
+	"message_id"	TEXT NOT NULL,
+	"channel_id"	TEXT NOT NULL,
+	PRIMARY KEY("message_id"),
+	FOREIGN KEY("channel_id") REFERENCES "channel_room"("channel_id") ON DELETE CASCADE
+) WITHOUT ROWID;
+-- 5
+-- don't copy any orphaned messages
+INSERT INTO new_message_channel (message_id, channel_id) SELECT message_id, channel_id FROM message_channel WHERE channel_id IN (SELECT channel_id FROM channel_room);
+-- 6
+DROP TABLE message_channel;
+-- 7
+ALTER TABLE new_message_channel RENAME TO message_channel;
+
+-- *** event_message ***
+
+-- clean up any orphaned events
+DELETE FROM event_message WHERE message_id NOT IN (SELECT message_id FROM message_channel);
+-- 4
+CREATE TABLE "new_event_message" (
+	"event_id"	TEXT NOT NULL,
+	"event_type"	TEXT,
+	"event_subtype"	TEXT,
+	"message_id"	TEXT NOT NULL,
+	"part"	INTEGER NOT NULL,
+	"reaction_part"	INTEGER NOT NULL,
+	"source"	INTEGER NOT NULL,
+	PRIMARY KEY("message_id","event_id"),
+	FOREIGN KEY("message_id") REFERENCES "message_channel"("message_id") ON DELETE CASCADE
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_event_message (event_id, event_type, event_subtype, message_id, part, reaction_part, source) SELECT event_id, event_type, event_subtype, message_id, part, reaction_part, source FROM event_message;
+-- 6
+DROP TABLE event_message;
+-- 7
+ALTER TABLE new_event_message RENAME TO event_message;
+
+-- *** guild_space ***
+
+-- 4
+CREATE TABLE "new_guild_space" (
+	"guild_id"	TEXT NOT NULL,
+	"space_id"	TEXT NOT NULL,
+	"privacy_level"	INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY("guild_id"),
+	FOREIGN KEY("guild_id") REFERENCES "guild_active"("guild_id") ON DELETE CASCADE
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_guild_space (guild_id, space_id, privacy_level) SELECT guild_id, space_id, privacy_level FROM guild_space;
+-- 6
+DROP TABLE guild_space;
+-- 7
+ALTER TABLE new_guild_space RENAME TO guild_space;
+
+-- *** reaction ***
+
+-- 4
+CREATE TABLE "new_reaction" (
+	"hashed_event_id"	INTEGER NOT NULL,
+	"message_id"	TEXT NOT NULL,
+	"encoded_emoji"	TEXT NOT NULL,
+	PRIMARY KEY("hashed_event_id"),
+	FOREIGN KEY("message_id") REFERENCES "message_channel"("message_id") ON DELETE CASCADE
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_reaction (hashed_event_id, message_id, encoded_emoji) SELECT hashed_event_id, message_id, encoded_emoji FROM reaction WHERE message_id IN (SELECT message_id FROM message_channel);
+-- 6
+DROP TABLE reaction;
+-- 7
+ALTER TABLE new_reaction RENAME TO reaction;
+
+-- *** webhook ***
+
+-- 4
+-- using RESTRICT instead of CASCADE as a reminder that the webhooks also need to be deleted using the Discord API, it can't just be entirely automatic
+CREATE TABLE "new_webhook" (
+	"channel_id"	TEXT NOT NULL,
+	"webhook_id"	TEXT NOT NULL,
+	"webhook_token"	TEXT NOT NULL,
+	PRIMARY KEY("channel_id"),
+	FOREIGN KEY("channel_id") REFERENCES "channel_room"("channel_id") ON DELETE RESTRICT
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_webhook (channel_id, webhook_id, webhook_token) SELECT channel_id, webhook_id, webhook_token FROM webhook WHERE channel_id IN (SELECT channel_id FROM channel_room);
+-- 6
+DROP TABLE webhook;
+-- 7
+ALTER TABLE new_webhook RENAME TO webhook;
+
+-- *** sim ***
+
+-- 4
+-- while we're at it, rebuild this table to give it WITHOUT ROWID, remove UNIQUE, and replace the localpart column with username. no foreign keys needed
+CREATE TABLE "new_sim" (
+	"user_id"	TEXT NOT NULL,
+	"username"	TEXT NOT NULL,
+	"sim_name"	TEXT NOT NULL,
+	"mxid"	TEXT NOT NULL,
+	PRIMARY KEY("user_id")
+) WITHOUT ROWID;
+-- 5
+INSERT INTO new_sim (user_id, username, sim_name, mxid) SELECT user_id, sim_name, sim_name, mxid FROM sim;
+-- 6
+DROP TABLE sim;
+-- 7
+ALTER TABLE new_sim RENAME TO sim;
+
+-- *** end ***
+
+-- 10
+PRAGMA foreign_key_check;
+-- 11
+COMMIT;

src/db/migrations/0017-analyze.sql

@@ -0,0 +1,225 @@
+-- https://www.sqlite.org/lang_analyze.html
+
+BEGIN TRANSACTION;
+
+ANALYZE sqlite_schema;
+
+DELETE FROM "sqlite_stat1";
+INSERT INTO "sqlite_stat1" ("tbl","idx","stat") VALUES ('sim','sim','625 1'),
+ ('reaction','reaction','3242 1'),
+ ('channel_room','channel_room','389 1'),
+ ('channel_room','sqlite_autoindex_channel_room_1','389 1'),
+ ('media_proxy','media_proxy','5068 1'),
+ ('sim_proxy','sim_proxy','36 1'),
+ ('webhook','webhook','155 1'),
+ ('member_cache','member_cache','784 3 1'),
+ ('member_power','member_power','1 1 1'),
+ ('file','file','21862 1'),
+ ('message_channel','message_channel','366884 1'),
+ ('lottie','lottie','19 1'),
+ ('event_message','event_message','382920 1 1'),
+ ('migration',NULL,'1'),
+ ('sim_member','sim_member','2871 7 1'),
+ ('guild_space','guild_space','32 1'),
+ ('guild_active','guild_active','34 1'),
+ ('emoji','emoji','2563 1'),
+ ('auto_emoji','auto_emoji','3 1');
+
+DELETE FROM "sqlite_stat4";
+INSERT INTO "sqlite_stat4" ("tbl","idx","neq","nlt","ndlt","sample") VALUES ('sim','sim','1','69','69',X'0231313137363631373038303932333039353039'),
+ ('sim','sim','1','139','139',X'0231313530383936363934333439373931323332'),
+ ('sim','sim','1','209','209',X'0231323231383737363334373737323139303732'),
+ ('sim','sim','1','279','279',X'0231333039313431353735353334313136383636'),
+ ('sim','sim','1','349','349',X'0231333935343433383235363034313635363434'),
+ ('sim','sim','1','419','419',X'0231353335363239373830383338353134373030'),
+ ('sim','sim','1','489','489',X'0231363930333339333730353930363636383034'),
+ ('sim','sim','1','559','559',X'0231383535353736303637393137323137383133'),
+ ('reaction','reaction','1','360','360',X'020699d5faceefb5fb4f'),
+ ('reaction','reaction','1','721','721',X'0206b61095e98b6b2fb1'),
+ ('reaction','reaction','1','1082','1082',X'0206d1dcb418603a5eaa'),
+ ('reaction','reaction','1','1443','1443',X'0206ef9fc42b9df746ad'),
+ ('reaction','reaction','1','1804','1804',X'02060f38c1f98f130605'),
+ ('reaction','reaction','1','2165','2165',X'02062b53df6dab7b1067'),
+ ('reaction','reaction','1','2526','2526',X'020645dd7e7f60c4aac7'),
+ ('reaction','reaction','1','2887','2887',X'0206658d2fe735805979'),
+ ('channel_room','channel_room','1','43','43',X'023331313434393131333330393139333231363330'),
+ ('channel_room','channel_room','1','87','87',X'023331313835343033343830303934303335393738'),
+ ('channel_room','channel_room','1','131','131',X'023331323139353036353836343139303638393839'),
+ ('channel_room','channel_room','1','175','175',X'023331323336353538333034323331303334393630'),
+ ('channel_room','channel_room','1','219','219',X'023331323933373932323135333930323234343235'),
+ ('channel_room','channel_room','1','263','263',X'023331333333323139363936393333323038303937'),
+ ('channel_room','channel_room','1','307','307',X'0231343835363635393733363433333738363938'),
+ ('channel_room','channel_room','1','351','351',X'0231373039303432313039353632323234363731'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','6 6','6 6',X'034b3321416a6c4c49464e6248646474424a6d4d73503a636164656e63652e6d6f6531313531333434383735363139343833373233'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','34 34','34 34',X'034b3321474b4a63424a6b527a47634e4855686c50613a636164656e63652e6d6f6531303237393433323532323237323630343637'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','43 43','43 43',X'034b3121484b50534d62736d694673506d6268414f513a636164656e63652e6d6f65313931343837343839393433343034353434'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','58 58','58 58',X'034b33214a4479425a685545706874784f6e6f6569513a636164656e63652e6d6f6531323937323836373434353633313236333532'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','87 87','87 87',X'034b33214e544d724e686e715271695755654d494d523a636164656e63652e6d6f6531323235323434353738393939373031353536'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','108 108','108 108',X'034b332151444e44796656674e7657565345656876713a636164656e63652e6d6f6531313432333134303935353535363435343830'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','131 131','131 131',X'034b3121544171536b575752654b43506f584c6a75483a636164656e63652e6d6f65383737303730363531343733363631393532'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','175 175','175 175',X'034b3321594249486864714e697255585941587845563a636164656e63652e6d6f6531323335303831373939353936373639333730'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','177 177','177 177',X'034b3321594b46454e79716667696951686956496b533a636164656e63652e6d6f6531323934363237303431343530333933373034'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','186 186','186 186',X'034b3321596f54644f55766a53765349767266716c653a636164656e63652e6d6f6531323734313936373733383435333430323933'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','202 202','202 202',X'034b3121625877616673695372655647676470535a463a636164656e63652e6d6f65373339303137363739373936343336393932'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','208 208','208 208',X'034b3321634a4b6843764943795377717a47634551423a636164656e63652e6d6f6531323732363632303331323238373331343834'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','219 219','219 219',X'034b3121656455786a56647a6755765844554951434b3a636164656e63652e6d6f65343937313631333530393334353630373738'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','242 242','242 242',X'034b31216a4d746e6e6f51414e4278466a486458494d3a636164656e63652e6d6f65373634353135323932303539323731313939'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','263 263','263 263',X'034b31216c7a776870666a5a6e59797468656a7453483a636164656e63652e6d6f65383838343831373132383438343030343534'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','264 264','264 264',X'034b33216d454c5846716a426958726d7558796943723a636164656e63652e6d6f6531313936393134373631303430393234373432'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','268 268','268 268',X'034b33216d557765577571546761574a767769576a653a636164656e63652e6d6f6531323936373131393236333032333830303834'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','291 291','291 291',X'034b3321704761494e45534643587a634e42497a724e3a636164656e63652e6d6f6531303237343531333333333533313532353232'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','306 306','306 306',X'034b3321717646656248564f4b6876454e54494563763a636164656e63652e6d6f6531323737373238383139323232303230313436'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','307 307','307 307',X'034b332171767370666d716f476449634a66794c506c3a636164656e63652e6d6f6531323936393138333638393539343633343735'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','351 351','351 351',X'034b3321774e7a7741724a47796f4c5168426f544e4b3a636164656e63652e6d6f6531303238303436373930333435333739383930'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','368 368','368 368',X'034b3321794e6d504c7765654a69756570725a677a733a636164656e63652e6d6f6531323531393631373233373731313632363234'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','376 376','376 376',X'034b3121796a4879795772466f704c66646878564e423a636164656e63652e6d6f65333336313537353037303734353233313336'),
+ ('channel_room','sqlite_autoindex_channel_room_1','1 1','379 379','379 379',X'034b31217a4c4f6b62766b44587551465948594555673a636164656e63652e6d6f65393933383838313433373030393330363331'),
+ ('media_proxy','media_proxy','1','563','563',X'02069e6054680b610946'),
+ ('media_proxy','media_proxy','1','1127','1127',X'0206bb489b717c9320e4'),
+ ('media_proxy','media_proxy','1','1691','1691',X'0206d75f602775b7a27c'),
+ ('media_proxy','media_proxy','1','2255','2255',X'0206f2c705ddca4e2b14'),
+ ('media_proxy','media_proxy','1','2819','2819',X'02061060db7a5151967b'),
+ ('media_proxy','media_proxy','1','3383','3383',X'02062cc47366f7550d22'),
+ ('media_proxy','media_proxy','1','3947','3947',X'020647d275ec0d781fc7'),
+ ('media_proxy','media_proxy','1','4511','4511',X'02066402024a7ea38249'),
+ ('sim_proxy','sim_proxy','1','4','4',X'025531316564343731342d636635652d346333372d393331382d376136353266383732636634'),
+ ('sim_proxy','sim_proxy','1','9','9',X'025533346636333932642d323263372d346337382d393063372d326536323734313535613266'),
+ ('sim_proxy','sim_proxy','1','14','14',X'025535396662363131392d626133392d346565382d393738612d386432376366303631393633'),
+ ('sim_proxy','sim_proxy','1','19','19',X'025539373066366536332d646234632d346531342d383063362d336639343938643961363665'),
+ ('sim_proxy','sim_proxy','1','24','24',X'025561636231613335642d313336662d343362332d626365622d326566646634616265306436'),
+ ('sim_proxy','sim_proxy','1','29','29',X'025563316635623735392d336136342d343633342d623634632d643461656436316539656632'),
+ ('sim_proxy','sim_proxy','1','34','34',X'025566323230373135632d633436332d343532622d626233612d373662646662306365353537'),
+ ('webhook','webhook','1','17','17',X'023331313532383834313435343038373230393936'),
+ ('webhook','webhook','1','35','35',X'023331313939303936333434333830343631313138'),
+ ('webhook','webhook','1','53','53',X'023331323331383036353337373032353736313938'),
+ ('webhook','webhook','1','71','71',X'023331323933373836383939343238383036363536'),
+ ('webhook','webhook','1','89','89',X'023331333132363031353130363535353537373132'),
+ ('webhook','webhook','1','107','107',X'0231323937323734313733303636333133373339'),
+ ('webhook','webhook','1','125','125',X'0231353239313736313536333938363832313137'),
+ ('webhook','webhook','1','143','143',X'0231363837303238373334333232313437333434'),
+ ('member_cache','member_cache','4 1','73 74','48 74',X'034b3921496f4866536e67625a6762747061747a494e3a636164656e63652e6d6f65406875636b6c65746f6e3a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','2 1','86 87','57 87',X'034b2d214b5169714663546e764f6f4f424475746a7a3a636164656e63652e6d6f6540726e6c3a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','4 1','101 104','68 104',X'034b43214e446249714e704a795076664b526e4e63723a636164656e63652e6d6f6540776f756e6465645f77617272696f723a6d61747269782e6f7267'),
+ ('member_cache','member_cache','4 1','110 113','73 113',X'034b3b214f485844457370624d485348716c4445614f3a636164656e63652e6d6f6540717561647261646963616c3a6d61747269782e6f7267'),
+ ('member_cache','member_cache','5 1','171 175','111 175',X'034b3b215450616f6a5454444446444847776c7276743a636164656e63652e6d6f6540766962656973766572796f3a6d61747269782e6f7267'),
+ ('member_cache','member_cache','39 1','180 208','116 208',X'034b4d2154716c79516d69667847556767456d64424e3a636164656e63652e6d6f6540726f626c6b796f6772653a6372616674696e67636f6d72616465732e6e6574'),
+ ('member_cache','member_cache','4 1','231 231','126 231',X'034b3b2156624f77675559777146614e4c5345644e413a636164656e63652e6d6f654061666c6f7765723a73796e646963617465642e676179'),
+ ('member_cache','member_cache','9 1','262 263','141 263',X'034b3b21594b46454e79716667696951686956496b533a636164656e63652e6d6f654062656e6d61633a636861742e62656e6d61632e78797a'),
+ ('member_cache','member_cache','3 1','283 283','149 283',X'034b35215a615a4d78456f52724d6d4e49554d79446c3a636164656e63652e6d6f6540636164656e63653a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','88 1','307 351','166 351',X'034b3b2163427874565278446c5a765356684a58564b3a636164656e63652e6d6f65406a61736b617274683a736c656570696e672e746f776e'),
+ ('member_cache','member_cache','11 1','408 415','177 415',X'034b5121654856655270706e6c6f57587177704a6e553a636164656e63652e6d6f65406a61636b736f6e6368656e3636363a6a61636b736f6e6368656e3636362e636f6d'),
+ ('member_cache','member_cache','7 1','423 424','181 424',X'034b4b2165724f7079584e465a486a48724568784e583a636164656e63652e6d6f6540616d796973636f6f6c7a3a6d61747269782e6174697573616d792e636f6d'),
+ ('member_cache','member_cache','96 1','436 439','187 439',X'034b4b21676865544b5a7451666c444e7070684c49673a636164656e63652e6d6f6540616c65783a73706163652e67616d65727374617665726e2e6f6e6c696e65'),
+ ('member_cache','member_cache','96 1','436 527','187 527',X'034b3121676865544b5a7451666c444e7070684c49673a636164656e63652e6d6f654078796c6f626f6c3a616d6265722e74656c'),
+ ('member_cache','member_cache','10 1','546 555','197 555',X'0351312169537958674e7851634575586f587073536e3a707573737468656361742e6f726740797562697175653a6e6f70652e63686174'),
+ ('member_cache','member_cache','13 1','594 601','224 601',X'034b2b216c7570486a715444537a774f744d59476d493a636164656e63652e6d6f6540656c6c69753a68617368692e7265'),
+ ('member_cache','member_cache','2 1','614 615','229 615',X'034b2f216d584978494644676c4861734e53427371773a636164656e63652e6d6f654077696e673a666561746865722e6f6e6c'),
+ ('member_cache','member_cache','4 1','616 619','230 619',X'034b2f216d616767455367755a427147425a74536e723a636164656e63652e6d6f654077696e673a666561746865722e6f6e6c'),
+ ('member_cache','member_cache','4 1','659 660','259 660',X'034b332172454f73706e5971644f414c4149466e69563a636164656e63652e6d6f6540656c797369613a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','4 1','699 701','284 701',X'034b3521766e717a56767678534a586c5a504f5276533a636164656e63652e6d6f6540636164656e63653a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','1 1','703 703','285 703',X'034b3521767165714c474851616842464a56566779483a636164656e63652e6d6f654063696465723a6361746769726c2e636c6f7564'),
+ ('member_cache','member_cache','4 1','705 705','287 705',X'034b35217750454472596b77497a6f744e66706e57503a636164656e63652e6d6f6540636164656e63653a636164656e63652e6d6f65'),
+ ('member_cache','member_cache','35 1','709 709','288 709',X'034b2d2177574f667376757356486f4e4e567242585a3a636164656e63652e6d6f654061613a6361747669626572732e6d65'),
+ ('member_cache','member_cache','14 1','747 749','291 749',X'034b3721776c534544496a44676c486d42474b7254703a636164656e63652e6d6f654062616461746e616d65733a62616461742e646576'),
+ ('member_power','member_power','1 1','0 0','0 0',X'03350f40636164656e63653a636164656e63652e6d6f652a'),
+ ('file','file','1','2429','2429',X'03815f68747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f313039393033313838373530303033343038382f313333313336303134333238333036303833372f50584c5f32303235303132315f3230323934323137372e6a7067'),
+ ('file','file','1','4859','4859',X'03817568747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f313134353832313533383832323637323436362f313330323232393131303834373936373331352f53637265656e73686f745f32303234313130325f3034313332365f5265646469742e6a7067'),
+ ('file','file','1','7289','7289',X'03815968747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f313231393439383932363436363636323433302f313239373634363930353038353636313234362f494d475f32303234313032305f3135323230302e6a7067'),
+ ('file','file','1','9719','9719',X'03814168747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f3135393136353731343139343735393638302f313236383537333933363531343337313635392f494d475f353433362e6a7067'),
+ ('file','file','1','12149','12149',X'03813b68747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f3236363736373539303634313233383032372f313237323430333931313939383730313630392f696d6167652e706e67'),
+ ('file','file','1','14579','14579',X'03816b68747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f3539383730363933323736303434343936392f313237373532343532343330383632373437372f45585445524e414c5f454449545f323032345f4d5f64726166745f322e646f6378'),
+ ('file','file','1','17009','17009',X'03815768747470733a2f2f63646e2e646973636f72646170702e636f6d2f6174746163686d656e74732f3635353231363137333639363238363734362f313333333630333132383634313036303938342f323032352d30312d32375f31372e30312e31352e706e67'),
+ ('file','file','1','19439','19439',X'027f68747470733a2f2f63646e2e646973636f72646170702e636f6d2f656d6f6a69732f313230323936303730343936313931323836322e706e67'),
+ ('message_channel','message_channel','1','40764','40764',X'023331313630333434353733303030383232383735'),
+ ('message_channel','message_channel','1','81529','81529',X'023331313830323437393130343238393837343333'),
+ ('message_channel','message_channel','1','122294','122294',X'023331313938303533383732383337363131363230'),
+ ('message_channel','message_channel','1','163059','163059',X'023331323237373739373330333839303738303537'),
+ ('message_channel','message_channel','1','203824','203824',X'023331323437303438333039303031313538373137'),
+ ('message_channel','message_channel','1','244589','244589',X'023331323635363939353734333034303830303034'),
+ ('message_channel','message_channel','1','285354','285354',X'023331323835343637363238323434313732383234'),
+ ('message_channel','message_channel','1','326119','326119',X'023331333038373932333935313031333736353732'),
+ ('lottie','lottie','1','2','2',X'0231373439303534363630373639323138363331'),
+ ('lottie','lottie','1','5','5',X'0231373534313037353339323030363731373635'),
+ ('lottie','lottie','1','8','8',X'0231373936313430363338303933343433303932'),
+ ('lottie','lottie','1','11','11',X'0231373936313431373032363935343835353030'),
+ ('lottie','lottie','1','14','14',X'0231383136303837373932323931323832393434'),
+ ('lottie','lottie','1','17','17',X'0231383233393736313032393736323930383636'),
+ ('event_message','event_message','11 1','14788 14796','14356 14796',X'03336531313532303033373639343137303839303434246d44714a474b3530424a715170394273684e534d64365f3768494354776e70362d793130786d6669766563'),
+ ('event_message','event_message','11 1','33806 33815','32914 33815',X'033365313135373630333638373035373836363736322459526f7139484b376e55677a397668796f6e3053424a49497978497a5750734e4f6e39756f765644664d45'),
+ ('event_message','event_message','1 1','42546 42546','41335 42546',X'03336531313630363236383737353835363938393237245033436e4f6d6a35462d6939454e4e79586b70796f5679306b74324a654464764276326e746d33566a4355'),
+ ('event_message','event_message','2 1','85093 85093','81999 85093',X'0333653131383039393939363531323930343831303424496748666562784533746e623047412d534f7176594a354e4e55385735706c68523159676854636a554734'),
+ ('event_message','event_message','11 1','116157 116165','111525 116165',X'03336531313933363837333730363137333237363536245a32305734766c737079566e387a6e2d526a6f64694a51745f5a7851644f5f33744e415169453755356477'),
+ ('event_message','event_message','1 1','127640 127640','122328 127640',X'0333653132303132353637313733373633303332323624702d626f415672476a4b45327a7158664f3738387a5a597a376a42624648717431334d386464705467476f'),
+ ('event_message','event_message','16 1','140270 140281','134379 140281',X'03336531323039333735363534373138343830343235246c374a4f7543526b7756306d627a69356e5843496b4538416a6951374f67473455456c2d7053445a516649'),
+ ('event_message','event_message','11 1','162065 162071','154933 162071',X'0333653132323434383135393033313937373537373424674c77513179796e4b6d5859496b5a597a4a55627a66557a55552d714c4b5f524f454e4250325f6e44766b'),
+ ('event_message','event_message','1 1','170187 170187','162530 170187',X'0333653132323937373533363839343532373038333424656c6c76416a544a5847627936767249767470677a555572787231716f5a75536e50525f474b4e35455945'),
+ ('event_message','event_message','11 1','178736 178736','170762 178736',X'03336531323333353238303533323338333337353537242d39304668552d36455373594b6435484d7237666d6a414a5f6a576149616e356c4776384e655436564959'),
+ ('event_message','event_message','10 1','180317 180325','172228 180325',X'03336531323334363237303030393333383130323637246a4679416449665a4f54432d2d735971715472473735374c7a50504f34386439657963485477686d797751'),
+ ('event_message','event_message','1 1','212734 212734','203278 212734',X'03336531323438303131373930303633373637363734246557493950456f4d576b614b4d33416a7030782d6d455f6c4b4a6c495f6d6d44686f6379464b5170534f59'),
+ ('event_message','event_message','11 1','228100 228103','217856 228103',X'033365313235333734353736363733373132313336322447326a7a746e5977716a3676304a7a4168576e30725950596470667a5f496f76426771574a4876434c516b'),
+ ('event_message','event_message','11 1','240172 240181','229123 240181',X'033365313235393239383538323233303630313735382472635679454c5a76647453547a37366f3669434a4f614a316a756f683835535778494d546c5072517a676f'),
+ ('event_message','event_message','10 1','240259 240264','229132 240264',X'0333653132353933303030343035333535373235323024535849376a465f696e71424d714c4f564b347659746852644b31724747502d435a5f355a354f6b774e3751'),
+ ('event_message','event_message','2 1','255280 255281','243230 255281',X'03336531323636373837343338353137343234313738246e6e4e576f526a54495757723441463770625454746b7a73784c4d336b312d6d6645665031496b43586e59'),
+ ('event_message','event_message','1 1','297828 297828','283436 297828',X'0333653132383637303937353334363834323032313024544342465970356f39767a2d4f6b2d33654f4e433772426d354966615934476b48536e58445257474f4767'),
+ ('event_message','event_message','1 1','340375 340375','323489 340375',X'033365313330393336363936343530353934303030392431664536386d2d50546e786d5474345458584c35754847594139353779396c76582d50797150496d395f30'),
+ ('event_message','event_message','11 1','343791 343799','326730 343799',X'03336531333131353731333337393331363537323737246731767241315269725951592d3933304f6973587142372d6961686e34684e492d6462374952714176616b'),
+ ('event_message','event_message','10 1','363207 363216','344868 363216',X'0333653133323434393732323633393438393834393524784f78636e4749364269545941734d7a4f7557316f526e68356b675378544e30466442386a3037695f4f67'),
+ ('event_message','event_message','10 1','363219 363219','344871 363219',X'033365313332343439373532363733323039393732352432586f7938567937643843375a47767472657966326b4c4f4159397546386b4755364c3642435f446b6d77'),
+ ('event_message','event_message','10 1','363340 363343','344918 363343',X'03336531333234353037313732333634373530393830244f46645731396649534176706d34646c36367a2d5543337236432d436354506e34752d63444f7036733345'),
+ ('event_message','event_message','11 1','369452 369456','350712 369456',X'0333653133323736353831313733343439323336383024574d774330644574417277375f4562554a534465546532577174506d3747584347774570646c4f79326d30'),
+ ('event_message','event_message','10 1','372353 372356','353425 372356',X'0333653133323933313737353039313234353035373224366259364d313472667163486d5854476349716d4a4366467471796839794472375a6a487463715a6d4f34'),
+ ('sim_member','sim_member','225 1','0 12','0 12',X'034b4721414956694e775a64636b4652764c4f4567433a636164656e63652e6d6f65405f6f6f79655f616b6972615f6e6965723a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','2 1','319 319','14 319',X'034b43214456706f6e54524d56456570486378744c423a636164656e63652e6d6f65405f6f6f79655f656e746f6c6f6d613a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','68 1','391 440','26 440',X'034b4921457a54624a496c496d45534f746b4e644e4a3a636164656e63652e6d6f65405f6f6f79655f73617475726461797465643a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','8 1','638 639','59 639',X'034b3f21497a4f675169446e757346516977796d614c3a636164656e63652e6d6f65405f6f6f79655f636f6f6b69653a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','31 1','743 771','86 771',X'034b49214d5071594e414a62576b72474f544a7461703a636164656e63652e6d6f65405f6f6f79655f746865666f6f6c323239343a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','26 1','774 787','87 787',X'034b49214d687950614b4250506f496c7365794d6d743a636164656e63652e6d6f65405f6f6f79655f6b79757567727970686f6e3a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','27 1','877 881','104 881',X'034b45215063734371724f466a48476f41424270414c3a636164656e63652e6d6f65405f6f6f79655f62696c6c795f626f623a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','16 1','956 959','117 959',X'034b43215158526f4a777a63506d5047546d454b454d3a636164656e63652e6d6f65405f6f6f79655f626f7472616334723a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','32 1','997 1012','121 1012',X'034b3f215170676c734e587a4c7751594d4c6c734f503a636164656e63652e6d6f65405f6f6f79655f6a75746f6d693a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','7 1','1274 1279','157 1279',X'034b4121554d6f6e68556765644d47585a78466658753a636164656e63652e6d6f65405f6f6f79655f6d696e696d75733a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','27 1','1415 1439','188 1439',X'034b3d21595868717249786d586e47736961796a59783a636164656e63652e6d6f65405f6f6f79655f73746161663a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','16 1','1597 1599','217 1599',X'034b512163466a4479477274466d48796d794c6652453a636164656e63652e6d6f65405f6f6f79655f626f6a61636b5f686f7273656d616e3a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','27 1','1758 1761','248 1761',X'034b3d2168665a74624d656f5355564e424850736a743a636164656e63652e6d6f65405f6f6f79655f617a7572653a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','25 1','1865 1886','270 1886',X'034b3f216b4c52714b4b555158636962494d744f706c3a636164656e63652e6d6f65405f6f6f79655f7361796f72693a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','19 1','1918 1919','276 1919',X'034b3d216b68497350756c465369736d43646c596e493a636164656e63652e6d6f65405f6f6f79655f617a7572653a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','33 1','1986 2015','286 2015',X'034b3d216d5451744d736a534c4f646c576f7265594d3a636164656e63652e6d6f65405f6f6f79655f73746161663a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','37 1','2027 2028','289 2028',X'034b4d216d616767455367755a427147425a74536e723a636164656e63652e6d6f65405f6f6f79655f2e7265616c2e706572736f6e2e3a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','28 1','2117 2130','297 2130',X'034b3f216e4e595a794b6f4e70797859417a50466f733a636164656e63652e6d6f65405f6f6f79655f6a75746f6d693a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','20 1','2230 2239','310 2239',X'034b41217046504c7270594879487a784e4c69594b413a636164656e63652e6d6f65405f6f6f79655f686578676f61743a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','30 1','2381 2398','332 2398',X'034b3b21717a44626c4b6c69444c577a52524f6e465a3a636164656e63652e6d6f65405f6f6f79655f6d6e696b3a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','38 1','2490 2518','344 2518',X'034b3d2173445250714549546e4f4e57474176496b423a636164656e63652e6d6f65405f6f6f79655f727974686d3a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','11 1','2555 2559','358 2559',X'034b4321746751436d526b426e6474516362687150583a636164656e63652e6d6f65405f6f6f79655f6a6f7365707065793a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','47 1','2633 2666','377 2666',X'034b4b217750454472596b77497a6f744e66706e57503a636164656e63652e6d6f65405f6f6f79655f6e61706f6c656f6e333038393a636164656e63652e6d6f65'),
+ ('sim_member','sim_member','52 1','2817 2837','414 2837',X'034b43217a66654e574d744b4f764f48766f727979563a636164656e63652e6d6f65405f6f6f79655f696e736f676e69613a636164656e63652e6d6f65'),
+ ('guild_space','guild_space','1','3','3',X'0231313132373630363639313738323431303234'),
+ ('guild_space','guild_space','1','7','7',X'023331313534383638343234373234343633363837'),
+ ('guild_space','guild_space','1','11','11',X'023331323139303338323637383430393235383138'),
+ ('guild_space','guild_space','1','15','15',X'023331323839353939363232353930383930313335'),
+ ('guild_space','guild_space','1','19','19',X'0231323733383737363437323234393935383431'),
+ ('guild_space','guild_space','1','23','23',X'0231353239313736313536333938363832313135'),
+ ('guild_space','guild_space','1','27','27',X'0231373535303134333534373334313533383138'),
+ ('guild_space','guild_space','1','31','31',X'0231393933383838313432343535323130303834'),
+ ('guild_active','guild_active','1','3','3',X'0231313132373630363639313738323431303234'),
+ ('guild_active','guild_active','1','7','7',X'023331313534383638343234373234343633363837'),
+ ('guild_active','guild_active','1','11','11',X'023331323139303338323637383430393235383138'),
+ ('guild_active','guild_active','1','15','15',X'023331323839353939363232353930383930313335'),
+ ('guild_active','guild_active','1','19','19',X'023331333333323139363936393333323038303934'),
+ ('guild_active','guild_active','1','23','23',X'0231343735353939303338353336373434393630'),
+ ('guild_active','guild_active','1','27','27',X'022f3636313932393535373737343836383438'),
+ ('guild_active','guild_active','1','31','31',X'0231383737303635303431393930353136373637'),
+ ('emoji','emoji','1','284','284',X'023331313132323031303430303637303339333332'),
+ ('emoji','emoji','1','569','569',X'023331323334393032303131393436383630363638'),
+ ('emoji','emoji','1','854','854',X'0231323735313734373438353034313935303732'),
+ ('emoji','emoji','1','1139','1139',X'0231333837343730383630303134393737303235'),
+ ('emoji','emoji','1','1424','1424',X'0231353435363639393734303130383838313932'),
+ ('emoji','emoji','1','1709','1709',X'0231363339313034333030393139393437323634'),
+ ('emoji','emoji','1','1994','1994',X'0231373532363932363230303638373832313231'),
+ ('emoji','emoji','1','2279','2279',X'0231383935343737353331303434363138323430'),
+ ('auto_emoji','auto_emoji','1','0','0',X'02114c31'),
+ ('auto_emoji','auto_emoji','1','1','1',X'02114c32'),
+ ('auto_emoji','auto_emoji','1','2','2',X'020f5f');
+
+ANALYZE sqlite_schema;
+
+COMMIT;

src/db/migrations/0018-add-custom-topic-to-channel-room.sql

@@ -0,0 +1,5 @@
+BEGIN TRANSACTION;
+
+ALTER TABLE channel_room ADD COLUMN custom_topic INTEGER DEFAULT 0;
+
+COMMIT;

src/db/migrations/0019-add-invite.sql

@@ -0,0 +1,13 @@
+BEGIN TRANSACTION;
+
+CREATE TABLE "invite" (
+	"mxid"	TEXT NOT NULL,
+	"room_id"	TEXT NOT NULL,
+	"type"	TEXT,
+	"name"	TEXT,
+	"topic" TEXT,
+	"avatar"	TEXT,
+	PRIMARY KEY("mxid","room_id")
+) WITHOUT ROWID;
+
+COMMIT;

src/db/migrations/0020-add-presence-to-guild-space.sql

@@ -0,0 +1,5 @@
+BEGIN TRANSACTION;
+
+ALTER TABLE guild_space ADD COLUMN presence INTEGER NOT NULL DEFAULT 1;
+
+COMMIT;

src/db/migrations/0021-add-url-preview-to-guild-space.sql

@@ -0,0 +1,5 @@
+BEGIN TRANSACTION;
+
+ALTER TABLE guild_space ADD COLUMN url_preview INTEGER NOT NULL DEFAULT 1;
+
+COMMIT;

src/db/migrations/0022-auto-emoji-without-guild.sql

@@ -0,0 +1,11 @@
+BEGIN TRANSACTION;
+
+DROP TABLE auto_emoji;
+
+CREATE TABLE auto_emoji (
+	name	TEXT NOT NULL,
+	emoji_id	TEXT NOT NULL,
+	PRIMARY KEY (name)
+) WITHOUT ROWID;
+
+COMMIT;

src/db/migrations/0023-add-original-encoding-to-reaction.sql

@@ -0,0 +1,5 @@
+BEGIN TRANSACTION;
+
+ALTER TABLE reaction ADD COLUMN original_encoding TEXT;
+
+COMMIT;

src/db/migrations/0024-add-direct.sql

@@ -0,0 +1,9 @@
+BEGIN TRANSACTION;
+
+CREATE TABLE direct (
+	mxid TEXT NOT NULL,
+	room_id TEXT NOT NULL,
+	PRIMARY KEY (mxid)
+) WITHOUT ROWID;
+
+COMMIT;

src/db/orm-defs.d.ts

@@ -1,4 +1,9 @@
 export type Models = {
+	auto_emoji: {
+		name: string
+		emoji_id: string
+	}
+
 	channel_room: {
 		channel_id: string
 		room_id: string
@@ -10,6 +15,20 @@ export type Models = {
 		speedbump_id: string | null
 		speedbump_webhook_id: string | null
 		speedbump_checked: number | null
+		guild_id: string | null
+		custom_topic: number
+	}
+
+	direct: {
+		mxid: string
+		room_id: string
+	}
+
+	emoji: {
+		emoji_id: string
+		name: string
+		animated: number
+		mxc_url: string
 	}
 
 	event_message: {
@@ -31,6 +50,8 @@ export type Models = {
 		guild_id: string
 		space_id: string
 		privacy_level: number
+		presence: 0 | 1
+		url_preview: 0 | 1
 	}
 
 	guild_active: {
@@ -38,11 +59,23 @@ export type Models = {
 		autocreate: 0 | 1
 	}
 
+	invite: {
+		mxid: string
+		room_id: string
+		type: string | null
+		name: string | null
+		avatar: string | null
+	}
+
 	lottie: {
 		sticker_id: string
 		mxc_url: string
 	}
 
+	media_proxy: {
+		permitted_hash: number
+	}
+
 	member_cache: {
 		room_id: string
 		mxid: string
@@ -87,27 +120,11 @@ export type Models = {
 		webhook_token: string
 	}
 
-	emoji: {
-		emoji_id: string
-		name: string
-		animated: number
-		mxc_url: string
-	}
-
 	reaction: {
 		hashed_event_id: number
 		message_id: string
 		encoded_emoji: string
-	}
-
-	auto_emoji: {
-		name: string
-		emoji_id: string
-		guild_id: string
-	}
-
-	media_proxy: {
-		permitted_hash: number
+		original_encoding: string | null
 	}
 }
 

src/db/orm.test.js

@@ -6,17 +6,17 @@ const data = require("../../test/data")
 const {db, select, from} = require("../passthrough")
 
 test("orm: select: get works", t => {
-	const row = select("guild_space", "guild_id", {}, "WHERE space_id = ?").get("!jjWAGMeQdNrVZSSfvz:cadence.moe")
+	const row = select("guild_space", "guild_id", {}, "WHERE space_id = ?").get("!jjmvBegULiLucuWEHU:cadence.moe")
 	t.equal(row?.guild_id, data.guild.general.id)
 })
 
 test("orm: from: get works", t => {
-	const row = from("guild_space").select("guild_id").and("WHERE space_id = ?").get("!jjWAGMeQdNrVZSSfvz:cadence.moe")
+	const row = from("guild_space").select("guild_id").and("WHERE space_id = ?").get("!jjmvBegULiLucuWEHU:cadence.moe")
 	t.equal(row?.guild_id, data.guild.general.id)
 })
 
 test("orm: select: get pluck works", t => {
-	const guildID = select("guild_space", "guild_id", {}, "WHERE space_id = ?").pluck().get("!jjWAGMeQdNrVZSSfvz:cadence.moe")
+	const guildID = select("guild_space", "guild_id", {}, "WHERE space_id = ?").pluck().get("!jjmvBegULiLucuWEHU:cadence.moe")
 	t.equal(guildID, data.guild.general.id)
 })
 
@@ -36,7 +36,7 @@ test("orm: select: in array works", t => {
 })
 
 test("orm: from: get pluck works", t => {
-	const guildID = from("guild_space").pluck("guild_id").and("WHERE space_id = ?").get("!jjWAGMeQdNrVZSSfvz:cadence.moe")
+	const guildID = from("guild_space").pluck("guild_id").and("WHERE space_id = ?").get("!jjmvBegULiLucuWEHU:cadence.moe")
 	t.equal(guildID, data.guild.general.id)
 })
 

src/discord/interactions/invite.test.js

@@ -59,7 +59,7 @@ test("invite: checks if guild exists", async t => { // it might not exist if the
 })
 
 test("invite: checks if channel exists or is autocreatable", async t => {
-	db.prepare("UPDATE guild_active SET autocreate = 0").run()
+	db.prepare("UPDATE guild_active SET autocreate = 0 WHERE guild_id = '112760669178241024'").run()
 	const msgs = await fromAsync(_interact({
 		data: {
 			options: [{
@@ -72,7 +72,7 @@ test("invite: checks if channel exists or is autocreatable", async t => {
 		guild_id: "112760669178241024"
 	}, {}))
 	t.equal(msgs[0].createInteractionResponse.data.content, "This channel isn't bridged, so you can't invite Matrix users yet. Try turning on automatic room-creation or link a Matrix room in the website.")
-	db.prepare("UPDATE guild_active SET autocreate = 1").run()
+	db.prepare("UPDATE guild_active SET autocreate = 1 WHERE guild_id = '112760669178241024'").run()
 })
 
 test("invite: checks if user is already invited to space", async t => {
@@ -91,7 +91,7 @@ test("invite: checks if user is already invited to space", async t => {
 		api: {
 			getStateEvent: async (roomID, type, stateKey) => {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(type, "m.room.member")
 				t.equal(stateKey, "@cadence:cadence.moe")
 				return {
@@ -121,14 +121,14 @@ test("invite: invites if user is not in space", async t => {
 		api: {
 			getStateEvent: async (roomID, type, stateKey) => {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(type, "m.room.member")
 				t.equal(stateKey, "@cadence:cadence.moe")
 				throw new MatrixServerError("State event doesn't exist or something")
 			},
 			inviteToRoom: async (roomID, mxid) => {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(mxid, "@cadence:cadence.moe")
 			}
 		}
@@ -155,7 +155,7 @@ test("invite: prompts to invite to room (if never joined)", async t => {
 				called++
 				t.equal(type, "m.room.member")
 				t.equal(stateKey, "@cadence:cadence.moe")
-				if (roomID === "!jjWAGMeQdNrVZSSfvz:cadence.moe") { // space ID
+				if (roomID === "!jjmvBegULiLucuWEHU:cadence.moe") { // space ID
 					return {
 						displayname: "cadence",
 						membership: "join"
@@ -188,7 +188,7 @@ test("invite: prompts to invite to room (if left)", async t => {
 				called++
 				t.equal(type, "m.room.member")
 				t.equal(stateKey, "@cadence:cadence.moe")
-				if (roomID === "!jjWAGMeQdNrVZSSfvz:cadence.moe") { // space ID
+				if (roomID === "!jjmvBegULiLucuWEHU:cadence.moe") { // space ID
 					return {
 						displayname: "cadence",
 						membership: "join"

src/discord/interactions/permissions.test.js

@@ -57,7 +57,7 @@ test("permissions: reports permissions of selected matrix user (implicit default
 			},
 			async getStateEvent(roomID, type, key) {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(type, "m.room.power_levels")
 				t.equal(key, "")
 				return {
@@ -91,7 +91,7 @@ test("permissions: reports permissions of selected matrix user (moderator)", asy
 			},
 			async getStateEvent(roomID, type, key) {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(type, "m.room.power_levels")
 				t.equal(key, "")
 				return {
@@ -127,7 +127,7 @@ test("permissions: reports permissions of selected matrix user (admin)", async t
 			},
 			async getStateEvent(roomID, type, key) {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(type, "m.room.power_levels")
 				t.equal(key, "")
 				return {
@@ -159,7 +159,7 @@ test("permissions: can update user to moderator", async t => {
 		api: {
 			async setUserPowerCascade(roomID, mxid, power) {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(mxid, "@cadence:cadence.moe")
 				t.equal(power, 50)
 			}
@@ -186,7 +186,7 @@ test("permissions: can update user to default", async t => {
 		api: {
 			async setUserPowerCascade(roomID, mxid, power) {
 				called++
-				t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe") // space ID
+				t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe") // space ID
 				t.equal(mxid, "@cadence:cadence.moe")
 				t.equal(power, 0)
 			}

src/discord/register-interactions.js

@@ -64,7 +64,9 @@ discord.snow.interaction.bulkOverwriteApplicationCommands(id, [{
 			}]
 		}
 	]
-}])
+}]).catch(e => {
+	console.error(e)
+})
 
 async function dispatchInteraction(interaction) {
 	const interactionId = interaction.data.custom_id || interaction.data.name

src/discord/utils.js

@@ -136,6 +136,24 @@ function getPublicUrlForCdn(url) {
 	return `${reg.ooye.bridge_origin}/download/discord${match[1]}/${match[2]}/${match[3]}/${match[4]}`
 }
 
+/**
+ * @param {string} oldTimestamp
+ * @param {string} newTimestamp
+ * @returns {string} "a x-day-old unbridged message"
+ */
+function howOldUnbridgedMessage(oldTimestamp, newTimestamp) {
+	const dateDifference = new Date(newTimestamp).getTime() - new Date(oldTimestamp).getTime()
+	const oneHour = 60 * 60 * 1000
+	if (dateDifference < oneHour) {
+		return "an unbridged message"
+	} else if (dateDifference < 25 * oneHour) {
+		var dateDisplay = `a ${Math.floor(dateDifference / oneHour)}-hour-old unbridged message`
+	} else {
+		var dateDisplay = `a ${Math.round(dateDifference / (24 * oneHour))}-day-old unbridged message`
+	}
+	return dateDisplay
+}
+
 module.exports.getPermissions = getPermissions
 module.exports.hasPermission = hasPermission
 module.exports.hasSomePermissions = hasSomePermissions
@@ -145,3 +163,4 @@ module.exports.isEphemeralMessage = isEphemeralMessage
 module.exports.snowflakeToTimestampExact = snowflakeToTimestampExact
 module.exports.timestampToSnowflakeInexact = timestampToSnowflakeInexact
 module.exports.getPublicUrlForCdn = getPublicUrlForCdn
+module.exports.howOldUnbridgedMessage = howOldUnbridgedMessage

src/m2d/actions/add-reaction.js

@@ -20,12 +20,25 @@ async function addReaction(event) {
 	if (!messageID) return // Nothing can be done if the parent message was never bridged.
 
 	const key = event.content["m.relates_to"].key
-	const discordPreferredEncoding = emoji.encodeEmoji(key, event.content.shortcode)
+	const discordPreferredEncoding = await emoji.encodeEmoji(key, event.content.shortcode)
 	if (!discordPreferredEncoding) return
 
-	await discord.snow.channel.createReaction(channelID, messageID, discordPreferredEncoding) // acting as the discord bot itself
+	try {
+		await discord.snow.channel.createReaction(channelID, messageID, discordPreferredEncoding) // acting as the discord bot itself
+	} catch (e) {
+		if (e.message?.includes("Maximum number of reactions reached")) {
+			// we'll silence this particular error to avoid spamming the chat
+			// not adding it to the database otherwise a m->d removal would try calling the API
+			return
+		}
+		if (e.message?.includes("Unknown Emoji")) {
+			// happens if a matrix user tries to add on to a super reaction
+			return
+		}
+		throw e
+	}
 
-	db.prepare("REPLACE INTO reaction (hashed_event_id, message_id, encoded_emoji) VALUES (?, ?, ?)").run(utils.getEventIDHash(event.event_id), messageID, discordPreferredEncoding)
+	db.prepare("REPLACE INTO reaction (hashed_event_id, message_id, encoded_emoji, original_encoding) VALUES (?, ?, ?, ?)").run(utils.getEventIDHash(event.event_id), messageID, discordPreferredEncoding, key)
 }
 
 module.exports.addReaction = addReaction

src/m2d/actions/channel-webhook.js

@@ -2,7 +2,7 @@
 
 const assert = require("assert").strict
 const DiscordTypes = require("discord-api-types/v10")
-const {Readable} = require("stream")
+const stream = require("stream")
 const passthrough = require("../../passthrough")
 const {discord, db, select} = passthrough
 
@@ -57,7 +57,7 @@ async function withWebhook(channelID, callback) {
 
 /**
  * @param {string} channelID
- * @param {DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | Readable}[]}} data
+ * @param {DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | stream.Readable}[]}} data
  * @param {string} [threadID]
  */
 async function sendMessageWithWebhook(channelID, data, threadID) {
@@ -70,7 +70,7 @@ async function sendMessageWithWebhook(channelID, data, threadID) {
 /**
  * @param {string} channelID
  * @param {string} messageID
- * @param {DiscordTypes.RESTPatchAPIWebhookWithTokenMessageJSONBody & {files?: {name: string, file: Buffer | Readable}[]}} data
+ * @param {DiscordTypes.RESTPatchAPIWebhookWithTokenMessageJSONBody & {files?: {name: string, file: Buffer | stream.Readable}[]}} data
  * @param {string} [threadID]
  */
 async function editMessageWithWebhook(channelID, messageID, data, threadID) {

src/m2d/actions/emoji-sheet.js

@@ -1,9 +1,6 @@
 // @ts-check
 
-const assert = require("assert")
-const fetch = require("node-fetch").default
-
-const utils = require("../converters/utils")
+const stream = require("stream")
 const {sync} = require("../../passthrough")
 
 /** @type {import("../converters/emoji-sheet")} */
@@ -18,16 +15,15 @@ const api = sync.require("../../matrix/api")
  */
 async function getAndConvertEmoji(mxc) {
 	const abortController = new AbortController()
-	/** @type {import("node-fetch").Response} */
 	// If it turns out to be a GIF, we want to abandon the connection without downloading the whole thing.
 	// If we were using connection pooling, we would be forced to download the entire GIF.
 	// So we set no agent to ensure we are not connection pooling.
-	// @ts-ignore the signal is slightly different from the type it wants (still works fine)
-	const res = await api.getMedia(mxc, {agent: false, signal: abortController.signal})
-	return emojiSheetConverter.convertImageStream(res.body, () => {
+	const res = await api.getMedia(mxc, {signal: abortController.signal})
+	const readable = stream.Readable.fromWeb(res.body)
+	return emojiSheetConverter.convertImageStream(readable, () => {
 		abortController.abort()
-		res.body.pause()
-		res.body.emit("end")
+		readable.emit("end")
+		readable.on("error", () => {}) // DOMException [AbortError]: This operation was aborted
 	})
 }
 

src/m2d/actions/redact.js

@@ -13,7 +13,6 @@ const utils = sync.require("../converters/utils")
  */
 async function deleteMessage(event) {
 	const rows = from("event_message").join("message_channel", "message_id").select("channel_id", "message_id").where({event_id: event.redacts}).all()
-	db.prepare("DELETE FROM event_message WHERE event_id = ?").run(event.redacts)
 	for (const row of rows) {
 		db.prepare("DELETE FROM message_channel WHERE message_id = ?").run(row.message_id)
 		await discord.snow.channel.deleteMessage(row.channel_id, row.message_id, event.content.reason)

src/m2d/actions/send-event.js

@@ -2,10 +2,9 @@
 
 const Ty = require("../../types")
 const DiscordTypes = require("discord-api-types/v10")
-const {Readable} = require("stream")
+const stream = require("stream")
 const assert = require("assert").strict
 const crypto = require("crypto")
-const fetch = require("node-fetch").default
 const passthrough = require("../../passthrough")
 const {sync, discord, db, select} = passthrough
 
@@ -23,8 +22,8 @@ const editMessage = sync.require("../../d2m/actions/edit-message")
 const emojiSheet = sync.require("../actions/emoji-sheet")
 
 /**
- * @param {DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | Readable}[], pendingFiles?: ({name: string, mxc: string} | {name: string, mxc: string, key: string, iv: string} | {name: string, buffer: Buffer | Readable})[]}} message
- * @returns {Promise<DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | Readable}[]}>}
+ * @param {DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | stream.Readable}[], pendingFiles?: ({name: string, mxc: string} | {name: string, mxc: string, key: string, iv: string} | {name: string, buffer: Buffer | stream.Readable})[]}} message
+ * @returns {Promise<DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | stream.Readable}[]}>}
  */
 async function resolvePendingFiles(message) {
 	if (!message.pendingFiles) return message
@@ -38,16 +37,14 @@ async function resolvePendingFiles(message) {
 		if ("key" in p) {
 			// Encrypted file
 			const d = crypto.createDecipheriv("aes-256-ctr", Buffer.from(p.key, "base64url"), Buffer.from(p.iv, "base64url"))
-			// @ts-ignore
-			await api.getMedia(p.mxc).then(res => res.body.pipe(d))
+			await api.getMedia(p.mxc).then(res => stream.Readable.fromWeb(res.body).pipe(d))
 			return {
 				name: p.name,
 				file: d
 			}
 		} else {
 			// Unencrypted file
-			/** @type {Readable} */ // @ts-ignore
-			const body = await api.getMedia(p.mxc).then(res => res.body)
+			const body = await api.getMedia(p.mxc).then(res => stream.Readable.fromWeb(res.body))
 			return {
 				name: p.name,
 				file: body
@@ -65,7 +62,7 @@ async function resolvePendingFiles(message) {
 /** @param {Ty.Event.Outer_M_Room_Message | Ty.Event.Outer_M_Room_Message_File | Ty.Event.Outer_M_Sticker} event */
 async function sendEvent(event) {
 	const row = select("channel_room", ["channel_id", "thread_parent"], {room_id: event.room_id}).get()
-	if (!row) return // allow the bot to exist in unbridged rooms, just don't do anything with it
+	if (!row) return [] // allow the bot to exist in unbridged rooms, just don't do anything with it
 	let channelID = row.channel_id
 	let threadID = undefined
 	if (row.thread_parent) {
@@ -102,14 +99,13 @@ async function sendEvent(event) {
 
 	for (const id of messagesToDelete) {
 		db.prepare("DELETE FROM message_channel WHERE message_id = ?").run(id)
-		db.prepare("DELETE FROM event_message WHERE message_id = ?").run(id)
 		await channelWebhook.deleteMessageWithWebhook(channelID, id, threadID)
 	}
 
 	for (const message of messagesToSend) {
 		const reactionPart = messagesToEdit.length === 0 && message === messagesToSend[messagesToSend.length - 1] ? 0 : 1
 		const messageResponse = await channelWebhook.sendMessageWithWebhook(channelID, message, threadID)
-		db.prepare("REPLACE INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(messageResponse.id, threadID || channelID)
+		db.prepare("INSERT INTO message_channel (message_id, channel_id) VALUES (?, ?)").run(messageResponse.id, threadID || channelID)
 		db.prepare("INSERT INTO event_message (event_id, event_type, event_subtype, message_id, part, reaction_part, source) VALUES (?, ?, ?, ?, ?, ?, 0)").run(event.event_id, event.type, event.content["msgtype"] || null, messageResponse.id, eventPart, reactionPart) // source 0 = matrix
 
 		eventPart = 1

src/m2d/actions/setup-emojis.js

@@ -0,0 +1,26 @@
+// @ts-check
+
+const fs = require("fs")
+const {join} = require("path")
+
+const passthrough = require("../../passthrough")
+
+const {id} = require("../../../addbot")
+
+async function setupEmojis() {
+	const {discord, db} = passthrough
+	const emojis = await discord.snow.assets.getAppEmojis(id)
+	for (const name of ["L1", "L2"]) {
+		const existing = emojis.items.find(e => e.name === name)
+		if (existing) {
+			db.prepare("REPLACE INTO auto_emoji (name, emoji_id) VALUES (?, ?)").run(existing.name, existing.id)
+		} else {
+			const filename = join(__dirname, "../../../docs/img", `${name}.png`)
+			const data = fs.readFileSync(filename, null)
+			const uploaded = await discord.snow.assets.createAppEmoji(id, {name, image: "data:image/png;base64," + data.toString("base64")})
+			db.prepare("REPLACE INTO auto_emoji (name, emoji_id) VALUES (?, ?)").run(uploaded.name, uploaded.id)
+		}
+	}
+}
+
+module.exports.setupEmojis = setupEmojis

src/m2d/converters/emoji-sheet.js

@@ -1,6 +1,7 @@
 // @ts-check
 
 const assert = require("assert").strict
+const stream = require("stream")
 const {pipeline} = require("stream").promises
 const sharp = require("sharp")
 const {GIFrame} = require("@cloudrac3r/giframe")
@@ -48,7 +49,7 @@ async function compositeMatrixEmojis(mxcs, mxcDownloader) {
 }
 
 /**
- * @param {import("node-fetch").Response["body"]} streamIn
+ * @param {stream.Readable} streamIn
  * @param {() => any} stopStream
  * @returns {Promise<Buffer | undefined>} Uncompressed PNG image
  */

src/m2d/converters/emoji.js

@@ -1,58 +1,98 @@
 // @ts-check
 
-const assert = require("assert").strict
-const Ty = require("../../types")
+const fsp = require("fs").promises
+const {join} = require("path")
+const emojisp = fsp.readFile(join(__dirname, "emojis.txt"), "utf8").then(content => content.split("\n"))
 
 const passthrough = require("../../passthrough")
-const {sync, select} = passthrough
+const {select} = passthrough
+
 
 /**
  * @param {string} input
  * @param {string | null | undefined} shortcode
  * @returns {string?}
  */
-function encodeEmoji(input, shortcode) {
-	let discordPreferredEncoding
-	if (input.startsWith("mxc://")) {
-		// Custom emoji
-		let row = select("emoji", ["emoji_id", "name"], {mxc_url: input}).get()
-		if (!row && shortcode) {
-			// Use the name to try to find a known emoji with the same name.
-			const name = shortcode.replace(/^:|:$/g, "")
-			row = select("emoji", ["emoji_id", "name"], {name: name}).get()
-		}
-		if (!row) {
-			// We don't have this emoji and there's no realistic way to just-in-time upload a new emoji somewhere.
-			// Sucks!
-			return null
-		}
-		// Cool, we got an exact or a candidate emoji.
-		discordPreferredEncoding = encodeURIComponent(`${row.name}:${row.emoji_id}`)
-	} else {
-		// Default emoji
-		// https://github.com/discord/discord-api-docs/issues/2723#issuecomment-807022205 ????????????
-		const encoded = encodeURIComponent(input)
-		const encodedTrimmed = encoded.replace(/%EF%B8%8F/g, "")
-
-		const forceTrimmedList = [
-			"%F0%9F%91%8D", // 👍
-			"%F0%9F%91%8E", // 👎️
-			"%E2%AD%90", // ⭐
-			"%F0%9F%90%88", // 🐈
-			"%E2%9D%93", // ❓
-			"%F0%9F%8F%86", // 🏆️
-			"%F0%9F%93%9A", // 📚️
-			"%F0%9F%90%9F", // 🐟️
-		]
-
-		discordPreferredEncoding =
-			( forceTrimmedList.includes(encodedTrimmed) ? encodedTrimmed
-			: encodedTrimmed !== encoded && [...input].length === 2 ? encoded
-			: encodedTrimmed)
-
-		console.log("add reaction from matrix:", input, encoded, encodedTrimmed, "chosen:", discordPreferredEncoding)
+function encodeCustomEmoji(input, shortcode) {
+	// Custom emoji
+	let row = select("emoji", ["emoji_id", "name"], {mxc_url: input}).get()
+	if (!row && shortcode) {
+		// Use the name to try to find a known emoji with the same name.
+		const name = shortcode.replace(/^:|:$/g, "")
+		row = select("emoji", ["emoji_id", "name"], {name: name}).get()
+	}
+	if (!row) {
+		// We don't have this emoji and there's no realistic way to just-in-time upload a new emoji somewhere. Sucks!
+		return null
+	}
+	return encodeURIComponent(`${row.name}:${row.emoji_id}`)
+}
+
+/**
+ * @param {string} input
+ * @returns {Promise<string?>} URL encoded!
+ */
+async function encodeDefaultEmoji(input) {
+	// Default emoji
+
+	// Shortcut: If there are ASCII letters then it's not an emoji, it's a freeform Matrix text reaction.
+	// (Regional indicator letters are not ASCII. ASCII digits might be part of an emoji.)
+	if (input.match(/[A-Za-z]/)) return null
+
+	// Check against the dataset
+	const emojis = await emojisp
+	const encoded = encodeURIComponent(input)
+
+	// Best case scenario: they reacted with an exact replica of a valid emoji.
+	if (emojis.includes(input)) return encoded
+
+	// Maybe it has some extraneous \ufe0f or \ufe0e (at the end or in the middle), and it'll be valid if they're removed.
+	const trimmed = input.replace(/\ufe0e|\ufe0f/g, "")
+	const trimmedEncoded = encodeURIComponent(trimmed)
+	if (trimmed !== input) {
+		if (emojis.includes(trimmed)) return trimmedEncoded
+	}
+
+	// Okay, well, maybe it was already missing one and it actually needs an extra \ufe0f, and it'll be valid if that's added.
+	else {
+		const appended = input + "\ufe0f"
+		const appendedEncoded = encodeURIComponent(appended)
+		if (emojis.includes(appended)) return appendedEncoded
+	}
+
+	// Hmm, so adding or removing that from the end didn't help, but maybe there needs to be one in the middle? We can try some heuristics.
+	// These heuristics come from executing scripts/emoji-surrogates-statistics.js.
+	if (trimmedEncoded.length <= 21 && trimmed.match(/^[*#0-9]/)) { // ->19: Keycap digit? 0️⃣ 1️⃣ 2️⃣ 3️⃣ 4️⃣ 5️⃣ 6️⃣ 7️⃣ 8️⃣ 9️⃣ *️⃣ #️⃣
+		const keycap = trimmed[0] + "\ufe0f" + trimmed.slice(1)
+		if (emojis.includes(keycap)) return encodeURIComponent(keycap)
+	} else if (trimmedEncoded.length === 27 && trimmed[0] === "⛹") { // ->45: ⛹️‍♀️ ⛹️‍♂️
+		const balling = trimmed[0] + "\ufe0f" + trimmed.slice(1) + "\ufe0f"
+		if (emojis.includes(balling)) return encodeURIComponent(balling)
+	} else if (trimmedEncoded.length === 30) { // ->39: ⛓️‍💥 ❤️‍🩹 ❤️‍🔥 or ->48: 🏳️‍⚧️ 🏌️‍♀️ 🕵️‍♀️ 🏋️‍♀️ and gender variants
+		const thriving = trimmed[0] + "\ufe0f" + trimmed.slice(1)
+		if (emojis.includes(thriving)) return encodeURIComponent(thriving)
+		const powerful = trimmed.slice(0, 2) + "\ufe0f" + trimmed.slice(2) + "\ufe0f"
+		if (emojis.includes(powerful)) return encodeURIComponent(powerful)
+	} else if (trimmedEncoded.length === 51 && trimmed[3] === "❤") { // ->60: 👩‍❤️‍👨 👩‍❤️‍👩 👨‍❤️‍👨
+		const yellowRomance = trimmed.slice(0, 3) + "❤\ufe0f" + trimmed.slice(4)
+		if (emojis.includes(yellowRomance)) return encodeURIComponent(yellowRomance)
+	}
+
+	// there are a few more longer ones but I got bored
+	return null
+}
+
+/**
+ * @param {string} input
+ * @param {string | null | undefined} shortcode
+ * @returns {Promise<string?>}
+ */
+async function encodeEmoji(input, shortcode) {
+	if (input.startsWith("mxc://")) {
+		return encodeCustomEmoji(input, shortcode)
+	} else {
+		return encodeDefaultEmoji(input)
 	}
-	return discordPreferredEncoding
 }
 
 module.exports.encodeEmoji = encodeEmoji

src/m2d/converters/emoji.test.js

@@ -0,0 +1,52 @@
+// @ts-check
+
+const {test} = require("supertape")
+const {encodeEmoji} = require("./emoji")
+
+test("emoji: valid", async t => {
+	t.equal(await encodeEmoji("🦄", null), "%F0%9F%A6%84")
+})
+
+test("emoji: freeform text", async t => {
+	t.equal(await encodeEmoji("ha", null), null)
+})
+
+test("emoji: suspicious unicode", async t => {
+	t.equal(await encodeEmoji("Ⓐ", null), null)
+})
+
+test("emoji: needs u+fe0f added", async t => {
+	t.equal(await encodeEmoji("☺", null), "%E2%98%BA%EF%B8%8F")
+})
+
+test("emoji: needs u+fe0f removed", async t => {
+	t.equal(await encodeEmoji("⭐️", null), "%E2%AD%90")
+})
+
+test("emoji: number key needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("3⃣", null), "3%EF%B8%8F%E2%83%A3")
+})
+
+test("emoji: hash key needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("#⃣", null), "%23%EF%B8%8F%E2%83%A3")
+})
+
+test("emoji: broken chains needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("⛓‍💥", null), "%E2%9B%93%EF%B8%8F%E2%80%8D%F0%9F%92%A5")
+})
+
+test("emoji: balling needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("⛹‍♀", null), "%E2%9B%B9%EF%B8%8F%E2%80%8D%E2%99%80%EF%B8%8F")
+})
+
+test("emoji: trans flag needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("🏳‍⚧", null), "%F0%9F%8F%B3%EF%B8%8F%E2%80%8D%E2%9A%A7%EF%B8%8F")
+})
+
+test("emoji: spy needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("🕵‍♀", null), "%F0%9F%95%B5%EF%B8%8F%E2%80%8D%E2%99%80%EF%B8%8F")
+})
+
+test("emoji: couple needs u+fe0f in the middle", async t => {
+	t.equal(await encodeEmoji("👩‍❤‍👩", null), "%F0%9F%91%A9%E2%80%8D%E2%9D%A4%EF%B8%8F%E2%80%8D%F0%9F%91%A9")
+})

src/m2d/converters/event-to-message.js

@@ -2,7 +2,7 @@
 
 const Ty = require("../../types")
 const DiscordTypes = require("discord-api-types/v10")
-const {Readable} = require("stream")
+const stream = require("stream")
 const chunk = require("chunk-text")
 const TurndownService = require("@cloudrac3r/turndown")
 const domino = require("domino")
@@ -19,6 +19,8 @@ const dUtils = sync.require("../../discord/utils")
 const file = sync.require("../../matrix/file")
 /** @type {import("./emoji-sheet")} */
 const emojiSheet = sync.require("./emoji-sheet")
+/** @type {import("../actions/setup-emojis")} */
+const setupEmojis = sync.require("../actions/setup-emojis")
 
 /** @type {[RegExp, string][]} */
 const markdownEscapes = [
@@ -154,6 +156,27 @@ turndownService.addRule("listItem", {
 	}
 })
 
+turndownService.addRule("table", {
+	filter: "table",
+	replacement: function (content, node, options) {
+		const trs = node.querySelectorAll("tr").cache
+		/** @type {{text: string, tag: string}[][]} */
+		const tableText = trs.map(tr => [...tr.querySelectorAll("th, td")].map(cell => ({text: cell.textContent, tag: cell.tagName})))
+		const tableTextByColumn = tableText[0].map((col, i) => tableText.map(row => row[i]))
+		const columnWidths = tableTextByColumn.map(col => Math.max(...col.map(cell => cell.text.length)))
+		const resultRows = tableText.map((row, rowIndex) =>
+			row.map((cell, colIndex) =>
+				cell.text.padEnd(columnWidths[colIndex])
+			).join("  ")
+		)
+		const tableHasHeader = tableText[0].slice(1).some(cell => cell.tag === "TH")
+		if (tableHasHeader) {
+			resultRows.splice(1, 0, "-".repeat(columnWidths.reduce((a, c) => a + c + 2)))
+		}
+		return "```\n" + resultRows.join("\n") + "```"
+	}
+})
+
 /** @type {string[]} SPRITE SHEET EMOJIS FEATURE: mxc urls for the currently processing message */
 let endOfMessageEmojis = []
 turndownService.addRule("emoji", {
@@ -258,7 +281,18 @@ async function getMemberFromCacheOrHomeserver(roomID, mxid, api) {
 	const row = select("member_cache", ["displayname", "avatar_url"], {room_id: roomID, mxid}).get()
 	if (row) return row
 	return api.getStateEvent(roomID, "m.room.member", mxid).then(event => {
-		db.prepare("REPLACE INTO member_cache (room_id, mxid, displayname, avatar_url) VALUES (?, ?, ?, ?)").run(roomID, mxid, event?.displayname || null, event?.avatar_url || null)
+		const room = select("channel_room", "room_id", {room_id: roomID}).get()
+		if (room) {
+			// save the member to the cache so we don't have to check with the homeserver next time
+			// the cache will be kept in sync by the `m.room.member` event listener
+			const displayname = event?.displayname || null
+			const avatar_url = event?.avatar_url || null
+			db.prepare("INSERT INTO member_cache (room_id, mxid, displayname, avatar_url) VALUES (?, ?, ?, ?) ON CONFLICT DO UPDATE SET displayname = ?, avatar_url = ?").run(
+				roomID, mxid,
+				displayname, avatar_url,
+				displayname, avatar_url
+			)
+		}
 		return event
 	}).catch(() => {
 		return {displayname: null, avatar_url: null}
@@ -304,7 +338,7 @@ function getUserOrProxyOwnerID(mxid) {
  * At the time of this executing, we know what the end of message emojis are, and we know that at least one of them is unknown.
  * This function will strip them from the content and generate the correct pending file of the sprite sheet.
  * @param {string} content
- * @param {{id: string, name: string}[]} attachments
+ * @param {{id: string, filename: string}[]} attachments
  * @param {({name: string, mxc: string} | {name: string, mxc: string, key: string, iv: string} | {name: string, buffer: Buffer})[]} pendingFiles
  * @param {(mxc: string) => Promise<Buffer | undefined>} mxcDownloader function that will download the mxc URLs and convert to uncompressed PNG data. use `getAndConvertEmoji` or a mock.
  */
@@ -318,9 +352,9 @@ async function uploadEndOfMessageSpriteSheet(content, attachments, pendingFiles,
 	// Create a sprite sheet of known and unknown emojis from the end of the message
 	const buffer = await emojiSheet.compositeMatrixEmojis(endOfMessageEmojis, mxcDownloader)
 	// Attach it
-	const name = "emojis.png"
-	attachments.push({id: String(attachments.length), name})
-	pendingFiles.push({name, buffer})
+	const filename = "emojis.png"
+	attachments.push({id: String(attachments.length), filename})
+	pendingFiles.push({name: filename, buffer})
 	return content
 }
 
@@ -415,7 +449,7 @@ async function checkWrittenMentions(content, senderMxid, roomID, guild, di) {
 					allowedMentionsParse: ["everyone"]
 				}
 			}
-		} else {
+		} else if (writtenMentionMatch[1].length < 40) { // the API supports up to 100 characters, but really if you're searching more than 40, something messed up
 			const results = await di.snow.guild.searchGuildMembers(guild.id, {query: writtenMentionMatch[1]})
 			if (results[0]) {
 				assert(results[0].user)
@@ -447,6 +481,23 @@ const attachmentEmojis = new Map([
 	["m.file", "📄"]
 ])
 
+async function getL1L2ReplyLine(called = false) {
+	// @ts-ignore
+	const autoEmoji = new Map(select("auto_emoji", ["name", "emoji_id"], {}, "WHERE name = 'L1' OR name = 'L2'").raw().all())
+	if (autoEmoji.size === 2) {
+		return `<:L1:${autoEmoji.get("L1")}><:L2:${autoEmoji.get("L2")}>`
+	}
+	/* c8 ignore start */
+	if (called) {
+		// Don't know how this could happen, but just making sure we don't enter an infinite loop.
+		console.warn("Warning: OOYE is missing data to format replies. To fix this: `npm run setup`")
+		return ""
+	}
+	await setupEmojis.setupEmojis()
+	return getL1L2ReplyLine(true)
+	/* c8 ignore stop */
+}
+
 /**
  * @param {Ty.Event.Outer_M_Room_Message | Ty.Event.Outer_M_Room_Message_File | Ty.Event.Outer_M_Sticker | Ty.Event.Outer_M_Room_Message_Encrypted_File} event
  * @param {import("discord-api-types/v10").APIGuild} guild
@@ -465,7 +516,7 @@ async function eventToMessage(event, guild, di) {
 	// Try to extract an accurate display name and avatar URL from the member event
 	const member = await getMemberFromCacheOrHomeserver(event.room_id, event.sender, di?.api)
 	if (member.displayname) displayName = member.displayname
-	if (member.avatar_url) avatarURL = mxUtils.getPublicUrlForMxc(member.avatar_url) || undefined
+	if (member.avatar_url) avatarURL = mxUtils.getPublicUrlForMxc(member.avatar_url)
 	// If the display name is too long to be put into the webhook (80 characters is the maximum),
 	// put the excess characters into displayNameRunoff, later to be put at the top of the message
 	let [displayNameShortened, displayNameRunoff] = splitDisplayName(displayName)
@@ -475,6 +526,7 @@ async function eventToMessage(event, guild, di) {
 	}
 
 	let content = event.content.body // ultimate fallback
+	/** @type {{id: string, filename: string}[]} */
 	const attachments = []
 	/** @type {({name: string, mxc: string} | {name: string, mxc: string, key: string, iv: string} | {name: string, buffer: Buffer})[]} */
 	const pendingFiles = []
@@ -482,7 +534,45 @@ async function eventToMessage(event, guild, di) {
 	const ensureJoined = []
 
 	// Convert content depending on what the message is
-	if (event.type === "m.room.message" && (event.content.msgtype === "m.text" || event.content.msgtype === "m.emote")) {
+	// Handle images first - might need to handle their `body`/`formatted_body` as well, which will fall through to the text processor
+	let shouldProcessTextEvent = event.type === "m.room.message" && (event.content.msgtype === "m.text" || event.content.msgtype === "m.emote")
+	if (event.type === "m.room.message" && (event.content.msgtype === "m.file" || event.content.msgtype === "m.video" || event.content.msgtype === "m.audio" || event.content.msgtype === "m.image")) {
+		content = ""
+		const filename = event.content.filename || event.content.body
+		if ("file" in event.content) {
+			// Encrypted
+			assert.equal(event.content.file.key.alg, "A256CTR")
+			attachments.push({id: "0", filename})
+			pendingFiles.push({name: filename, mxc: event.content.file.url, key: event.content.file.key.k, iv: event.content.file.iv})
+		} else {
+			// Unencrypted
+			attachments.push({id: "0", filename})
+			pendingFiles.push({name: filename, mxc: event.content.url})
+		}
+		// Check if we also need to process a text event for this image - if it has a caption that's different from its filename
+		if ((event.content.body && event.content.filename && event.content.body !== event.content.filename) || event.content.formatted_body) {
+			shouldProcessTextEvent = true
+		}
+	}
+	if (event.type === "m.sticker") {
+		content = ""
+		let filename = event.content.body
+		if (event.type === "m.sticker") {
+			let mimetype
+			if (event.content.info?.mimetype?.includes("/")) {
+				mimetype = event.content.info.mimetype
+			} else {
+				const res = await di.api.getMedia(event.content.url, {method: "HEAD"})
+				if (res.status === 200) {
+					mimetype = res.headers.get("content-type")
+				}
+				if (!mimetype) throw new Error(`Server error ${res.status} or missing content-type while detecting sticker mimetype`)
+			}
+			filename += "." + mimetype.split("/")[1]
+		}
+		attachments.push({id: "0", filename})
+		pendingFiles.push({name: filename, mxc: event.content.url})
+	} else if (shouldProcessTextEvent) {
 		// Handling edits. If the edit was an edit of a reply, edits do not include the reply reference, so we need to fetch up to 2 more events.
 		// this event ---is an edit of--> original event ---is a reply to--> past event
 		await (async () => {
@@ -496,13 +586,12 @@ async function eventToMessage(event, guild, di) {
 			if (!messageIDsToEdit.length) return
 
 			// Ok, it's an edit.
-			event.content = event.content["m.new_content"]
+			event = {...event, content: event.content["m.new_content"]}
 
 			// Is it editing a reply? We need special handling if it is.
 			// Get the original event, then check if it was a reply
 			const originalEvent = await di.api.getEvent(event.room_id, originalEventId)
-			if (!originalEvent) return
-			const repliedToEventId = originalEvent.content["m.relates_to"]?.["m.in_reply_to"]?.event_id
+			const repliedToEventId = originalEvent?.content?.["m.relates_to"]?.["m.in_reply_to"]?.event_id
 			if (!repliedToEventId) return
 
 			// After all that, it's an edit of a reply.
@@ -558,41 +647,32 @@ async function eventToMessage(event, guild, di) {
 				return
 			}
 
-			// @ts-ignore
-			const autoEmoji = new Map(select("auto_emoji", ["name", "emoji_id"], {}, "WHERE name = 'L1' OR name = 'L2'").raw().all())
-			replyLine = `<:L1:${autoEmoji.get("L1")}><:L2:${autoEmoji.get("L2")}>`
+			replyLine = await getL1L2ReplyLine()
 			const row = from("event_message").join("message_channel", "message_id").select("channel_id", "message_id").where({event_id: repliedToEventId}).and("ORDER BY part").get()
 			if (row) {
 				replyLine += `https://discord.com/channels/${guild.id}/${row.channel_id}/${row.message_id} `
 			}
-			const sender = repliedToEvent.sender
-			const authorID = getUserOrProxyOwnerID(sender)
-			if (authorID) {
-				replyLine += `<@${authorID}>`
-			} else {
-				let senderName = select("member_cache", "displayname", {mxid: sender}).pluck().get()
-				if (!senderName) {
-					const match = sender.match(/@([^:]*)/)
-					assert(match)
-					senderName = match[1]
-				}
-				replyLine += `**Ⓜ${senderName}**`
-			}
 			// If the event has been edited, the homeserver will include the relation in `unsigned`.
 			if (repliedToEvent.unsigned?.["m.relations"]?.["m.replace"]?.content?.["m.new_content"]) {
 				repliedToEvent = repliedToEvent.unsigned["m.relations"]["m.replace"] // Note: this changes which event_id is in repliedToEvent.
 				repliedToEvent.content = repliedToEvent.content["m.new_content"]
 			}
-			let contentPreview
+			/** @type {string} */
+			let repliedToContent = repliedToEvent.content.formatted_body || repliedToEvent.content.body
 			const fileReplyContentAlternative = attachmentEmojis.get(repliedToEvent.content.msgtype)
+			let contentPreview
 			if (fileReplyContentAlternative) {
 				contentPreview = " " + fileReplyContentAlternative
 			} else if (repliedToEvent.unsigned?.redacted_because) {
 				contentPreview = " (in reply to a deleted message)"
+			} else if (typeof repliedToContent !== "string") {
+				// in reply to a weird metadata event like m.room.name, m.room.member...
+				// I'm not implementing text fallbacks for arbitrary room events. this should cover most cases
+				// this has never ever happened in the wild anyway
+				repliedToEvent.sender = ""
+				contentPreview = " (channel details edited)"
 			} else {
 				// Generate a reply preview for a standard message
-				/** @type {string} */
-				let repliedToContent = repliedToEvent.content.formatted_body || repliedToEvent.content.body
 				repliedToContent = repliedToContent.replace(/.*<\/mx-reply>/s, "") // Remove everything before replies, so just use the actual message body
 				repliedToContent = repliedToContent.replace(/^\s*<blockquote>.*?<\/blockquote>(.....)/s, "$1") // If the message starts with a blockquote, don't count it and use the message body afterwards
 				repliedToContent = repliedToContent.replace(/(?:\n|<br>)+/g, " ") // Should all be on one line
@@ -613,6 +693,15 @@ async function eventToMessage(event, guild, di) {
 					contentPreview = ""
 				}
 			}
+			const sender = repliedToEvent.sender
+			const authorID = getUserOrProxyOwnerID(sender)
+			if (authorID) {
+				replyLine += `<@${authorID}>`
+			} else {
+				let senderName = select("member_cache", "displayname", {mxid: sender}).pluck().get()
+				if (!senderName) senderName = sender.match(/@([^:]*)/)?.[1]
+				if (senderName) replyLine += `**Ⓜ${senderName}**`
+			}
 			replyLine = `-# > ${replyLine}${contentPreview}\n`
 		})()
 
@@ -768,47 +857,13 @@ async function eventToMessage(event, guild, di) {
 			// @ts-ignore bad type from turndown
 			content = turndownService.escape(content)
 		}
-	} else if (event.type === "m.room.message" && (event.content.msgtype === "m.file" || event.content.msgtype === "m.video" || event.content.msgtype === "m.audio" || event.content.msgtype === "m.image")) {
-		content = ""
-		const filename = event.content.filename || event.content.body
-		// A written `event.content.body` will be bridged to Discord's image `description` which is like alt text.
-		// Bridging as description rather than message content in order to match Matrix clients (Element, Neochat) which treat this as alt text or title text.
-		const description = (event.content.body !== event.content.filename && event.content.filename && event.content.body) || undefined
-		if ("url" in event.content) {
-			// Unencrypted
-			attachments.push({id: "0", description, filename})
-			pendingFiles.push({name: filename, mxc: event.content.url})
-		} else {
-			// Encrypted
-			assert.equal(event.content.file.key.alg, "A256CTR")
-			attachments.push({id: "0", description, filename})
-			pendingFiles.push({name: filename, mxc: event.content.file.url, key: event.content.file.key.k, iv: event.content.file.iv})
-		}
-	} else if (event.type === "m.sticker") {
-		content = ""
-		let filename = event.content.body
-		if (event.type === "m.sticker") {
-			let mimetype
-			if (event.content.info?.mimetype?.includes("/")) {
-				mimetype = event.content.info.mimetype
-			} else {
-				const res = await di.api.getMedia(event.content.url, {method: "HEAD"})
-				if (res.status === 200) {
-					mimetype = res.headers.get("content-type")
-				}
-				if (!mimetype) throw new Error(`Server error ${res.status} or missing content-type while detecting sticker mimetype`)
-			}
-			filename += "." + mimetype.split("/")[1]
-		}
-		attachments.push({id: "0", filename})
-		pendingFiles.push({name: filename, mxc: event.content.url})
 	}
 
 	content = displayNameRunoff + replyLine + content
 
 	// Split into 2000 character chunks
 	const chunks = chunk(content, 2000)
-	/** @type {(DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | Readable}[]})[]} */
+	/** @type {(DiscordTypes.RESTPostAPIWebhookWithTokenJSONBody & {files?: {name: string, file: Buffer | stream.Readable}[]})[]} */
 	const messages = chunks.map(content => ({
 		content,
 		allowed_mentions: {

src/m2d/converters/event-to-message.test.js

@@ -559,7 +559,7 @@ test("event2message: lists are bridged correctly", async t => {
 				"transaction_id": "m1692967313951.441"
 			},
 			"event_id": "$l-xQPY5vNJo3SNxU9d8aOWNVD1glMslMyrp4M_JEF70",
-			"room_id": "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			"room_id": "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -662,7 +662,7 @@ test("event2message: code block contents are formatted correctly and not escaped
 				formatted_body: "<pre><code>input = input.replace(/(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?\\n(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?/g,\n_input_ = input = input.replace(/(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?\\n(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?/g,\n</code></pre>\n<p><code>input = input.replace(/(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?\\n(&lt;\\/?([^ &gt;]+)[^&gt;]*&gt;)?/g,</code></p>\n"
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -692,7 +692,7 @@ test("event2message: code blocks use double backtick as delimiter when necessary
 				formatted_body: "<code>backtick in ` the middle</code>, <code>backtick at the edge`</code>"
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -722,7 +722,7 @@ test("event2message: inline code is converted to code block if it contains both
 				formatted_body: "<code>` one two ``</code>"
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -752,7 +752,7 @@ test("event2message: code blocks are uploaded as attachments instead if they con
 				formatted_body: 'So if you run code like this<pre><code class="language-java">System.out.println("```");</code></pre>it should print a markdown formatted code block'
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -784,7 +784,7 @@ test("event2message: code blocks are uploaded as attachments instead if they con
 				formatted_body: 'So if you run code like this<pre><code>System.out.println("```");</code></pre>it should print a markdown formatted code block'
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -821,7 +821,7 @@ test("event2message: characters are encoded properly in code blocks", async t =>
 			  		+ '\n</code></pre>'
 			},
 			event_id: "$pGkWQuGVmrPNByrFELxhzI6MCBgJecr5I2J3z88Gc2s",
-			room_id: "!BpMdOUkWWhFxmTrENV:cadence.moe"
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 		}),
 		{
 			ensureJoined: [],
@@ -902,7 +902,7 @@ test("event2message: lists have appropriate line breaks", async t => {
 				'm.mentions': {},
 				msgtype: 'm.text'
 			},
-			room_id: '!cBxtVRxDlZvSVhJXVK:cadence.moe',
+			room_id: '!TqlyQmifxGUggEmdBN:cadence.moe',
 			sender: '@Milan:tchncs.de',
 			type: 'm.room.message',
 		}),
@@ -943,7 +943,7 @@ test("event2message: ordered list start attribute works", async t => {
 				'm.mentions': {},
 				msgtype: 'm.text'
 			},
-			room_id: '!cBxtVRxDlZvSVhJXVK:cadence.moe',
+			room_id: '!TqlyQmifxGUggEmdBN:cadence.moe',
 			sender: '@Milan:tchncs.de',
 			type: 'm.room.message',
 		}),
@@ -1088,7 +1088,7 @@ test("event2message: rich reply to a rich reply to a multi-line message should c
 			content: {
 				body: "> <@cadence:cadence.moe> I just checked in a fix that will probably work, can you try reproducing this on the latest `main` branch and see if I fixed it?\n\nwill try later (tomorrow if I don't forgor)",
 				format: "org.matrix.custom.html",
-				formatted_body: "<mx-reply><blockquote><a href=\"https://matrix.to/#/!cBxtVRxDlZvSVhJXVK:cadence.moe/$A0Rj559NKOh2VndCZSTJXcvgi42gZWVfVQt73wA2Hn0?via=matrix.org&via=cadence.moe&via=syndicated.gay\">In reply to</a> <a href=\"https://matrix.to/#/@cadence:cadence.moe\">@cadence:cadence.moe</a><br />I just checked in a fix that will probably work, can you try reproducing this on the latest <code>main</code> branch and see if I fixed it?</blockquote></mx-reply>will try later (tomorrow if I don't forgor)",
+				formatted_body: "<mx-reply><blockquote><a href=\"https://matrix.to/#/!TqlyQmifxGUggEmdBN:cadence.moe/$A0Rj559NKOh2VndCZSTJXcvgi42gZWVfVQt73wA2Hn0?via=matrix.org&via=cadence.moe&via=syndicated.gay\">In reply to</a> <a href=\"https://matrix.to/#/@cadence:cadence.moe\">@cadence:cadence.moe</a><br />I just checked in a fix that will probably work, can you try reproducing this on the latest <code>main</code> branch and see if I fixed it?</blockquote></mx-reply>will try later (tomorrow if I don't forgor)",
 				"m.relates_to": {
 					"m.in_reply_to": {
 						event_id: "$A0Rj559NKOh2VndCZSTJXcvgi42gZWVfVQt73wA2Hn0"
@@ -1111,7 +1111,7 @@ test("event2message: rich reply to a rich reply to a multi-line message should c
 						"msgtype": "m.text",
 						"body": "> <@solonovamax:matrix.org> multipart messages will be deleted if the message is edited to require less space\n>  \n> \n> steps to reproduce:\n> \n> 1. send a message that is longer than 2000 characters (discord character limit)\n>   - bot will split message into two messages on discord\n> 2. edit message to be under 2000 characters (discord character limit)\n>   - bot will delete one of the messages on discord, and then edit the other one to include the edited content\n>   - the bot will *then* delete the message on matrix (presumably) because one of the messages on discord was deleted (by \n\nI just checked in a fix that will probably work, can you try reproducing this on the latest `main` branch and see if I fixed it?",
 						"format": "org.matrix.custom.html",
-						"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!cBxtVRxDlZvSVhJXVK:cadence.moe/$u4OD19vd2GETkOyhgFVla92oDKI4ojwBf2-JeVCG7EI?via=cadence.moe&via=matrix.org&via=conduit.rory.gay\">In reply to</a> <a href=\"https://matrix.to/#/@solonovamax:matrix.org\">@solonovamax:matrix.org</a><br /><p>multipart messages will be deleted if the message is edited to require less space</p>\n<p>steps to reproduce:</p>\n<ol>\n<li>send a message that is longer than 2000 characters (discord character limit)</li>\n</ol>\n<ul>\n<li>bot will split message into two messages on discord</li>\n</ul>\n<ol start=\"2\">\n<li>edit message to be under 2000 characters (discord character limit)</li>\n</ol>\n<ul>\n<li>bot will delete one of the messages on discord, and then edit the other one to include the edited content</li>\n<li>the bot will <em>then</em> delete the message on matrix (presumably) because one of the messages on discord was deleted (by</li>\n</ul>\n</blockquote></mx-reply>I just checked in a fix that will probably work, can you try reproducing this on the latest <code>main</code> branch and see if I fixed it?",
+						"formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!TqlyQmifxGUggEmdBN:cadence.moe/$u4OD19vd2GETkOyhgFVla92oDKI4ojwBf2-JeVCG7EI?via=cadence.moe&via=matrix.org&via=conduit.rory.gay\">In reply to</a> <a href=\"https://matrix.to/#/@solonovamax:matrix.org\">@solonovamax:matrix.org</a><br /><p>multipart messages will be deleted if the message is edited to require less space</p>\n<p>steps to reproduce:</p>\n<ol>\n<li>send a message that is longer than 2000 characters (discord character limit)</li>\n</ol>\n<ul>\n<li>bot will split message into two messages on discord</li>\n</ul>\n<ol start=\"2\">\n<li>edit message to be under 2000 characters (discord character limit)</li>\n</ol>\n<ul>\n<li>bot will delete one of the messages on discord, and then edit the other one to include the edited content</li>\n<li>the bot will <em>then</em> delete the message on matrix (presumably) because one of the messages on discord was deleted (by</li>\n</ul>\n</blockquote></mx-reply>I just checked in a fix that will probably work, can you try reproducing this on the latest <code>main</code> branch and see if I fixed it?",
 						"m.relates_to": {
 							"m.in_reply_to": {
 								"event_id": "$u4OD19vd2GETkOyhgFVla92oDKI4ojwBf2-JeVCG7EI"
@@ -1123,7 +1123,7 @@ test("event2message: rich reply to a rich reply to a multi-line message should c
 						"age": 19069564
 					},
 					"event_id": "$A0Rj559NKOh2VndCZSTJXcvgi42gZWVfVQt73wA2Hn0",
-					"room_id": "!cBxtVRxDlZvSVhJXVK:cadence.moe"
+					"room_id": "!TqlyQmifxGUggEmdBN:cadence.moe"
 				})
 			},
 			snow: {
@@ -2625,6 +2625,52 @@ test("event2message: rich reply to a deleted event", async t => {
 	)
 })
 
+test("event2message: rich reply to a state event with no body", async t => {
+	t.deepEqual(
+		await eventToMessage({
+			type: "m.room.message",
+			sender: "@ampflower:matrix.org",
+			content: {
+				msgtype: "m.text",
+				body: "> <@ampflower:matrix.org> changed the room topic\n\nnice room topic",
+				format: "org.matrix.custom.html",
+				formatted_body: "<mx-reply><blockquote><a href=\"https://matrix.to/#/!TqlyQmifxGUggEmdBN:cadence.moe/$f-noT-d-Eo_Xgpc05Ww89ErUXku4NwKWYGHLzWKo1kU?via=cadence.moe\">In reply to</a> <a href=\"https://matrix.to/#/@ampflower:matrix.org\">@ampflower:matrix.org</a> changed the room topic<br></blockquote></mx-reply>nice room topic",
+				"m.relates_to": {
+					"m.in_reply_to": {
+						event_id: "$f-noT-d-Eo_Xgpc05Ww89ErUXku4NwKWYGHLzWKo1kU"
+					}
+				}
+			},
+			event_id: "$v_Gtr-bzv9IVlSLBO5DstzwmiDd-GSFaNfHX66IupV8",
+			room_id: "!TqlyQmifxGUggEmdBN:cadence.moe"
+		 }, data.guild.general, {
+			api: {
+				getEvent: mockGetEvent(t, "!TqlyQmifxGUggEmdBN:cadence.moe", "$f-noT-d-Eo_Xgpc05Ww89ErUXku4NwKWYGHLzWKo1kU", {
+					type: "m.room.topic",
+					sender: "@ampflower:matrix.org",
+					content: {
+						topic: "you're cute"
+					},
+					user_id: "@ampflower:matrix.org"
+				})
+			}
+		}),
+		{
+			ensureJoined: [],
+			messagesToDelete: [],
+			messagesToEdit: [],
+			messagesToSend: [{
+				username: "Ampflower 🌺",
+				content: "-# > <:L1:1144820033948762203><:L2:1144820084079087647> (channel details edited)\nnice room topic",
+				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/PRfhXYBTOalvgQYtmCLeUXko",
+				allowed_mentions: {
+					parse: ["users", "roles"]
+				}
+			}]
+		}
+	)
+})
+
 test("event2message: raw mentioning discord users in plaintext body works", async t => {
 	t.deepEqual(
 		await eventToMessage({
@@ -3485,7 +3531,7 @@ test("event2message: caches the member if the member is not known", async t => {
 			},
 			event_id: "$g07oYSZFWBkxohNEfywldwgcWj1hbhDzQ1sBAKvqOOU",
 			origin_server_ts: 1688301929913,
-			room_id: "!should_be_newly_cached:cadence.moe",
+			room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe",
 			sender: "@should_be_newly_cached:cadence.moe",
 			type: "m.room.message",
 			unsigned: {
@@ -3495,7 +3541,7 @@ test("event2message: caches the member if the member is not known", async t => {
 			api: {
 				getStateEvent: async (roomID, type, stateKey) => {
 					called++
-					t.equal(roomID, "!should_be_newly_cached:cadence.moe")
+					t.equal(roomID, "!qzDBLKlildpzrrOnFZ:cadence.moe")
 					t.equal(type, "m.room.member")
 					t.equal(stateKey, "@should_be_newly_cached:cadence.moe")
 					return {
@@ -3519,12 +3565,60 @@ test("event2message: caches the member if the member is not known", async t => {
 		}
 	)
 
-	t.deepEqual(select("member_cache", ["avatar_url", "displayname", "mxid"], {room_id: "!should_be_newly_cached:cadence.moe"}).all(), [
+	t.deepEqual(select("member_cache", ["avatar_url", "displayname", "mxid"], {room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe"}).all(), [
 		{avatar_url: "mxc://cadence.moe/this_is_the_avatar", displayname: null, mxid: "@should_be_newly_cached:cadence.moe"}
 	])
 	t.equal(called, 1, "getStateEvent should be called once")
 })
 
+test("event2message: does not cache the member if the room is not known", async t => {
+	let called = 0
+	t.deepEqual(
+		await eventToMessage({
+			content: {
+				body: "testing the member state cache",
+				msgtype: "m.text"
+			},
+			event_id: "$g07oYSZFWBkxohNEfywldwgcWj1hbhDzQ1sBAKvqOOU",
+			origin_server_ts: 1688301929913,
+			room_id: "!not_real:cadence.moe",
+			sender: "@should_not_be_cached:cadence.moe",
+			type: "m.room.message",
+			unsigned: {
+				age: 405299
+			}
+		}, {}, {
+			api: {
+				getStateEvent: async (roomID, type, stateKey) => {
+					called++
+					t.equal(roomID, "!not_real:cadence.moe")
+					t.equal(type, "m.room.member")
+					t.equal(stateKey, "@should_not_be_cached:cadence.moe")
+					return {
+						avatar_url: "mxc://cadence.moe/this_is_the_avatar"
+					}
+				}
+			}
+		}),
+		{
+			ensureJoined: [],
+			messagesToDelete: [],
+			messagesToEdit: [],
+			messagesToSend: [{
+				username: "should_not_be_cached",
+				content: "testing the member state cache",
+				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/this_is_the_avatar",
+				allowed_mentions: {
+					parse: ["users", "roles"]
+				}
+			}]
+		}
+	)
+
+	t.deepEqual(select("member_cache", ["avatar_url", "displayname", "mxid"], {room_id: "!not_real:cadence.moe"}).all(), [])
+	t.equal(called, 1, "getStateEvent should be called once")
+})
+
 test("event2message: skips caching the member if the member does not exist, somehow", async t => {
 	let called = 0
 	t.deepEqual(
@@ -3580,7 +3674,7 @@ test("event2message: overly long usernames are shifted into the message content"
 			},
 			event_id: "$g07oYSZFWBkxohNEfywldwgcWj1hbhDzQ1sBAKvqOOU",
 			origin_server_ts: 1688301929913,
-			room_id: "!should_be_newly_cached_2:cadence.moe",
+			room_id: "!cqeGDbPiMFAhLsqqqq:cadence.moe",
 			sender: "@should_be_newly_cached_2:cadence.moe",
 			type: "m.room.message",
 			unsigned: {
@@ -3590,7 +3684,7 @@ test("event2message: overly long usernames are shifted into the message content"
 			api: {
 				getStateEvent: async (roomID, type, stateKey) => {
 					called++
-					t.equal(roomID, "!should_be_newly_cached_2:cadence.moe")
+					t.equal(roomID, "!cqeGDbPiMFAhLsqqqq:cadence.moe")
 					t.equal(type, "m.room.member")
 					t.equal(stateKey, "@should_be_newly_cached_2:cadence.moe")
 					return {
@@ -3613,7 +3707,7 @@ test("event2message: overly long usernames are shifted into the message content"
 			}]
 		}
 	)
-	t.deepEqual(select("member_cache", ["avatar_url", "displayname", "mxid"], {room_id: "!should_be_newly_cached_2:cadence.moe"}).all(), [
+	t.deepEqual(select("member_cache", ["avatar_url", "displayname", "mxid"], {room_id: "!cqeGDbPiMFAhLsqqqq:cadence.moe"}).all(), [
 		{avatar_url: null, displayname: "I am BLACK I am WHITE I am SHORT I am LONG I am EVERYTHING YOU THINK IS IMPORTANT and I DON'T MATTER", mxid: "@should_be_newly_cached_2:cadence.moe"}
 	])
 	t.equal(called, 1, "getStateEvent should be called once")
@@ -3628,7 +3722,7 @@ test("event2message: overly long usernames are not treated specially when the ms
 			},
 			event_id: "$g07oYSZFWBkxohNEfywldwgcWj1hbhDzQ1sBAKvqOOU",
 			origin_server_ts: 1688301929913,
-			room_id: "!should_be_newly_cached_2:cadence.moe",
+			room_id: "!cqeGDbPiMFAhLsqqqq:cadence.moe",
 			sender: "@should_be_newly_cached_2:cadence.moe",
 			type: "m.room.message",
 			unsigned: {
@@ -3676,7 +3770,7 @@ test("event2message: text attachments work", async t => {
 				username: "cadence [they]",
 				content: "",
 				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/azCAhThKTojXSZJRoWwZmhvU",
-				attachments: [{id: "0", description: undefined, filename: "chiki-powerups.txt"}],
+				attachments: [{id: "0", filename: "chiki-powerups.txt"}],
 				pendingFiles: [{name: "chiki-powerups.txt", mxc: "mxc://cadence.moe/zyThGlYQxvlvBVbVgKDDbiHH"}]
 			}]
 		}
@@ -3712,14 +3806,14 @@ test("event2message: image attachments work", async t => {
 				username: "cadence [they]",
 				content: "",
 				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/azCAhThKTojXSZJRoWwZmhvU",
-				attachments: [{id: "0", description: undefined, filename: "cool cat.png"}],
+				attachments: [{id: "0", filename: "cool cat.png"}],
 				pendingFiles: [{name: "cool cat.png", mxc: "mxc://cadence.moe/IvxVJFLEuksCNnbojdSIeEvn"}]
 			}]
 		}
 	)
 })
 
-test("event2message: image attachments can have a custom description", async t => {
+test("event2message: image attachments can have a plaintext caption", async t => {
 	t.deepEqual(
 		await eventToMessage({
 			type: "m.room.message",
@@ -3746,10 +3840,62 @@ test("event2message: image attachments can have a custom description", async t =
 			messagesToEdit: [],
 			messagesToSend: [{
 				username: "cadence [they]",
-				content: "",
+				content: "Cat emoji surrounded by pink hearts",
 				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/azCAhThKTojXSZJRoWwZmhvU",
-				attachments: [{id: "0", description: "Cat emoji surrounded by pink hearts", filename: "cool cat.png"}],
-				pendingFiles: [{name: "cool cat.png", mxc: "mxc://cadence.moe/IvxVJFLEuksCNnbojdSIeEvn"}]
+				attachments: [{id: "0", filename: "cool cat.png"}],
+				pendingFiles: [{name: "cool cat.png", mxc: "mxc://cadence.moe/IvxVJFLEuksCNnbojdSIeEvn"}],
+				allowed_mentions: {
+					parse: ["users", "roles"]
+				}
+			}]
+		}
+	)
+})
+
+test("event2message: image attachments can have a formatted caption", async t => {
+	t.deepEqual(
+		await eventToMessage({
+			content: {
+				body: "this event has `formatting`",
+				filename: "5740.jpg",
+				format: "org.matrix.custom.html",
+				formatted_body: "this event has <code>formatting</code>",
+				info: {
+					h: 1340,
+					mimetype: "image/jpeg",
+					size: 226689,
+					thumbnail_info: {
+						h: 670,
+						mimetype: "image/jpeg",
+						size: 80157,
+						w: 540
+					},
+					thumbnail_url: "mxc://thomcat.rocks/XhLsOCDBYyearsLQgUUrbAvw",
+					w: 1080,
+					"xyz.amorgan.blurhash": "KHJQG*55ic-.}?0M58J.9v"
+				},
+				msgtype: "m.image",
+				url: "mxc://thomcat.rocks/RTHsXmcMPXmuHqVNsnbKtRbh"
+			},
+			origin_server_ts: 1740607766895,
+			sender: "@cadence:cadence.moe",
+			type: "m.room.message",
+			event_id: "$NqNqVgukiQm1nynm9vIr9FIq31hZpQ3udOd7cBIW46U",
+			room_id: "!BnKuBPCvyfOkhcUjEu:cadence.moe"
+		}),
+		{
+			ensureJoined: [],
+			messagesToDelete: [],
+			messagesToEdit: [],
+			messagesToSend: [{
+				username: "cadence [they]",
+				content: "this event has `formatting`",
+				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/azCAhThKTojXSZJRoWwZmhvU",
+				attachments: [{id: "0", filename: "5740.jpg"}],
+				pendingFiles: [{name: "5740.jpg", mxc: "mxc://thomcat.rocks/RTHsXmcMPXmuHqVNsnbKtRbh"}],
+				allowed_mentions: {
+					parse: ["users", "roles"]
+				}
 			}]
 		}
 	)
@@ -3798,7 +3944,7 @@ test("event2message: encrypted image attachments work", async t => {
 				username: "cadence [they]",
 				content: "",
 				avatar_url: "https://bridge.example.org/download/matrix/cadence.moe/azCAhThKTojXSZJRoWwZmhvU",
-				attachments: [{id: "0", description: undefined, filename: "image.png"}],
+				attachments: [{id: "0", filename: "image.png"}],
 				pendingFiles: [{
 					name: "image.png",
 					mxc: "mxc://heyquark.com/LOGkUTlVFrqfiExlGZNgCJJX",
@@ -3810,6 +3956,91 @@ test("event2message: encrypted image attachments work", async t => {
 	)
 })
 
+test("event2message: evil encrypted image attachment works", async t => {
+	t.deepEqual(
+		await eventToMessage({
+			sender: "@austin:tchncs.de",
+			type: "m.room.message",
+			content: {
+				body: "Screenshot 2025-06-29 at 13.36.46.png",
+				file: {
+					hashes: {
+						sha256: "Vh1apd8wSFu/BpUdQbIrKUzFB0Uu+l1octgZL+aVGTQ"
+					},
+					iv: "sd33K7pSZNMAAAAAAAAAAA",
+					key: {
+						alg: "A256CTR",
+						ext: true,
+						k: "-nyqk1eqI-g-ND59P9qHp310-Qyc2A5gSAYm1BxopSg",
+						key_ops: [
+							"encrypt",
+							"decrypt"
+						],
+						kty: "oct"
+					},
+					url: "mxc://tchncs.de/eac5f83fa97cd74062daf75dfa04d6e5356897281939377544214085632",
+					v: "v2"
+				},
+				info: {
+					h: 682,
+					mimetype: "image/png",
+					"org.matrix.msc4230.is_animated": false,
+					size: 1813154,
+					thumbnail_file: {
+						hashes: {
+							sha256: "o3xykQwfsTUf5Y8qP5fjT7qBv5lAT3rtkmPpise5eQw"
+						},
+						iv: "SNxIZsJkju4AAAAAAAAAAA",
+						key: {
+							alg: "A256CTR",
+							ext: true,
+							k: "CcibYjzzSDexOWBbcBh_kCDiLibg8vUZthz5CnxV0es",
+							key_ops: [
+								"encrypt",
+								"decrypt"
+							],
+							kty: "oct"
+						},
+						url: "mxc://tchncs.de/ecd811d913ed1b240ebfc81517a5de2c3a1e9d401939377537079574528",
+						v: "v2"
+					},
+					thumbnail_info: {
+						h: 600,
+						mimetype: "image/png",
+						size: 451773,
+						w: 507
+					},
+					thumbnail_url: null,
+					w: 577,
+					"xyz.amorgan.blurhash": "TqN1Ais=t1~qRjWFxURiWCM{ofof"
+				},
+				"m.mentions": {},
+				msgtype: "m.image",
+				url: null
+			},
+			event_id: "$UKMbzTlqlyLYN78utVEtiivABFvOe39nx5trHwqNmeQ",
+			room_id: "!iSyXgNxQcEuXoXpsSn:pussthecat.org"
+		}),
+		{
+			ensureJoined: [],
+			messagesToDelete: [],
+			messagesToEdit: [],
+			messagesToSend: [{
+				username: "Austin Huang",
+				content: "",
+				avatar_url: "https://bridge.example.org/download/matrix/tchncs.de/090a2b5e07eed2f71e84edad5207221e6c8f8b8e",
+				attachments: [{id: "0", filename: "Screenshot 2025-06-29 at 13.36.46.png"}],
+				pendingFiles: [{
+					name: "Screenshot 2025-06-29 at 13.36.46.png",
+					mxc: "mxc://tchncs.de/eac5f83fa97cd74062daf75dfa04d6e5356897281939377544214085632",
+					key: "-nyqk1eqI-g-ND59P9qHp310-Qyc2A5gSAYm1BxopSg",
+					iv: "sd33K7pSZNMAAAAAAAAAAA"
+				}]
+			}]
+		}
+	)
+})
+
 test("event2message: stickers work", async t => {
 	t.deepEqual(
 		await eventToMessage({
@@ -4391,6 +4622,42 @@ test("event2message: @room in the middle of a link is not converted", async t =>
 	)
 })
 
+test("event2message: table", async t => {
+	t.deepEqual(
+		await eventToMessage({
+			type: "m.room.message",
+			sender: "@cadence:cadence.moe",
+			content: {
+				msgtype: "m.text",
+				body: "wrong body",
+				format: "org.matrix.custom.html",
+				formatted_body: "content<table><thead><tr><th>Col 1</th><th>Col 2</th><th>Col 3</th></tr></thead><tbody><tr><th>Apple</th><td>Banana</td><td>Cherry</td></tr><tr><th>Aardvark</th><td>Bee</td><td>Crocodile</td></tr><tr><td>Argon</td><td>Boron</td><td>Carbon</td></tr></tbody></table>more content"
+			},
+			room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe",
+			event_id: "$SiXetU9h9Dg-M9Frcw_C6ahnoXZ3QPZe3MVJR5tcB9A"
+		}),
+		{
+			messagesToDelete: [],
+			messagesToEdit: [],
+			messagesToSend: [{
+				username: "cadence [they]",
+				content: "content```"
+				+ "\nCol 1     Col 2   Col 3  "
+				+ "\n---------------------------"
+				+ "\nApple     Banana  Cherry "
+				+ "\nAardvark  Bee     Crocodile"
+				+ "\nArgon     Boron   Carbon   ```"
+				+ "more content",
+				avatar_url: undefined,
+				allowed_mentions: {
+					parse: ["users", "roles"]
+				}
+			}],
+			ensureJoined: []
+		}
+	)
+})
+
 slow()("event2message: unknown emoji at the end is reuploaded as a sprite sheet", async t => {
 	const messages = await eventToMessage({
 		type: "m.room.message",
@@ -4477,7 +4744,7 @@ slow()("event2message: all unknown chess emojis are reuploaded as a sprite sheet
 			formatted_body: "testing <img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/lHfmJpzgoNyNtYHdAmBHxXix\" title=\":chess_good_move:\" alt=\":chess_good_move:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/MtRdXixoKjKKOyHJGWLsWLNU\" title=\":chess_incorrect:\" alt=\":chess_incorrect:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/HXfFuougamkURPPMflTJRxGc\" title=\":chess_blund:\" alt=\":chess_blund:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/ikYKbkhGhMERAuPPbsnQzZiX\" title=\":chess_brilliant_move:\" alt=\":chess_brilliant_move:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/AYPpqXzVJvZdzMQJGjioIQBZ\" title=\":chess_blundest:\" alt=\":chess_blundest:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/UVuzvpVUhqjiueMxYXJiFEAj\" title=\":chess_draw_black:\" alt=\":chess_draw_black:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/lHfmJpzgoNyNtYHdAmBHxXix\" title=\":chess_good_move:\" alt=\":chess_good_move:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/MtRdXixoKjKKOyHJGWLsWLNU\" title=\":chess_incorrect:\" alt=\":chess_incorrect:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/HXfFuougamkURPPMflTJRxGc\" title=\":chess_blund:\" alt=\":chess_blund:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/ikYKbkhGhMERAuPPbsnQzZiX\" title=\":chess_brilliant_move:\" alt=\":chess_brilliant_move:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/AYPpqXzVJvZdzMQJGjioIQBZ\" title=\":chess_blundest:\" alt=\":chess_blundest:\"><img data-mx-emoticon height=\"32\" src=\"mxc://cadence.moe/UVuzvpVUhqjiueMxYXJiFEAj\" title=\":chess_draw_black:\" alt=\":chess_draw_black:\">"
 		},
 		event_id: "$Me6iE8C8CZyrDEOYYrXKSYRuuh_25Jj9kZaNrf7LKr4",
-		room_id: "!maggESguZBqGBZtSnr:cadence.moe"
+		room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
 	}, {}, {mxcDownloader: mockGetAndConvertEmoji})
 	const testResult = {
 		content: messages.messagesToSend[0].content,

src/m2d/converters/utils.js

@@ -217,12 +217,12 @@ async function getViaServersQuery(roomID, api) {
  * @see https://matrix.org/blog/2024/06/20/matrix-v1.11-release/ implementation details
  * @see https://www.sqlite.org/fileformat2.html#record_format SQLite integer field size
  * @param {string} mxc
- * @returns {string?}
+ * @returns {string | undefined}
  */
 function getPublicUrlForMxc(mxc) {
 	assert(hasher, "xxhash is not ready yet")
 	const mediaParts = mxc?.match(/^mxc:\/\/([^/]+)\/(\w+)$/)
-	if (!mediaParts) return null
+	if (!mediaParts) return undefined
 
 	const serverAndMediaID = `${mediaParts[1]}/${mediaParts[2]}`
 	const unsignedHash = hasher.h64(serverAndMediaID)

src/m2d/event-dispatcher.js

@@ -22,43 +22,132 @@ const matrixCommandHandler = sync.require("../matrix/matrix-command-handler")
 const utils = sync.require("./converters/utils")
 /** @type {import("../matrix/api")}) */
 const api = sync.require("../matrix/api")
+/** @type {import("../d2m/actions/create-room")} */
+const createRoom = sync.require("../d2m/actions/create-room")
 const {reg} = require("../matrix/read-registration")
 
 let lastReportedEvent = 0
 
+/**
+ * This function is adapted from Evan Kaufman's fantastic work.
+ * The original function and my adapted function are both MIT licensed.
+ * @url https://github.com/EvanK/npm-loggable-error/
+ * @param {number} [depth]
+ * @returns {string}
+*/
+function stringifyErrorStack(err, depth = 0) {
+	let collapsed = " ".repeat(depth);
+	if (!(err instanceof Error)) {
+		return collapsed + err
+	}
+
+	// add full stack trace if one exists, otherwise convert to string
+	let stackLines = String(err?.stack ?? err).replace(/^/gm, " ".repeat(depth)).trim().split("\n")
+	let cloudstormLine = stackLines.findIndex(l => l.includes("/node_modules/cloudstorm/"))
+	if (cloudstormLine !== -1) {
+		stackLines = stackLines.slice(0, cloudstormLine - 2)
+	}
+	collapsed += stackLines.join("\n")
+
+	const props = Object.getOwnPropertyNames(err).filter(p => !["message", "stack"].includes(p))
+
+	// only break into object notation if we have additional props to dump
+	if (props.length) {
+		const dedent = " ".repeat(depth);
+		const indent = " ".repeat(depth + 2);
+
+		collapsed += " {\n";
+
+		// loop and print each (indented) prop name
+		for (let property of props) {
+			collapsed += `${indent}[${property}]: `;
+
+			// if another error object, stringify it too
+			if (err[property] instanceof Error) {
+				collapsed += stringifyErrorStack(err[property], depth + 2).trimStart();
+			}
+			// otherwise stringify as JSON
+			else {
+				collapsed += JSON.stringify(err[property]);
+			}
+
+			collapsed += "\n";
+		}
+
+		collapsed += `${dedent}}\n`;
+	}
+
+	return collapsed;
+}
+
+/**
+ * @param {string} roomID
+ * @param {"Discord" | "Matrix"} source
+ * @param {any} type
+ * @param {any} e
+ * @param {any} payload
+ */
+async function sendError(roomID, source, type, e, payload) {
+	console.error(`Error while processing a ${type} ${source} event:`)
+	console.error(e)
+	console.dir(payload, {depth: null})
+
+	if (Date.now() - lastReportedEvent < 5000) return null
+	lastReportedEvent = Date.now()
+
+	let errorIntroLine = e.toString()
+	if (e.cause) {
+		errorIntroLine += ` (cause: ${e.cause})`
+	}
+
+	const builder = new utils.MatrixStringBuilder()
+
+	const cloudflareErrorTitle = errorIntroLine.match(/<!DOCTYPE html>.*?<title>discord\.com \| ([^<]*)<\/title>/s)?.[1]
+	if (cloudflareErrorTitle) {
+		builder.addLine(
+			`\u26a0 Matrix event not delivered to Discord. Discord might be down right now. Cloudflare error: ${cloudflareErrorTitle}`,
+			`\u26a0 <strong>Matrix event not delivered to Discord</strong><br>Discord might be down right now. Cloudflare error: ${cloudflareErrorTitle}`
+		)
+	} else {
+		// What
+		const what = source === "Discord" ? "Bridged event from Discord not delivered" : "Matrix event not delivered to Discord"
+		builder.addLine(`\u26a0 ${what}`, `\u26a0 <strong>${what}</strong>`)
+
+		// Who
+		builder.addLine(`Event type: ${type}`)
+
+		// Why
+		builder.addLine(errorIntroLine)
+
+		// Where
+		const stack = stringifyErrorStack(e)
+		builder.addLine(`Error trace:\n${stack}`, `<details><summary>Error trace</summary><pre>${stack}</pre></details>`)
+
+		// How
+		builder.addLine("", `<details><summary>Original payload</summary><pre>${util.inspect(payload, false, 4, false)}</pre></details>`)
+	}
+
+	// Send
+	try {
+		await api.sendEvent(roomID, "m.room.message", {
+			...builder.get(),
+			"moe.cadence.ooye.error": {
+				source: source.toLowerCase(),
+				payload
+			},
+			"m.mentions": {
+				user_ids: ["@cadence:cadence.moe"]
+			}
+		})
+	} catch (e) {}
+}
+
 function guard(type, fn) {
 	return async function(event, ...args) {
 		try {
 			return await fn(event, ...args)
 		} catch (e) {
-			console.error("hit event-dispatcher's error handler with this exception:")
-			console.error(e) // TODO: also log errors into a file or into the database, maybe use a library for this? or just wing it?
-			console.error(`while handling this ${type} gateway event:`)
-			console.dir(event, {depth: null})
-
-			if (Date.now() - lastReportedEvent < 5000) return
-			lastReportedEvent = Date.now()
-
-			let stackLines = e.stack.split("\n")
-			api.sendEvent(event.room_id, "m.room.message", {
-				msgtype: "m.text",
-				body: "\u26a0 Matrix event not delivered to Discord. See formatted content for full details.",
-				format: "org.matrix.custom.html",
-				formatted_body: "\u26a0 <strong>Matrix event not delivered to Discord</strong>"
-					+ `<br>Event type: ${type}`
-					+ `<br>${e.toString()}`
-					+ `<br><details><summary>Error trace</summary>`
-					+ `<pre>${stackLines.join("\n")}</pre></details>`
-					+ `<details><summary>Original payload</summary>`
-					+ `<pre>${util.inspect(event, false, 4, false)}</pre></details>`,
-				"moe.cadence.ooye.error": {
-					source: "matrix",
-					payload: event
-				},
-				"m.mentions": {
-					user_ids: ["@cadence:cadence.moe"]
-				}
-			})
+			await sendError(event.room_id, "Matrix", type, e, event)
 		}
 	}
 }
@@ -90,7 +179,7 @@ async function onRetryReactionAdd(reactionEvent) {
 	}
 
 	// Redact the error to stop people from executing multiple retries
-	api.redactEvent(roomID, event.event_id)
+	await api.redactEvent(roomID, event.event_id)
 }
 
 sync.addTemporaryListener(as, "type:m.room.message", guard("m.room.message",
@@ -100,6 +189,7 @@ sync.addTemporaryListener(as, "type:m.room.message", guard("m.room.message",
 async event => {
 	if (utils.eventSenderIsFromDiscord(event.sender)) return
 	const messageResponses = await sendEvent.sendEvent(event)
+	if (!messageResponses.length) return
 	if (event.type === "m.room.message" && event.content.msgtype === "m.text") {
 		// @ts-ignore
 		await matrixCommandHandler.execute(event)
@@ -164,6 +254,20 @@ async event => {
 	db.prepare("UPDATE channel_room SET nick = ? WHERE room_id = ?").run(name, event.room_id)
 }))
 
+sync.addTemporaryListener(as, "type:m.room.topic", guard("m.room.topic",
+/**
+ * @param {Ty.Event.StateOuter<Ty.Event.M_Room_Topic>} event
+ */
+async event => {
+	if (event.state_key !== "") return
+	if (utils.eventSenderIsFromDiscord(event.sender)) return
+	const customTopic = +!!event.content.topic
+	const row = select("channel_room", ["channel_id", "custom_topic"], {room_id: event.room_id}).get()
+	if (!row) return
+	if (customTopic !== row.custom_topic) db.prepare("UPDATE channel_room SET custom_topic = ? WHERE channel_id = ?").run(customTopic, row.channel_id)
+	if (!customTopic) await createRoom.syncRoom(row.channel_id) // if it's cleared we should reset it to whatever's on discord
+}))
+
 sync.addTemporaryListener(as, "type:m.room.pinned_events", guard("m.room.pinned_events",
 /**
  * @param {Ty.Event.StateOuter<Ty.Event.M_Room_PinnedEvents>} event
@@ -192,25 +296,75 @@ async event => {
 	await api.ackEvent(event)
 }))
 
+function getFromInviteRoomState(inviteRoomState, nskey, key) {
+	if (!Array.isArray(inviteRoomState)) return null
+	for (const event of inviteRoomState) {
+		if (event.type === nskey && event.state_key === "") {
+			return event.content[key]
+		}
+	}
+	return null
+}
+
+sync.addTemporaryListener(as, "type:m.space.child", guard("m.space.child",
+/**
+ * @param {Ty.Event.StateOuter<Ty.Event.M_Space_Child>} event
+ */
+async event => {
+	if (Array.isArray(event.content.via) && event.content.via.length) { // space child is being added
+		await api.joinRoom(event.state_key).catch(() => {}) // try to join if able, it's okay if it doesn't want, bot will still respond to invites
+	}
+}))
+
 sync.addTemporaryListener(as, "type:m.room.member", guard("m.room.member",
 /**
  * @param {Ty.Event.StateOuter<Ty.Event.M_Room_Member>} event
  */
 async event => {
 	if (event.state_key[0] !== "@") return
+
+	if (event.content.membership === "invite" && event.state_key === `@${reg.sender_localpart}:${reg.ooye.server_name}`) {
+		// We were invited to a room. We should join, and register the invite details for future reference in web.
+		const name = getFromInviteRoomState(event.unsigned?.invite_room_state, "m.room.name", "name")
+		const topic = getFromInviteRoomState(event.unsigned?.invite_room_state, "m.room.topic", "topic")
+		const avatar = getFromInviteRoomState(event.unsigned?.invite_room_state, "m.room.avatar", "url")
+		const creationType = getFromInviteRoomState(event.unsigned?.invite_room_state, "m.room.create", "type")
+		if (!name) return await api.leaveRoomWithReason(event.room_id, "Please only invite me to rooms that have a name/avatar set. Update the room details and reinvite!")
+		await api.joinRoom(event.room_id)
+		db.prepare("INSERT OR IGNORE INTO invite (mxid, room_id, type, name, topic, avatar) VALUES (?, ?, ?, ?, ?, ?)").run(event.sender, event.room_id, creationType, name, topic, avatar)
+		if (avatar) utils.getPublicUrlForMxc(avatar) // make sure it's available in the media_proxy allowed URLs
+	}
+
 	if (utils.eventSenderIsFromDiscord(event.state_key)) return
+
 	if (event.content.membership === "leave" || event.content.membership === "ban") {
 		// Member is gone
 		db.prepare("DELETE FROM member_cache WHERE room_id = ? and mxid = ?").run(event.room_id, event.state_key)
-	} else {
-		// Member is here
-		db.prepare("INSERT INTO member_cache (room_id, mxid, displayname, avatar_url) VALUES (?, ?, ?, ?) ON CONFLICT DO UPDATE SET displayname = ?, avatar_url = ?")
-			.run(
-				event.room_id, event.state_key,
-				event.content.displayname || null, event.content.avatar_url || null,
-				event.content.displayname || null, event.content.avatar_url || null
-			)
+
+		// Unregister room's use as a direct chat if the bot itself left
+		const bot = `@${reg.sender_localpart}:${reg.ooye.server_name}`
+		if (event.state_key === bot) {
+			db.prepare("DELETE FROM direct WHERE room_id = ?").run(event.room_id)
+		}
 	}
+
+	const exists = select("channel_room", "room_id", {room_id: event.room_id}) ?? select("guild_space", "space_id", {space_id: event.room_id})
+	if (!exists) return // don't cache members in unbridged rooms
+
+	// Member is here
+	let powerLevel = 0
+	try {
+		/** @type {Ty.Event.M_Power_Levels} */
+		const powerLevelsEvent = await api.getStateEvent(event.room_id, "m.room.power_levels", "")
+		powerLevel = powerLevelsEvent.users?.[event.state_key] ?? powerLevelsEvent.users_default ?? 0
+	} catch (e) {}
+	const displayname = event.content.displayname || null
+	const avatar_url = event.content.avatar_url
+	db.prepare("INSERT INTO member_cache (room_id, mxid, displayname, avatar_url, power_level) VALUES (?, ?, ?, ?, ?) ON CONFLICT DO UPDATE SET displayname = ?, avatar_url = ?, power_level = ?").run(
+		event.room_id, event.state_key,
+		displayname, avatar_url, powerLevel,
+		displayname, avatar_url, powerLevel
+	)
 }))
 
 sync.addTemporaryListener(as, "type:m.room.power_levels", guard("m.room.power_levels",
@@ -225,3 +379,6 @@ async event => {
 		db.prepare("UPDATE member_cache SET power_level = ? WHERE room_id = ? AND mxid = ?").run(newPower[mxid] || 0, event.room_id, mxid)
 	}
 }))
+
+module.exports.stringifyErrorStack = stringifyErrorStack
+module.exports.sendError = sendError

src/m2d/event-dispatcher.test.js

@@ -0,0 +1,23 @@
+// @ts-check
+
+const {test} = require("supertape")
+const {stringifyErrorStack} = require("./event-dispatcher")
+
+test("stringify error stack: works", t => {
+	function a() {
+		const e = new Error("message", {cause: new Error("inner")})
+		// @ts-ignore
+		e.prop = 2.1
+		throw e
+	}
+	try {
+		a()
+		t.fail("shouldn't get here")
+	} catch (e) {
+		const str = stringifyErrorStack(e)
+		t.match(str, /^Error: message$/m)
+		t.match(str, /^    at a \(.*event-dispatcher\.test\.js/m)
+		t.match(str, /^  \[cause\]: Error: inner$/m)
+		t.match(str, /^  \[prop\]: 2.1$/m)
+	}
+})

src/matrix/api.js

@@ -2,8 +2,7 @@
 
 const Ty = require("../types")
 const assert = require("assert").strict
-
-const fetch = require("node-fetch").default
+const streamWeb = require("stream/web")
 
 const passthrough = require("../passthrough")
 const {sync} = passthrough
@@ -82,6 +81,16 @@ async function leaveRoom(roomID, mxid) {
 	await mreq.mreq("POST", path(`/client/v3/rooms/${roomID}/leave`, mxid), {})
 }
 
+/**
+ * @param {string} roomID
+ * @param {string} reason
+ * @param {string} [mxid]
+ */
+async function leaveRoomWithReason(roomID, reason, mxid) {
+	console.log(`[api] leave: ${roomID}: ${mxid}, because ${reason}`)
+	await mreq.mreq("POST", path(`/client/v3/rooms/${roomID}/leave`, mxid), {reason})
+}
+
 /**
  * @param {string} roomID
  * @param {string} eventID
@@ -172,6 +181,23 @@ async function getFullHierarchy(roomID) {
 	return rooms
 }
 
+/**
+ * Like `getFullHierarchy` but reveals a page at a time through an async iterator.
+ * @param {string} roomID
+ */
+async function* generateFullHierarchy(roomID) {
+	/** @type {string | undefined} */
+	let nextBatch = undefined
+	do {
+		/** @type {Ty.HierarchyPagination<Ty.R.Hierarchy>} */
+		const res = await getHierarchy(roomID, {from: nextBatch})
+		for (const room of res.rooms) {
+			yield room
+		}
+		nextBatch = res.next_batch
+	} while (nextBatch)
+}
+
 /**
  * @param {string} roomID
  * @param {string} eventID
@@ -282,21 +308,33 @@ async function profileSetAvatarUrl(mxid, avatar_url) {
  * Set a user's power level within a room.
  * @param {string} roomID
  * @param {string} mxid
- * @param {number} power
+ * @param {number} newPower
  */
-async function setUserPower(roomID, mxid, power) {
+async function setUserPower(roomID, mxid, newPower) {
 	assert(roomID[0] === "!")
 	assert(mxid[0] === "@")
 	// Yes there's no shortcut https://github.com/matrix-org/matrix-appservice-bridge/blob/2334b0bae28a285a767fe7244dad59f5a5963037/src/components/intent.ts#L352
-	const powerLevels = await getStateEvent(roomID, "m.room.power_levels", "")
-	powerLevels.users = powerLevels.users || {}
-	if (power != null) {
-		powerLevels.users[mxid] = power
+	const power = await getStateEvent(roomID, "m.room.power_levels", "")
+	power.users = power.users || {}
+
+	// Check if it has really changed to avoid sending a useless state event
+	// (Can't diff kstate here because of (a) circular imports (b) kstate has special behaviour diffing power levels)
+	const oldPowerLevel = power.users?.[mxid] ?? power.users_default ?? 0
+	if (oldPowerLevel === newPower) return
+
+	// Bridge bot can't demote equal power users, so need to decide which user will send the event
+	const botPowerLevel = power.users?.[`@${reg.sender_localpart}:${reg.ooye.server_name}`] ?? power.users_default ?? 0
+	const eventSender = oldPowerLevel >= botPowerLevel ? mxid : undefined
+
+	// Update the event content
+	if (newPower == null || newPower === (power.users_default ?? 0)) {
+		delete power.users[mxid]
 	} else {
-		delete powerLevels.users[mxid]
+		power.users[mxid] = newPower
 	}
-	await sendState(roomID, "m.room.power_levels", "", powerLevels)
-	return powerLevels
+
+	await sendState(roomID, "m.room.power_levels", "", power, eventSender)
+	return power
 }
 
 /**
@@ -334,17 +372,21 @@ async function ping() {
 
 /**
  * @param {string} mxc
- * @param {fetch.RequestInit} [init]
+ * @param {RequestInit} [init]
+ * @return {Promise<Response & {body: streamWeb.ReadableStream<Uint8Array>}>}
  */
-function getMedia(mxc, init = {}) {
+async function getMedia(mxc, init = {}) {
 	const mediaParts = mxc?.match(/^mxc:\/\/([^/]+)\/(\w+)$/)
 	assert(mediaParts)
-	return fetch(`${mreq.baseUrl}/client/v1/media/download/${mediaParts[1]}/${mediaParts[2]}`, {
+	const res = await fetch(`${mreq.baseUrl}/client/v1/media/download/${mediaParts[1]}/${mediaParts[2]}`, {
 		headers: {
 			Authorization: `Bearer ${reg.as_token}`
 		},
 		...init
 	})
+	assert(res.body)
+	// @ts-ignore
+	return res
 }
 
 /**
@@ -377,12 +419,50 @@ async function getAlias(alias) {
 	return root.room_id
 }
 
+/**
+ * @param {string} type namespaced event type, e.g. m.direct
+ * @param {string} [mxid] you
+ * @returns the *content* of the account data "event"
+ */
+async function getAccountData(type, mxid) {
+	if (!mxid) mxid = `@${reg.sender_localpart}:${reg.ooye.server_name}`
+	const root = await mreq.mreq("GET", `/client/v3/user/${mxid}/account_data/${type}`)
+	return root
+}
+
+/**
+ * @param {string} type namespaced event type, e.g. m.direct
+ * @param {any} content whatever you want
+ * @param {string} [mxid] you
+ */
+async function setAccountData(type, content, mxid) {
+	if (!mxid) mxid = `@${reg.sender_localpart}:${reg.ooye.server_name}`
+	await mreq.mreq("PUT", `/client/v3/user/${mxid}/account_data/${type}`, content)
+}
+
+/**
+ * @param {{presence: "online" | "offline" | "unavailable", status_msg?: string}} data
+ * @param {string} mxid
+ */
+async function setPresence(data, mxid) {
+	await mreq.mreq("PUT", path(`/client/v3/presence/${mxid}/status`, mxid), data)
+}
+
+/**
+ * @param {string} mxid
+ * @returns {Promise<{displayname?: string, avatar_url?: string}>}
+ */
+function getProfile(mxid) {
+	return mreq.mreq("GET", `/client/v3/profile/${mxid}`)
+}
+
 module.exports.path = path
 module.exports.register = register
 module.exports.createRoom = createRoom
 module.exports.joinRoom = joinRoom
 module.exports.inviteToRoom = inviteToRoom
 module.exports.leaveRoom = leaveRoom
+module.exports.leaveRoomWithReason = leaveRoomWithReason
 module.exports.getEvent = getEvent
 module.exports.getEventForTimestamp = getEventForTimestamp
 module.exports.getAllState = getAllState
@@ -391,6 +471,7 @@ module.exports.getJoinedMembers = getJoinedMembers
 module.exports.getMembers = getMembers
 module.exports.getHierarchy = getHierarchy
 module.exports.getFullHierarchy = getFullHierarchy
+module.exports.generateFullHierarchy = generateFullHierarchy
 module.exports.getRelations = getRelations
 module.exports.getFullRelations = getFullRelations
 module.exports.sendState = sendState
@@ -406,3 +487,7 @@ module.exports.getMedia = getMedia
 module.exports.sendReadReceipt = sendReadReceipt
 module.exports.ackEvent = ackEvent
 module.exports.getAlias = getAlias
+module.exports.getAccountData = getAccountData
+module.exports.setAccountData = setAccountData
+module.exports.setPresence = setPresence
+module.exports.getProfile = getProfile

src/matrix/file.js

@@ -1,8 +1,9 @@
 // @ts-check
 
-const fetch = require("node-fetch").default
-
 const passthrough = require("../passthrough")
+const {reg, writeRegistration} = require("./read-registration.js")
+const Ty = require("../types")
+
 const {sync, db, select} = passthrough
 /** @type {import("./mreq")} */
 const mreq = sync.require("./mreq")
@@ -46,11 +47,8 @@ async function uploadDiscordFileToMxc(path) {
 		return existingFromDb
 	}
 
-	// Download from Discord
-	const promise = fetch(url, {}).then(/** @param {import("node-fetch").Response} res */ async res => {
-		// Upload to Matrix
-		const root = await module.exports._actuallyUploadDiscordFileToMxc(urlNoExpiry, res)
-
+	// Download from Discord and upload to Matrix
+	const promise = module.exports._actuallyUploadDiscordFileToMxc(url).then(root => {
 		// Store relationship in database
 		db.prepare("INSERT INTO file (discord_url, mxc_url) VALUES (?, ?)").run(urlNoExpiry, root.content_uri)
 		inflight.delete(urlNoExpiry)
@@ -62,15 +60,33 @@ async function uploadDiscordFileToMxc(path) {
 	return promise
 }
 
-async function _actuallyUploadDiscordFileToMxc(url, res) {
-	const body = res.body
-	/** @type {import("../types").R.FileUploaded} */
-	const root = await mreq.mreq("POST", "/media/v3/upload", body, {
-		headers: {
-			"Content-Type": res.headers.get("content-type")
+/**
+ * @param {string} url
+ * @returns {Promise<Ty.R.FileUploaded>}
+ */
+async function _actuallyUploadDiscordFileToMxc(url) {
+	const res = await fetch(url, {})
+	try {
+		/** @type {Ty.R.FileUploaded} */
+		const root = await mreq.mreq("POST", "/media/v3/upload", res.body, {
+			headers: {
+				"Content-Type": res.headers.get("content-type")
+			}
+		})
+		return root
+	} catch (e) {
+		if (e instanceof mreq.MatrixServerError && e.data.error?.includes("Content-Length") && !reg.ooye.content_length_workaround) {
+			reg.ooye.content_length_workaround = true
+			const root = await _actuallyUploadDiscordFileToMxc(url)
+			console.error("OOYE cannot stream uploads to Synapse. The `content_length_workaround` option"
+				+ "\nhas been activated in registration.yaml, which works around the problem, but"
+				+ "\nhalves the speed of bridging d->m files. A better way to resolve this problem"
+				+ "\nis to run an nginx reverse proxy to Synapse and re-run OOYE setup.")
+			writeRegistration(reg)
+			return root
 		}
-	})
-	return root
+		throw e
+	}
 }
 
 function guildIcon(guild) {
@@ -82,8 +98,8 @@ function userAvatar(user) {
 }
 
 function memberAvatar(guildID, user, member) {
-	if (!member.avatar) return userAvatar(user)
-	return `/guilds/${guildID}/users/${user.id}/avatars/${member.avatar}.png?size=${IMAGE_SIZE}`
+	if (!member?.avatar) return userAvatar(user)
+	return `/guilds/${guildID}/users/${user.id}/avatars/${member?.avatar}.png?size=${IMAGE_SIZE}`
 }
 
 function emoji(emojiID, animated) {
@@ -93,18 +109,17 @@ function emoji(emojiID, animated) {
 }
 
 const stickerFormat = new Map([
-	[1, {label: "PNG", ext: "png", mime: "image/png"}],
-	[2, {label: "APNG", ext: "png", mime: "image/apng"}],
-	[3, {label: "LOTTIE", ext: "json", mime: "lottie"}],
-	[4, {label: "GIF", ext: "gif", mime: "image/gif"}]
+	[1, {label: "PNG", ext: "png", mime: "image/png", endpoint: "/stickers/"}],
+	[2, {label: "APNG", ext: "png", mime: "image/apng", endpoint: "/stickers/"}],
+	[3, {label: "LOTTIE", ext: "json", mime: "lottie", endpoint: "/stickers/"}],
+	[4, {label: "GIF", ext: "gif", mime: "image/gif", endpoint: "https://media.discordapp.net/stickers/"}]
 ])
 
 /** @param {{id: string, format_type: number}} sticker */
 function sticker(sticker) {
 	const format = stickerFormat.get(sticker.format_type)
 	if (!format) throw new Error(`No such format ${sticker.format_type} for sticker ${JSON.stringify(sticker)}`)
-	const ext = format.ext
-	return `/stickers/${sticker.id}.${ext}`
+	return `${format.endpoint}${sticker.id}.${format.ext}`
 }
 
 module.exports.DISCORD_IMAGES_BASE = DISCORD_IMAGES_BASE

src/matrix/kstate.js

@@ -87,6 +87,12 @@ function diffKState(actual, target) {
 				diff[key] = temp
 			}
 
+		} else if (key === "chat.schildi.hide_ui/read_receipts") {
+			// Special handling: don't add this key if it's new. Do overwrite if already present.
+			if (key in actual) {
+				diff[key] = target[key]
+			}
+
 		} else if (key in actual) {
 			// diff
 			if (!isDeepStrictEqual(actual[key], target[key])) {

src/matrix/kstate.test.js

@@ -234,3 +234,31 @@ test("diffKState: kstate keys must contain a slash separator", t => {
 	, /does not contain a slash separator/)
 	t.pass()
 })
+
+test("diffKState: don't add hide_ui when not present", t => {
+	test("diffKState: detects new properties", t => {
+		t.deepEqual(
+			diffKState({
+			}, {
+				"chat.schildi.hide_ui/read_receipts/": {}
+			}),
+			{
+			}
+		)
+	})
+})
+
+test("diffKState: overwriten hide_ui when present", t => {
+	test("diffKState: detects new properties", t => {
+		t.deepEqual(
+			diffKState({
+				"chat.schildi.hide_ui/read_receipts/": {hidden: true}
+			}, {
+				"chat.schildi.hide_ui/read_receipts/": {}
+			}),
+			{
+				"chat.schildi.hide_ui/read_receipts/": {}
+			}
+		)
+	})
+})

src/matrix/matrix-command-handler.js

@@ -224,7 +224,7 @@ const commands = [{
 							.png()
 							.toBuffer({resolveWithObject: true})
 						console.log(`uploading emoji ${resizeOutput.data.length} bytes to :${e.name}:`)
-						const emoji = await discord.snow.guildAssets.createEmoji(guildID, {name: e.name, image: "data:image/png;base64," + resizeOutput.data.toString("base64")})
+						await discord.snow.assets.createGuildEmoji(guildID, {name: e.name, image: "data:image/png;base64," + resizeOutput.data.toString("base64")})
 					}
 					api.sendEvent(event.room_id, "m.room.message", {
 						...ctx,

src/matrix/mreq.js

@@ -1,12 +1,11 @@
 // @ts-check
 
-const fetch = require("node-fetch").default
-const mixin = require("@cloudrac3r/mixin-deep")
 const stream = require("stream")
-const getStream = require("get-stream")
-
-const {reg, writeRegistration} = require("./read-registration.js")
+const streamWeb = require("stream/web")
+const {buffer} = require("stream/consumers")
+const mixin = require("@cloudrac3r/mixin-deep")
 
+const {reg} = require("./read-registration.js")
 const baseUrl = `${reg.ooye.server_origin}/_matrix`
 
 class MatrixServerError extends Error {
@@ -19,43 +18,49 @@ class MatrixServerError extends Error {
 	}
 }
 
+/**
+ * @param {undefined | string | object | streamWeb.ReadableStream | stream.Readable} body
+ * @returns {Promise<string | streamWeb.ReadableStream | stream.Readable | Buffer>}
+ */
+async function _convertBody(body) {
+	if (body == undefined || Object.is(body.constructor, Object)) {
+		return JSON.stringify(body) // almost every POST request is going to follow this one
+	} else if (body instanceof stream.Readable && reg.ooye.content_length_workaround) {
+		return await buffer(body) // content length workaround is set, so convert to buffer. the buffer consumer accepts node streams.
+	} else if (body instanceof stream.Readable) {
+		return stream.Readable.toWeb(body) // native fetch can only consume web streams
+	} else if (body instanceof streamWeb.ReadableStream && reg.ooye.content_length_workaround) {
+		return await buffer(body) // content lenght workaround is set, so convert to buffer. the buffer consumer accepts async iterables, which web streams are.
+	}
+	return body
+}
+
+/* c8 ignore start */
+
 /**
  * @param {string} method
  * @param {string} url
- * @param {any} [body]
+ * @param {string | object | streamWeb.ReadableStream | stream.Readable} [bodyIn]
  * @param {any} [extra]
  */
-async function mreq(method, url, body, extra = {}) {
-	if (body == undefined || Object.is(body.constructor, Object)) {
-		body = JSON.stringify(body)
-	} else if (body instanceof stream.Readable && reg.ooye.content_length_workaround) {
-		body = await getStream.buffer(body)
-	}
+async function mreq(method, url, bodyIn, extra = {}) {
+	const body = await _convertBody(bodyIn)
 
+	/** @type {RequestInit} */
 	const opts = mixin({
 		method,
 		body,
 		headers: {
 			Authorization: `Bearer ${reg.as_token}`
-		}
+		},
+		...(body && {duplex: "half"}), // https://github.com/octokit/request.js/pull/571/files
 	}, extra)
 
-	// console.log(baseUrl + url, opts)
 	const res = await fetch(baseUrl + url, opts)
 	const root = await res.json()
 
 	if (!res.ok || root.errcode) {
-		if (root.error?.includes("Content-Length") && !reg.ooye.content_length_workaround) {
-			reg.ooye.content_length_workaround = true
-			const root = await mreq(method, url, body, extra)
-			console.error("OOYE cannot stream uploads to Synapse. The `content_length_workaround` option"
-				+ "\nhas been activated in registration.yaml, which works around the problem, but"
-				+ "\nhalves the speed of bridging d->m files. A better way to resolve this problem"
-				+ "\nis to run an nginx reverse proxy to Synapse and re-run OOYE setup.")
-			writeRegistration(reg)
-			return root
-		}
-		delete opts.headers.Authorization
+		delete opts.headers?.["Authorization"]
 		throw new MatrixServerError(root, {baseUrl, url, ...opts})
 	}
 	return root
@@ -83,3 +88,4 @@ module.exports.MatrixServerError = MatrixServerError
 module.exports.baseUrl = baseUrl
 module.exports.mreq = mreq
 module.exports.withAccessToken = withAccessToken
+module.exports._convertBody = _convertBody

src/matrix/mreq.test.js

@@ -0,0 +1,47 @@
+// @ts-check
+
+const assert = require("assert")
+const stream = require("stream")
+const streamWeb = require("stream/web")
+const {buffer} = require("stream/consumers")
+const {test} = require("supertape")
+const {_convertBody} = require("./mreq")
+const {reg} = require("./read-registration")
+
+async function *generator() {
+	yield "a"
+	yield "b"
+}
+
+reg.ooye.content_length_workaround = false
+
+test("convert body: converts object to string", async t => {
+	t.equal(await _convertBody({a: "1"}), `{"a":"1"}`)
+})
+
+test("convert body: leaves undefined as undefined", async t => {
+	t.equal(await _convertBody(undefined), undefined)
+})
+
+test("convert body: leaves web readable as web readable", async t => {
+	const webReadable = stream.Readable.toWeb(stream.Readable.from(generator()))
+	t.equal(await _convertBody(webReadable), webReadable)
+})
+
+test("convert body: converts node readable to web readable (for native fetch upload)", async t => {
+	const readable = stream.Readable.from(generator())
+	const webReadable = await _convertBody(readable)
+	assert(webReadable instanceof streamWeb.ReadableStream)
+	t.deepEqual(await buffer(webReadable), Buffer.from("ab"))
+})
+
+test("convert body: converts node readable to buffer", async t => {
+	reg.ooye.content_length_workaround = true
+	const readable = stream.Readable.from(generator())
+	t.deepEqual(await _convertBody(readable), Buffer.from("ab"))
+})
+
+test("convert body: converts web readable to buffer", async t => {
+	const webReadable = stream.Readable.toWeb(stream.Readable.from(generator()))
+	t.deepEqual(await _convertBody(webReadable), Buffer.from("ab"))
+})

src/passthrough.js

@@ -4,7 +4,7 @@
  * @typedef {Object} Passthrough
  * @property {import("repl").REPLServer} repl
  * @property {import("./d2m/discord-client")} discord
- * @property {import("heatsync").default} sync
+ * @property {import("heatsync")} sync
  * @property {import("better-sqlite3/lib/database")} db
  * @property {import("@cloudrac3r/in-your-element").AppService} as
  * @property {import("./db/orm").from} from

src/stdin.js

@@ -5,7 +5,7 @@ const util = require("util")
 const {addbot} = require("../addbot")
 
 const passthrough = require("./passthrough")
-const {discord, sync, db} = passthrough
+const {discord, sync, db, select, from, as} = passthrough
 
 const data = sync.require("../test/data")
 const createSpace = sync.require("./d2m/actions/create-space")
@@ -19,19 +19,18 @@ const eventDispatcher = sync.require("./d2m/event-dispatcher")
 const updatePins = sync.require("./d2m/actions/update-pins")
 const speedbump = sync.require("./d2m/actions/speedbump")
 const ks = sync.require("./matrix/kstate")
+const setPresence = sync.require("./d2m/actions/set-presence")
+const channelWebhook = sync.require("./m2d/actions/channel-webhook")
 const guildID = "112760669178241024"
 
-const extraContext = {}
-
 if (process.stdin.isTTY) {
-	setImmediate(() => { // assign after since old extraContext data will get removed
+	setImmediate(() => {
 		if (!passthrough.repl) {
 			const cli = repl.start({ prompt: "", eval: customEval, writer: s => s })
-			Object.assign(cli.context, extraContext, passthrough)
+			Object.assign(cli.context, passthrough)
 			passthrough.repl = cli
-		} else {
-			Object.assign(passthrough.repl.context, extraContext)
 		}
+		// @ts-ignore
 		sync.addTemporaryListener(passthrough.repl, "exit", () => process.exit())
 	})
 }
@@ -60,9 +59,3 @@ async function customEval(input, _context, _filename, callback) {
 		return callback(null, util.inspect(e, false, 100, true))
 	}
 }
-
-sync.events.once(__filename, () => {
-	for (const key in extraContext) {
-		delete passthrough.repl.context[key]
-	}
-})

src/types.d.ts

@@ -28,6 +28,9 @@ export type AppServiceRegistrationConfig = {
 		content_length_workaround: boolean
 		include_user_id_in_mxid: boolean
 		invite: string[]
+		discord_origin?: string
+		discord_cdn_origin?: string,
+		web_password: string
 	}
 	old_bridge?: {
 		as_token: string
@@ -164,6 +167,8 @@ export namespace Event {
 	export type M_Room_Message_File = {
 		msgtype: "m.file" | "m.image" | "m.video" | "m.audio"
 		body: string
+		format?: "org.matrix.custom.html"
+		formatted_body?: string
 		filename?: string
 		url: string
 		info?: any
@@ -181,6 +186,8 @@ export namespace Event {
 	export type M_Room_Message_Encrypted_File = {
 		msgtype: "m.file" | "m.image" | "m.video" | "m.audio"
 		body: string
+		format?: "org.matrix.custom.html"
+		formatted_body?: string
 		filename?: string
 		file: {
 			url: string
@@ -241,6 +248,10 @@ export namespace Event {
 		name?: string
 	}
 
+	export type M_Room_Topic = {
+		topic?: string
+	}
+
 	export type M_Room_PinnedEvents = {
 		pinned: string[]
 	}
@@ -275,6 +286,11 @@ export namespace Event {
 		users_default?: number
 	}
 
+	export type M_Space_Child = {
+		via?: string[]
+		suggested?: boolean
+	}
+
 	export type M_Reaction = {
 		"m.relates_to": {
 			rel_type: "m.annotation"

src/web/auth.js

@@ -0,0 +1,33 @@
+// @ts-check
+
+const h3 = require("h3")
+const {db} = require("../passthrough")
+const {reg} = require("../matrix/read-registration")
+
+/**
+ * Combined guilds managed by Discord account + Matrix account.
+ * @param {h3.H3Event} event
+ * @returns {Promise<Set<string>>} guild IDs
+ */
+async function getManagedGuilds(event) {
+	const session = await useSession(event)
+	const managed = new Set(session.data.managedGuilds || [])
+	if (session.data.mxid) {
+		const matrixGuilds = db.prepare("SELECT guild_id FROM guild_space INNER JOIN member_cache ON space_id = room_id WHERE mxid = ? AND power_level >= 50").pluck().all(session.data.mxid)
+		for (const id of matrixGuilds) {
+			managed.add(id)
+		}
+	}
+	return managed
+}
+
+/**
+ * @param {h3.H3Event} event
+ * @returns {ReturnType<typeof h3.useSession<{userID?: string, mxid?: string, managedGuilds?: string[], state?: string, selfService?: boolean, password?: string}>>}
+ */
+function useSession(event) {
+	return h3.useSession(event, {password: reg.as_token, maxAge: 365 * 24 * 60 * 60})
+}
+
+module.exports.getManagedGuilds = getManagedGuilds
+module.exports.useSession = useSession

src/web/pug-sync.js

@@ -5,10 +5,13 @@ const fs = require("fs")
 const {join} = require("path")
 const getRelativePath = require("get-relative-path")
 const h3 = require("h3")
-const {defineEventHandler, defaultContentType, setResponseStatus, useSession, getQuery} = h3
+const {defineEventHandler, defaultContentType, setResponseStatus, getQuery} = h3
 const {compileFile} = require("@cloudrac3r/pug")
+const pretty = process.argv.join(" ").includes("test")
 
-const {reg} = require("../matrix/read-registration")
+const {sync} = require("../passthrough")
+/** @type {import("./auth")} */
+const auth = sync.require("./auth")
 
 // Pug
 
@@ -31,16 +34,24 @@ function render(event, filename, locals) {
 
 	function compile() {
 		try {
-			const template = compileFile(path, {})
+			const template = compileFile(path, {pretty})
 			pugCache.set(path, async (event, locals) => {
 				defaultContentType(event, "text/html; charset=utf-8")
-				const session = await useSession(event, {password: reg.as_token})
-				const rel = x => getRelativePath(event.path, x)
+				const session = await auth.useSession(event)
+				const managed = await auth.getManagedGuilds(event)
+				const rel = (to, paramsObject) => {
+					let result = getRelativePath(event.path, to)
+					if (paramsObject) {
+						const params = new URLSearchParams(paramsObject)
+						result += "?" + params.toString()
+					}
+					return result
+				}
 				return template(Object.assign({},
 					getQuery(event), // Query parameters can be easily accessed on the top level but don't allow them to overwrite anything
 					globals, // Globals
 					locals, // Explicit locals overwrite globals in case we need to DI something
-					{session, event, rel} // These are assigned last so they overwrite everything else. It would be catastrophically bad if they can't be trusted.
+					{session, event, rel, managed} // These are assigned last so they overwrite everything else. It would be catastrophically bad if they can't be trusted.
 				))
 			})
 		/* c8 ignore start */

src/web/pug/guild.pug

@@ -11,7 +11,9 @@ mixin badge-private
     | Private
 
 mixin discord(channel, radio=false)
-  - let permissions = dUtils.getPermissions([], guild.roles, null, channel.permission_overwrites)
+  //- Previously, we passed guild.roles as the second parameter, but this doesn't quite match Discord's behaviour. See issue #42 for why this was changed.
+  //- Basically we just want to assign badges based on the channel overwrites, without considering the guild's base permissions. /shrug
+  - let permissions = dUtils.getPermissions([], [{id: guild_id, name: "@everyone", permissions: 1<<10 | 1<<11}], null, channel.permission_overwrites)
   .s-user-card.s-user-card__small
     if !dUtils.hasPermission(permissions, DiscordTypes.PermissionFlagsBits.ViewChannel)
       != icons.Icons.IconLock
@@ -52,13 +54,13 @@ block body
   .s-page-title.mb24
     h1.s-page-title--header= guild.name
 
-  .d-flex.g16
+  .d-flex.g16(class="sm:fw-wrap")
     .fl-grow1
       h2.fs-headline1 Invite a Matrix user
 
-      form.d-grid.g-af-column.gy4.gx8.jc-start(method="post" action="/api/invite" style="grid-template-rows: repeat(2, auto)")
+      form.d-grid.g-af-column.gy4.gx8.jc-start(method="post" action=rel("/api/invite") hx-post=rel("/api/invite") hx-trigger="submit" hx-swap="none" hx-on::after-request="if (event.detail.successful) this.reset()" hx-disabled-elt="input, button" hx-indicator="#invite-button")
         label.s-label(for="mxid") Matrix ID
-        input.fl-grow1.s-input.wmx3#mxid(name="mxid" required placeholder="@user:example.org")
+        input.fl-grow1.s-input.wmx3#mxid(name="mxid" required placeholder="@user:example.org" pattern="@([^:]+):([a-z0-9:\\-]+\\.[a-z0-9.:\\-]+)")
         label.s-label(for="permissions") Permissions
         .s-select
           select#permissions(name="permissions")
@@ -67,89 +69,151 @@ block body
             option(value="admin") Admin
         input(type="hidden" name="guild_id" value=guild_id)
         .grid--row-start2
-          button.s-btn.s-btn__filled.htmx-indicator Invite
+          button.s-btn.s-btn__filled#invite-button Invite
     div
-      != svg
+      .s-card.d-flex.ai-center.jc-center(style="min-width: 132px; min-height: 132px;")
+        button.s-btn(class=space_id ? "s-btn__muted" : "s-btn__filled" hx-get=rel(`/qr?guild_id=${guild_id}`) hx-indicator="closest button" hx-swap="outerHTML" hx-disabled-elt="this") Show QR
 
-  h2.mt48.fs-headline1 Moderation
+  if space_id
+    h3.mt32.fs-category Privacy level
+      span#privacy-level-loading
+    .s-card
+      form(hx-post=rel("/api/privacy-level") hx-trigger="change" hx-indicator="#privacy-level-loading" hx-disabled-elt="input")
+        input(type="hidden" name="guild_id" value=guild_id)
 
-  h2.mt48.fs-headline1 Matrix setup
+        .s-toggle-switch.s-toggle-switch__multiple.s-toggle-switch__incremental.d-grid.gx16.ai-center(style="grid-template-columns: auto 1fr")
+          input(type="radio" name="privacy_level" value="directory" id="privacy-level-directory" checked=(privacy_level === 2))
+          label.d-flex.gx8.jc-center.grid--row-start3(for="privacy-level-directory")
+            != icons.Icons.IconPlusSm
+            != icons.Icons.IconInternationalSm
+            .fl-grow1 Directory
 
-  h3.mt32.fs-category Linked channels
+          input(type="radio" name="privacy_level" value="link" id="privacy-level-link" checked=(privacy_level === 1))
+          label.d-flex.gx8.jc-center.grid--row-start2(for="privacy-level-link")
+            != icons.Icons.IconPlusSm
+            != icons.Icons.IconLinkSm
+            .fl-grow1 Link
 
-  .s-card.bs-sm.p0
-    .s-table-container
-      table.s-table.s-table__bx-simple
-        each row in linkedChannelsWithDetails
-          tr
-            td.w40: +discord(row.channel)
-            td.p2: button.s-btn.s-btn__muted.s-btn__xs!= icons.Icons.IconLinkSm
-            td: +matrix(row)
-        else
-          tr
-            td(colspan="3")
-              .s-empty-state No channels linked between Discord and Matrix yet...
+          input(type="radio" name="privacy_level" value="invite" id="privacy-level-invite" checked=(privacy_level === 0))
+          label.d-flex.gx8.jc-center.grid--row-start1(for="privacy-level-invite")
+            svg.svg-icon(width="14" height="14" viewBox="0 0 14 14")
+            != icons.Icons.IconLockSm
+            .fl-grow1 Invite
 
-  h3.mt32.fs-category Auto-create
-  .s-card
-    form.d-flex.ai-center.g8
+          p.s-description.m0 In-app direct invite from another user
+          p.s-description.m0 Shareable invite links, like Discord
+          p.s-description.m0 Publicly listed in directory, like Discord server discovery
+
+    h2.mt48.fs-headline1 Features
+    .s-card.d-grid.px0.g16
+      form.d-flex.ai-center.g16
+        #url-preview-loading.p8
+        - let value = !!select("guild_space", "url_preview", {guild_id}).pluck().get()
+        input(type="hidden" name="guild_id" value=guild_id)
+        input.s-toggle-switch#url-preview(name="url_preview" type="checkbox" hx-post=rel("/api/url-preview") hx-indicator="#url-preview-loading" hx-disabled-elt="this" checked=value autocomplete="off")
+        label.s-label.fl-grow1(for="url-preview")
+          | Show Discord's URL previews on Matrix
+          p.s-description Shows info about links posted to chat. Discord's previews are generally better quality than Synapse's, especially for social media and videos.
+
+      form.d-flex.ai-center.g16
+        #presence-loading.p8
+        - value = !!select("guild_space", "presence", {guild_id}).pluck().get()
+        input(type="hidden" name="guild_id" value=guild_id)
+        input.s-toggle-switch#presence(name="presence" type="checkbox" hx-post=rel("/api/presence") hx-indicator="#presence-loading" hx-disabled-elt="this" checked=value autocomplete="off")
+        label.s-label(for="presence")
+          | Show online statuses on Matrix
+          p.s-description This might cause lag on really big Discord servers.
+
+  if space_id
+    h2.mt48.fs-headline1 Channel setup
+
+    h3.mt32.fs-category Linked channels
+    .s-card.bs-sm.p0
+      form.s-table-container(method="post" action=rel("/api/unlink") hx-confirm="Do you want to unlink these channels?\nIt may take a moment to clean up Matrix resources.")
+        input(type="hidden" name="guild_id" value=guild_id)
+        table.s-table.s-table__bx-simple
+          each row in linkedChannelsWithDetails
+            tr
+              td.w40: +discord(row.channel)
+              td.p2: button.s-btn.s-btn__muted.s-btn__xs(name="channel_id" value=row.channel.id hx-post=rel("/api/unlink") hx-trigger="click" hx-disabled-elt="this")!= icons.Icons.IconLinkSm
+              td: +matrix(row)
+          else
+            tr
+              td(colspan="3")
+                .s-empty-state No channels linked between Discord and Matrix yet...
+
+  h3.fs-category.mt32 Auto-create
+  .s-card.d-grid.px0
+    form.d-flex.ai-center.g16
+      #autocreate-loading.p8
+      - let value = !!select("guild_active", "autocreate", {guild_id}).pluck().get()
+      input(type="hidden" name="guild_id" value=guild_id)
+      input.s-toggle-switch#autocreate(name="autocreate" type="checkbox" hx-post=rel("/api/autocreate") hx-indicator="#autocreate-loading" hx-disabled-elt="this" checked=value autocomplete="off")
       label.s-label.fl-grow1(for="autocreate")
         | Create new Matrix rooms automatically
         p.s-description If you want, OOYE can automatically create new Matrix rooms and link them when an unlinked Discord channel is spoken in.
-      - let value = !!select("guild_active", "autocreate", {guild_id}).pluck().get()
+
+  if space_id
+    h3.mt32.fs-category Manually link channels
+    form.d-flex.g16.ai-start(hx-post=rel("/api/link") hx-trigger="submit" hx-disabled-elt="input, button" hx-indicator="#link-button")
+      .fl-grow2.s-btn-group.fd-column.w40
+        each channel in unlinkedChannels
+          input.s-btn--radio(type="radio" name="discord" required id=channel.id value=channel.id)
+          label.s-btn.s-btn__muted.ta-left.truncate(for=channel.id)
+            +discord(channel, true, "Announcement")
+        else
+          .s-empty-state.p8 All Discord channels are linked.
+      .fl-grow1.s-btn-group.fd-column.w30
+        each room in unlinkedRooms
+          input.s-btn--radio(type="radio" name="matrix" required id=room.room_id value=room.room_id)
+          label.s-btn.s-btn__muted.ta-left.truncate(for=room.room_id)
+            +matrix(room, true)
+        else
+          .s-empty-state.p8 All Matrix rooms are linked.
       input(type="hidden" name="guild_id" value=guild_id)
-      input.s-toggle-switch.order-last#autocreate(name="autocreate" type="checkbox" hx-post="/api/autocreate" hx-indicator="#autocreate-loading" hx-disabled-elt="this" checked=value)
-      .is-loading#autocreate-loading
+      div
+        button.s-btn.s-btn__icon.s-btn__filled#link-button
+          != icons.Icons.IconMerge
+          = ` Link`
 
-  h3.mt32.fs-category Privacy level
-  .s-card
-    form(hx-post="/api/privacy-level" hx-trigger="change" hx-indicator="#privacy-level-loading" hx-disabled-elt="this")
-      input(type="hidden" name="guild_id" value=guild_id)
-      .d-flex.ai-center.mb4
-        label.s-label.fl-grow1
-          | How people can join on Matrix
-          span.is-loading#privacy-level-loading
-      .s-toggle-switch.s-toggle-switch__multiple.s-toggle-switch__incremental.d-grid.gx16.ai-center(style="grid-template-columns: auto 1fr")
-        input(type="radio" name="level" value="directory" id="privacy-level-directory" checked=(privacy_level === 2))
-        label.d-flex.gx8.jc-center.grid--row-start3(for="privacy-level-directory")
-          != icons.Icons.IconPlusSm
-          != icons.Icons.IconInternationalSm
-          .fl-grow1 Directory
-
-        input(type="radio" name="level" value="link" id="privacy-level-link" checked=(privacy_level === 1))
-        label.d-flex.gx8.jc-center.grid--row-start2(for="privacy-level-link")
-          != icons.Icons.IconPlusSm
-          != icons.Icons.IconLinkSm
-          .fl-grow1 Link
-
-        input(type="radio" name="level" value="invite" id="privacy-level-invite" checked=(privacy_level === 0))
-        label.d-flex.gx8.jc-center.grid--row-start1(for="privacy-level-invite")
-          svg.svg-icon(width="14" height="14" viewBox="0 0 14 14")
-          != icons.Icons.IconLockSm
-          .fl-grow1 Invite
-
-        p.s-description.m0 In-app direct invite from another user
-        p.s-description.m0 Shareable invite links, like Discord
-        p.s-description.m0 Publicly listed in directory, like Discord server discovery
-
-  h3.mt32.fs-category Manually link channels
-  form.d-flex.g16.ai-start(hx-post="/api/link" hx-trigger="submit" hx-disabled-elt="this")
-    .fl-grow2.s-btn-group.fd-column.w40
-      each channel in unlinkedChannels
-        input.s-btn--radio(type="radio" name="discord" id=channel.id value=channel.id)
-        label.s-btn.s-btn__muted.ta-left.truncate(for=channel.id)
-          +discord(channel, true, "Announcement")
-      else
-        .s-empty-state.p8 All Discord channels are linked.
-    .fl-grow1.s-btn-group.fd-column.w30
-      each room in unlinkedRooms
-        input.s-btn--radio(type="radio" name="matrix" id=room.room_id value=room.room_id)
-        label.s-btn.s-btn__muted.ta-left.truncate(for=room.room_id)
-          +matrix(room, true)
-      else
-        .s-empty-state.p8 All Matrix rooms are linked.
-    input(type="hidden" name="guild_id" value=guild_id)
-    div
-      button.s-btn.s-btn__icon.s-btn__filled.htmx-indicator
-        != icons.Icons.IconMerge
-        = ` Link`
+    details.mt48
+      summary Debug room list
+      .d-grid.grid__2.gx24
+        div
+          h3.mt24 Channels
+          p Channels are read from the channel_room table and then merged with the discord.channels memory cache to make the merged list. Anything in memory cache that's not in channel_room is considered unlinked.
+        div
+          h3.mt24 Rooms
+          p Rooms use the same merged list as channels, based on augmented channel_room data. Then, rooms are read from the space. Anything in the space that's not merged is considered unlinked.
+        div
+          h3.mt24 Unavailable channels: Deleted from Discord
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedUncachedChannels
+                li: a(href=`https://discord.com/channels/${guild_id}/${row.id}`)= row.nick || row.name
+          h3.mt24 Unavailable channels: Wrong type
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedWrongTypeChannels
+                li: a(href=`https://discord.com/channels/${guild_id}/${row.id}`) (#{row.type}) #{row.name}
+          h3.mt24 Unavailable channels: Bridge can't access
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedPrivateChannels
+                li: a(href=`https://discord.com/channels/${guild_id}/${row.id}`)= row.name
+        div- // Rooms
+          h3.mt24 Unavailable rooms: Already linked
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedLinkedRooms
+                li: a(href=`https://matrix.to/#/${row.room_id}`)= row.name
+          h3.mt24 Unavailable rooms: Wrong type
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedWrongTypeRooms
+                li: a(href=`https://matrix.to/#/${row.room_id}`) (#{row.room_type}) #{row.name}
+          h3.mt24 Unavailable rooms: Archived thread
+          .s-card.p0
+            ul.my8.ml24
+              each row in removedArchivedThreadRooms
+                li: a(href=`https://matrix.to/#/${row.room_id}`)= row.name

src/web/pug/guild_access_denied.pug

@@ -1,13 +1,17 @@
 extends includes/template.pug
 
 block body
-  if !session.data.managedGuilds
+  if !session.data.userID
     .s-empty-state.wmx4.p48
       != icons.Spots.SpotEmptyXL
       p You need to log in to manage your servers.
-      a.s-btn.s-btn__icon.s-btn__filled(href=rel("/oauth"))
-        != icons.Icons.IconDiscord
-        = ` Log in with Discord`
+      .d-flex.jc-center.g8
+        a.s-btn.s-btn__icon.s-btn__featured.s-btn__filled(href=rel("/oauth"))
+          != icons.Icons.IconDiscord
+          = ` Log in with Discord`
+        a.s-btn.s-btn__icon.s-btn__matrix.s-btn__filled(href=rel("/log-in-with-matrix"))
+          != icons.Icons.IconSpeechBubble
+          = ` Log in with Matrix`
 
   else if !guild_id
     .s-empty-state.wmx4.p48
@@ -15,8 +19,18 @@ block body
       p Select a server from the top right corner to continue.
       p If the server you're looking for isn't there, try #[a(href=rel("/oauth?action=add")) logging in again.]
 
-  else if !discord.guilds.has(guild_id) || !session.data.managedGuilds.includes(guild_id)
+  else if !discord.guilds.has(guild_id) || !managed.has(guild_id)
     .s-empty-state.wmx4.p48
       != icons.Spots.SpotAlertXL
       p Either the selected server doesn't exist, or you don't have the Manage Server permission on Discord.
       p If you've checked your permissions, try #[a(href=rel("/oauth")) logging in again.]
+
+  else if !row
+    .s-empty-state.wmx4.p48
+      != icons.Spots.SpotAlertXL
+      p Please add the bot to your server using the buttons on the home page.
+
+  else
+    .s-empty-state.wmx4.p48
+      != icons.Spots.SpotAlertXL
+      p Access denied.

src/web/pug/guild_not_linked.pug

@@ -0,0 +1,53 @@
+extends includes/template.pug
+
+mixin space(space)
+  .s-user-card.flex__1
+    span.s-avatar.s-avatar__32.s-user-card--avatar
+      if space.avatar
+        img.s-avatar--image(src=mUtils.getPublicUrlForMxc(space.avatar))
+      else
+        .s-avatar--letter.bg-silver-400.bar-md(aria-hidden="true")= space.name[0]
+    .s-user-card--info.ai-start
+      strong= space.name
+      if space.topic
+        ul.s-user-card--awards
+          li= space.topic
+
+block body
+  .s-notice.s-notice__info.d-flex.g16
+    div
+      != icons.Icons.IconInfo
+    div
+      - const self = `@${reg.sender_localpart}:${reg.ooye.server_name}`
+      strong You picked self-service mode
+      .mt4 To complete setup, you need to manually choose a Matrix space to link with #[strong= guild.name].
+      .mt4 On Matrix, invite #[code.s-code-block: a.fc-black.s-link(href=`https://matrix.to/#/${self}` target="_blank")= self] to a space. Then you can pick the space on this page.
+
+  h3.mt32.fs-category Choose a space
+
+  form.s-card.bs-sm.p0.s-table-container.bar-md(method="post" action=rel("/api/link-space"))
+    input(type="hidden" name="guild_id" value=guild_id)
+    table.s-table.s-table__bx-simple
+      each space in spaces
+        tr
+          td.p0: +space(space)
+          td: button.s-btn(name="space_id" value=space.room_id hx-post=rel("/api/link-space") hx-trigger="click" hx-disabled-elt="this") Link with this space
+      else
+        if session.data.mxid
+          tr
+            td.p16 Invite the bridge to a space, and the space will show up here.
+        else
+          tr
+            td.d-flex.ai-center.pl16.g16
+              | You need to log in with Matrix first.
+              a.s-btn.s-btn__matrix.s-btn__outlined(href=rel(`/log-in-with-matrix`, {next: `./guild?guild_id=${guild_id}`})) Log in with Matrix
+
+  h3.mt48.fs-category Auto-create
+  .s-card
+    form.d-flex.ai-center.g8(method="post" action=rel("/api/autocreate") hx-post=rel("/api/autocreate") hx-indicator="#easy-mode-button")
+      input(type="hidden" name="guild_id" value=guild_id)
+      input(type="hidden" name="autocreate" value="true")
+      label.s-label.fl-grow1
+        | Changed your mind?
+        p.s-description If you want, OOYE can create and manage the Matrix space so you don't have to.
+      button.s-btn.s-btn__outlined#easy-mode-button Use easy mode

src/web/pug/home.pug

@@ -1,24 +1,56 @@
 extends includes/template.pug
 
 block body
-  .s-page-title.mb24
-    h1.s-page-title--header Bridge a Discord server
+  - let locked = reg.ooye.web_password && reg.ooye.web_password !== session.data.password
 
-  .d-grid.g24.grid__2(class="sm:grid__1")
-    .s-card.bs-md.d-flex.fd-column
-      h2 Easy mode
-      p Add the bot to your Discord server.
-      p It will automatically create new Matrix rooms for you.
-      .fl-grow1
-      a.s-btn.s-btn__filled.s-btn__icon(href=rel("/oauth?action=add"))
-        != icons.Icons.IconPlus
-        = ` Add to server`
-    .s-card.bs-md.d-flex.fd-column
-      h2 Self-service
-      p OOYE will link an existing Discord server and Matrix space together.
-      p Choose this option if you already have a community set up on Matrix.
-      p Or, choose this if you're migrating from a different bridge.
-      .fl-grow1
-      a.s-btn.s-btn__outlined.s-btn__icon(href=rel("/oauth?action=add-self-service"))
-        != icons.Icons.IconUnorderedList
-        = ` Set up self-service`
+  if locked
+    aside.s-notice.s-notice__warning.p8
+      .d-flex.flex__center.jc-space-between.s-banner--container.g8(class="md:fw-wrap")
+        .d-flex.ai-center.g8
+          .flex--item!= icons.Icons.IconLock
+          p.m0 <strong>Private instance.</strong> You need the password to use this instance of Out Of Your Element.
+        form(method="post" action=rel("/api/password"))
+          input.s-input(placeholder="Enter password" name="password")
+
+    .h32
+
+    .s-page-title.mb24
+      h1.s-page-title--header Out Of Your Element
+
+  else
+    .s-page-title.mb24
+      h1.s-page-title--header Bridge a Discord server
+
+    .d-grid.g24.grid__2.mb24(class="sm:grid__1")
+      .s-card.bs-md.d-flex.fd-column
+        h2 Easy mode
+        p Add the bot to your Discord server.
+        p It will automatically create new Matrix rooms for you.
+        .fl-grow1
+        a.s-btn.s-btn__filled.s-btn__icon(href=rel("/oauth?action=add"))
+          != icons.Icons.IconPlus
+          = ` Add to server`
+      .s-card.bs-md.d-flex.fd-column
+        h2 Self-service
+        p OOYE will link an existing Discord server and Matrix space together.
+        p Choose this option if you already have a community set up on Matrix.
+        p Or, choose this if you're migrating from a different bridge.
+        .fl-grow1
+        a.s-btn.s-btn__outlined.s-btn__icon(href=rel("/oauth?action=add-self-service"))
+          != icons.Icons.IconUnorderedList
+          = ` Set up self-service`
+
+  .s-prose
+    h2 What is this?
+    p #[a(href="https://gitdab.com/cadence/out-of-your-element") Out Of Your Element] is a bridge between the Discord and Matrix chat apps. It lets people on both platforms chat with each other without needing to get everyone on the same app.
+    p Just chat like usual, and the bridge will forward messages back and forth between the two platforms, so everyone sees the whole conversation.
+    p All kinds of content are supported, including pictures, threads, emojis, and @mentions.
+    p It's really easy to set up, even if you only have Discord. Just add the bot to your server, and it'll make everything available on Matrix automatically.
+
+    if locked
+      h2 This is a private instance
+      p Anybody can run their own instance of the Out Of Your Element software. The person running this instance has made it private, so you can't add it to your server just yet. If you know who's in charge of #{reg.ooye.server_name}, ask them for the password.
+
+      h2 Run your own instance
+      p You can still use Out Of Your Element by running your own copy of the software, but this requires some technical skill.
+      p To get started, #[a(href="https://gitdab.com/cadence/out-of-your-element/src/branch/main/docs/get-started.md") check the installation instructions.]

src/web/pug/includes/hash.svg

@@ -1,46 +1 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   aria-hidden="true"
-   class="svg-icon iconItalic"
-   width="18"
-   height="18"
-   viewBox="0 0 18 18"
-   version="1.1"
-   id="svg1"
-   sodipodi:docname="hash.svg"
-   inkscape:version="1.3.2 (091e20ef0f, 2023-11-25)"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg">
-  <defs
-     id="defs1" />
-  <path
-     style="opacity:1;stroke-width:24.2222;stroke-linecap:square;paint-order:stroke fill markers"
-     d="m 13.949463,2.0417087 c 0,0 0.664304,0.00704 0.854464,0.00134 0.19016,-0.0057 0.924873,0.2384962 0.57664,0.9863413 -0.288846,0.6203095 -5.045042,11.035358 -5.4783833,11.984378 -0.4333415,0.949021 -0.7881247,0.945761 -1.3553087,0.945761 -0.567184,0 -0.3175392,0 -0.734375,0 -0.4168358,0 -0.7985231,-0.467356 -0.5770328,-0.951217 C 7.4569576,14.524452 12.479729,3.5512928 12.725807,3.0070042 13.022379,2.3510304 13.336114,2.0361844 13.949463,2.0417087 Z"
-     id="path4"
-     sodipodi:nodetypes="czszzzzsc" />
-  <rect
-     style="opacity:1;stroke-width:27.7591;stroke-linecap:square;paint-order:stroke fill markers"
-     id="rect4"
-     width="11.987322"
-     height="2"
-     x="2.002677"
-     y="11.007812"
-     rx="1"
-     ry="1" />
-  <rect
-     style="opacity:1;stroke-width:27.7591;stroke-linecap:square;paint-order:stroke fill markers"
-     id="rect5"
-     width="11.987322"
-     height="2"
-     x="4.0100012"
-     y="5.007813"
-     rx="1"
-     ry="1" />
-  <path
-     style="opacity:1;stroke-width:24.2222;stroke-linecap:square;paint-order:stroke fill markers"
-     d="m 9.1764922,2.0417087 c 0,0 0.664304,0.00704 0.8544638,0.00134 0.19016,-0.0057 0.924873,0.2384962 0.57664,0.9863413 -0.288846,0.6203095 -5.0450418,11.035358 -5.4783831,11.984378 -0.4333415,0.949021 -0.7881247,0.945761 -1.3553087,0.945761 -0.567184,0 -0.3175392,0 -0.734375,0 -0.4168358,0 -0.7985231,-0.467356 -0.5770328,-0.951217 C 2.6839868,14.524452 7.7067582,3.5512928 7.9528362,3.0070042 8.2494082,2.3510304 8.5631432,2.0361844 9.1764922,2.0417087 Z"
-     id="path1"
-     sodipodi:nodetypes="czszzzzsc" />
-</svg>
+<svg fill="none" viewBox="0 0 16 16" height="16" width="16"><path stroke="currentcolor" stroke-width="2" d="m6.75 1-2.5 14m7.5-14-2.5 14M14 10.25H1m14-4.5H2"></path></svg>

src/web/pug/includes/template.pug

@@ -1,71 +1,148 @@
-mixin guild(guild)
-  span.s-avatar.s-avatar__32.s-user-card--avatar
-    if guild.icon
-      img.s-avatar--image(src=`https://cdn.discordapp.com/icons/${guild.id}/${guild.icon}.png?size=32`)
-    else
-      .s-avatar--letter.bg-silver-400.bar-md(aria-hidden="true")= guild.name[0]
-  .s-user-card--info.ai-start
-    strong= guild.name
-    ul.s-user-card--awards
-      li #{discord.guildChannelMap.get(guild.id).filter(c => [0, 5, 15, 16].includes(discord.channels.get(c).type)).length} channels
-
-doctype html
-html(lang="en")
-  head
-    title Out Of Your Element
-    <meta name="viewport" content="width=device-width, initial-scale=1">
-    link(rel="stylesheet" type="text/css" href=rel("/static/stacks.min.css"))
-    <link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 80%22><text y=%22.83em%22 font-size=%2283%22>💬</text></svg>">
-    meta(name="htmx-config" content='{"indicatorClass":"is-loading"}')
-    style.
-      .themed {
-        --theme-base-primary-color-h: 266;
-        --theme-base-primary-color-s: 53%;
-        --theme-base-primary-color-l: 63%;
-        --theme-dark-primary-color-h: 266;
-        --theme-dark-primary-color-s: 53%;
-        --theme-dark-primary-color-l: 63%;
-      }
-      .s-toggle-switch.s-toggle-switch__multiple.s-toggle-switch__incremental input[type="radio"]:checked ~ label:not(.s-toggle-switch--label-off) {
-        --_ts-multiple-bg: var(--green-400);
-        --_ts-multiple-fc: var(--white);
-      }
-  body.themed.theme-system
-    header.s-topbar
-      .s-topbar--skip-link(href="#content") Skip to main content
-      .s-topbar--container.wmx9
-        a.s-topbar--logo(href=rel("/"))
-          img.s-avatar.s-avatar__32(src=rel("/icon.png"))
-        nav.s-topbar--navigation
-          ul.s-topbar--content
-            li.ps-relative
-              if !session.data.managedGuilds || session.data.managedGuilds.length === 0
-                a.s-btn.s-btn__icon.as-center(href=rel("/oauth"))
-                  != icons.Icons.IconDiscord
-                  = ` Log in`
-              else if guild_id && session.data.managedGuilds.includes(guild_id) && discord.guilds.has(guild_id)
-                button.s-topbar--item.s-btn.s-btn__muted.s-user-card(popovertarget="guilds")
-                  +guild(discord.guilds.get(guild_id))
-              else if session.data.managedGuilds
-                button.s-topbar--item.s-btn.s-btn__muted.s-btn__dropdown.pr24.s-user-card.s-label(popovertarget="guilds")
-                  | Your servers
-              #guilds(popover data-popper-placement="bottom" style="display: revert; width: revert;").s-popover.overflow-visible
-                .s-popover--arrow.s-popover--arrow__tc
-                .s-popover--content.overflow-y-auto.overflow-x-hidden
-                  ul.s-menu(role="menu")
-                    each guild in (session.data.managedGuilds || []).map(id => discord.guilds.get(id)).filter(g => g)
-                      li(role="menuitem")
-                        a.s-topbar--item.s-user-card.d-flex.p4(href=rel(`/guild?guild_id=${guild.id}`))
-                          +guild(guild)
-    .mx-auto.w100.wmx9.py24.px8.fs-body1#content
-      block body
-    script.
-      document.querySelectorAll("[popovertarget]").forEach(e => {
-        e.addEventListener("click", () => {
-          const rect = e.getBoundingClientRect()
-          const t = `:popover-open { position: absolute; top: ${Math.floor(rect.bottom)}px; left: ${Math.floor(rect.left + rect.width / 2)}px; width: ${Math.floor(rect.width)}px; transform: translateX(-50%); box-sizing: content-box; margin: 0 }`
-          // console.log(t)
-          document.styleSheets[0].insertRule(t)
-        })
-      })
-    script(src=rel("/static/htmx.min.js"))
+mixin guild(guild)
+  span.s-avatar.s-avatar__32.s-user-card--avatar
+    if guild.icon
+      img.s-avatar--image(src=`https://cdn.discordapp.com/icons/${guild.id}/${guild.icon}.png?size=32`)
+    else
+      .s-avatar--letter.bg-silver-400.bar-md(aria-hidden="true")= guild.name[0]
+  .s-user-card--info.ai-start
+    strong= guild.name
+    ul.s-user-card--awards
+      li #{discord.guildChannelMap.get(guild.id).filter(c => [0, 5, 15, 16].includes(discord.channels.get(c).type)).length} channels
+
+mixin define-theme(name, h, s, l)
+  style.
+    :root {
+      --#{name}-h: #{h};
+      --#{name}-s: #{s};
+      --#{name}-l: #{l};
+      --#{name}: var(--#{name}-400);
+      --#{name}-100: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(70%, calc(var(--#{name}-l) + 50 * 1%), 95%));
+      --#{name}-200: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(55%, calc(var(--#{name}-l) + 35 * 1%), 90%));
+      --#{name}-300: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(35%, calc(var(--#{name}-l) + 15 * 1%), 75%));
+      --#{name}-400: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(20%, calc(var(--#{name}-l) + 0 * 1%), 60%));
+      --#{name}-500: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(15%, calc(var(--#{name}-l) + -14 * 1%), 45%));
+      --#{name}-600: hsl(var(--#{name}-h), calc(var(--#{name}-s) + 0 * 1%), clamp(5%, calc(var(--#{name}-l) + -26 * 1%), 30%));
+    }
+
+mixin define-themed-button(name, theme)
+  style.
+    .s-btn.s-btn__#{name} {
+      --_bu-bg-active: var(--#{theme}-300);
+      --_bu-bg-hover: var(--#{theme}-200);
+      --_bu-bg-selected: var(--#{theme}-300);
+      --_bu-fc: var(--#{theme}-500);
+      --_bu-fc-active: var(--_bu-fc);
+      --_bu-fc-hover: var(--#{theme}-500);
+      --_bu-fc-selected: var(--#{theme}-600);
+      --_bu-filled-bc: transparent;
+      --_bu-filled-bc-selected: var(--_bu-filled-bc);
+      --_bu-filled-bg: var(--#{theme}-400);
+      --_bu-filled-bg-active: var(--#{theme}-500);
+      --_bu-filled-bg-hover: var(--#{theme}-500);
+      --_bu-filled-bg-selected: var(--#{theme}-600);
+      --_bu-filled-fc: var(--white);
+      --_bu-filled-fc-active: var(--_bu-filled-fc);
+      --_bu-filled-fc-hover: var(--_bu-filled-fc);
+      --_bu-filled-fc-selected: var(--_bu-filled-fc);
+      --_bu-outlined-bc: var(--#{theme}-400);
+      --_bu-outlined-bc-selected: var(--#{theme}-500);
+      --_bu-outlined-bg-selected: var(--_bu-bg-selected);
+      --_bu-outlined-fc-selected: var(--_bu-fc-selected);
+      --_bu-number-fc: var(--white);
+      --_bu-number-fc-filled: var(--#{theme});
+    }
+
+doctype html
+html(lang="en")
+  head
+    title Out Of Your Element
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    link(rel="stylesheet" type="text/css" href=rel("/static/stacks.min.css"))
+    <link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 80%22><text y=%22.83em%22 font-size=%2283%22>💬</text></svg>">
+    meta(name="htmx-config" content='{"requestClass":"is-loading"}')
+    style.
+      .s-prose a {
+        text-decoration: underline;
+      }
+      .themed {
+        --theme-base-primary-color-h: 266;
+        --theme-base-primary-color-s: 53%;
+        --theme-base-primary-color-l: 63%;
+        --theme-dark-primary-color-h: 266;
+        --theme-dark-primary-color-s: 53%;
+        --theme-dark-primary-color-l: 63%;
+      }
+      .s-toggle-switch.s-toggle-switch__multiple.s-toggle-switch__incremental input[type="radio"]:checked ~ label:not(.s-toggle-switch--label-off) {
+        --_ts-multiple-bg: var(--green-400);
+        --_ts-multiple-fc: var(--white);
+      }
+      .s-btn__dropdown:has(+ :popover-open) {
+        background-color: var(--theme-topbar-item-background-hover, var(--black-200)) !important;
+      }
+    +define-themed-button("matrix", "black")
+  body.themed.theme-system
+    header.s-topbar
+      .s-topbar--skip-link(href="#content") Skip to main content
+      .s-topbar--container.wmx9
+        a.s-topbar--logo(href=rel("/"))
+          img.s-avatar.s-avatar__32(src=rel("/icon.png"))
+        nav.s-topbar--navigation
+          ul.s-topbar--content
+            li.ps-relative.g8
+              if !session.data.mxid
+                a.s-btn.s-btn__icon.s-btn__matrix.s-btn__outlined.as-center(href=rel("/log-in-with-matrix"))
+                  != icons.Icons.IconSpeechBubble
+                  = ` Log in`
+                  span(class="sm:d-none")= ` with Matrix`
+              if !session.data.userID
+                a.s-btn.s-btn__icon.s-btn__featured.s-btn__outlined.as-center(href=rel("/oauth"))
+                  != icons.Icons.IconDiscord
+                  = ` Log in`
+                  span(class="sm:d-none")= ` with Discord`
+              if guild_id && managed.has(guild_id) && discord.guilds.has(guild_id)
+                button.s-topbar--item.s-btn.s-btn__muted.s-btn__dropdown.pr32.bar0.s-user-card(popovertarget="guilds")
+                  +guild(discord.guilds.get(guild_id))
+              else if managed.size
+                button.s-topbar--item.s-btn.s-btn__muted.s-btn__dropdown.pr24.s-user-card.bar0.fc-black(popovertarget="guilds")
+                  | Your servers
+              else if session.data.mxid || session.data.userID
+                .d-flex.ai-center
+                  .s-badge.s-badge__bot.py6.px16.bar-md
+                    | No servers available
+              #guilds(popover data-popper-placement="bottom" style="display: revert; width: revert;").s-popover.overflow-visible
+                .s-popover--arrow.s-popover--arrow__tc
+                .s-popover--content.overflow-y-auto.overflow-x-hidden
+                  ul.s-menu(role="menu")
+                    each guild in [...managed].map(id => discord.guilds.get(id)).filter(g => g).sort((a, b) => a.name.toLowerCase() < b.name.toLowerCase() ? -1 : 1)
+                      li(role="menuitem")
+                        a.s-topbar--item.s-user-card.d-flex.p4(href=rel(`/guild?guild_id=${guild.id}`))
+                          +guild(guild)
+    //- Body
+    .mx-auto.w100.wmx9.py24.px8.fs-body1#content
+      block body
+    //- Guild list popover
+    script.
+      document.querySelectorAll("[popovertarget]").forEach(e => {
+        e.addEventListener("click", () => {
+          const rect = e.getBoundingClientRect()
+          const t = `:popover-open { position: absolute; top: ${Math.floor(rect.bottom)}px; left: ${Math.floor(rect.left + rect.width / 2)}px; width: ${Math.floor(rect.width)}px; transform: translateX(-50%); box-sizing: content-box; margin: 0 }`
+          document.styleSheets[0].insertRule(t, document.styleSheets[0].cssRules.length)
+        })
+      })
+    script(src=rel("/static/htmx.js"))
+    //- Error dialog
+    aside.s-modal#server-error(aria-hidden="true")
+      .s-modal--dialog
+        h1.s-modal--header Server error
+        pre.overflow-auto#server-error-content
+        button.s-modal--close.s-btn.s-btn__muted(aria-label="Close" type="button" onclick="hideError()")!= icons.Icons.IconClearSm
+        .s-modal--footer
+          button.s-btn.s-btn__outlined.s-btn__muted(type="button" onclick="hideError()") OK
+    script.
+      function hideError() {
+        document.getElementById("server-error").setAttribute("aria-hidden", "true")
+      }
+      document.body.addEventListener("htmx:responseError", event => {
+        document.getElementById("server-error").setAttribute("aria-hidden", "false")
+        document.getElementById("server-error-content").textContent = event.detail.xhr.responseText
+      })

src/web/pug/invite.pug

@@ -13,11 +13,11 @@ block body
     .s-page-title.mb24
       h1.s-page-title--header= guild.name
 
-    .d-flex.g16
+    .d-flex.g16#form-container
       .fl-grow1
         h2.fs-headline1 Invite a Matrix user
 
-        form.d-flex.gy16.fd-column(method="post" action="/api/invite" style="grid-template-rows: repeat(2, auto)")
+        form.d-flex.gy16.fd-column(method="post" action=rel("/api/invite") hx-post=rel("/api/invite") hx-indicator="#invite-button" hx-select="#ok" hx-target="#form-container")
           .d-flex.gy4.fd-column
             label.s-label(for="mxid") Matrix ID
             input.fl-grow1.s-input.wmx3#mxid(name="mxid" required placeholder="@user:example.org")
@@ -30,4 +30,4 @@ block body
                 option(value="admin") Admin
           input(type="hidden" name="nonce" value=nonce)
           div
-            button.s-btn.s-btn__filled.htmx-indicator Invite
+            button.s-btn.s-btn__filled#invite-button Invite

src/web/pug/log-in-with-matrix.pug

@@ -0,0 +1,16 @@
+extends includes/template.pug
+
+block body
+  .s-page-title.mb24
+    h1.s-page-title--header Log in with Matrix
+
+  .d-flex.g16#form-container
+    .fl-grow1
+      form.d-flex.gy16.fd-column(method="post" action=rel("/api/log-in-with-matrix") hx-post=rel("/api/log-in-with-matrix") hx-indicator="#log-in-button" hx-select="#ok" hx-target="#form-container")
+        if next
+          input(type="hidden" name="next" value=next)
+        .d-flex.gy4.fd-column
+          label.s-label(for="mxid") Your Matrix ID
+          input.fl-grow1.s-input.wmx3#mxid(name="mxid" required placeholder="@user:example.org" pattern="@([^:]+):([a-z0-9:\\-]+\\.[a-z0-9.:\\-]+)")
+        div
+          button.s-btn.s-btn__github#log-in-button Continue with Matrix

src/web/pug/ok.pug

@@ -1,6 +1,6 @@
 extends includes/template.pug
 
 block body
-  .ta-center.wmx5.p48.mx-auto
-    != icons.Spots.SpotApproveXL
+  .ta-center.wmx5.p48.mx-auto#ok
+    != spot ? icons.Spots[spot] : icons.Spots.SpotApproveXL
     p.mt24.fs-body2= msg

src/web/routes/download-matrix.test.js

@@ -3,7 +3,6 @@
 const tryToCatch = require("try-to-catch")
 const {test} = require("supertape")
 const {router} = require("../../../test/web")
-const fetch = require("node-fetch")
 
 test("web download matrix: access denied if not a known attachment", async t => {
 	const [error] = await tryToCatch(() =>
@@ -27,7 +26,7 @@ test("web download matrix: works if a known attachment", async t => {
 		event,
 		api: {
 			async getMedia(mxc, init) {
-				return new fetch.Response("", {status: 200, headers: {"content-type": "image/png"}})
+				return new Response("", {status: 200, headers: {"content-type": "image/png"}})
 			}
 		}
 	})

src/web/routes/guild-settings.js

@@ -2,44 +2,93 @@
 
 const assert = require("assert/strict")
 const {z} = require("zod")
-const {defineEventHandler, sendRedirect, useSession, createError, readValidatedBody} = require("h3")
+const {defineEventHandler, createError, readValidatedBody, getRequestHeader, setResponseHeader, sendRedirect, H3Event} = require("h3")
 
-const {as, db, sync} = require("../../passthrough")
-const {reg} = require("../../matrix/read-registration")
+const {as, db, sync, select} = require("../../passthrough")
 
-/** @type {import("../../d2m/actions/create-space")} */
-const createSpace = sync.require("../../d2m/actions/create-space")
+/** @type {import("../auth")} */
+const auth = sync.require("../auth")
+/** @type {import("../../d2m/actions/set-presence")} */
+const setPresence = sync.require("../../d2m/actions/set-presence")
 
-/** @type {["invite", "link", "directory"]} */
-const levels = ["invite", "link", "directory"]
-const schema = {
-	autocreate: z.object({
-		guild_id: z.string(),
-		autocreate: z.string().optional()
-	}),
-	privacyLevel: z.object({
-		guild_id: z.string(),
-		level: z.enum(levels)
+/**
+ * @param {H3Event} event
+ * @returns {import("../../d2m/actions/create-space")}
+ */
+function getCreateSpace(event) {
+	/* c8 ignore next */
+	return event.context.createSpace || sync.require("../../d2m/actions/create-space")
+}
+
+/**
+ * @typedef Options
+ * @prop {(value: string?) => number} transform
+ * @prop {(event: H3Event, guildID: string) => any} [after]
+ * @prop {keyof import("../../db/orm-defs").Models} table
+ */
+
+/**
+ * @template {string} T
+ * @param {T} key
+ * @param {Partial<Options>} [inputOptions]
+ */
+function defineToggle(key, inputOptions) {
+	/** @type {Options} */
+	const options = {
+		transform: x => +!!x, // convert toggle to 0 or 1
+		table: "guild_space"
+	}
+	Object.assign(options, inputOptions)
+	return defineEventHandler(async event => {
+		const bodySchema = z.object({
+			guild_id: z.string(),
+			[key]: z.string().optional()
+		})
+		/** @type {Record<T, string?> & Record<"guild_id", string> & Record<string, unknown>} */ // @ts-ignore
+		const parsedBody = await readValidatedBody(event, bodySchema.parse)
+		const managed = await auth.getManagedGuilds(event)
+		if (!managed.has(parsedBody.guild_id)) throw createError({status: 403, message: "Forbidden", data: "Can't change settings for a guild you don't have Manage Server permissions in"})
+
+		const value = options.transform(parsedBody[key])
+		assert(typeof value === "number")
+		db.prepare(`UPDATE ${options.table} SET ${key} = ? WHERE guild_id = ?`).run(value, parsedBody.guild_id)
+
+		return (options.after && await options.after(event, parsedBody.guild_id)) || null
 	})
 }
 
-as.router.post("/api/autocreate", defineEventHandler(async event => {
-	const parsedBody = await readValidatedBody(event, schema.autocreate.parse)
-	const session = await useSession(event, {password: reg.as_token})
-	if (!(session.data.managedGuilds || []).includes(parsedBody.guild_id)) throw createError({status: 403, message: "Forbidden", data: "Can't change settings for a guild you don't have Manage Server permissions in"})
-
-	db.prepare("UPDATE guild_active SET autocreate = ? WHERE guild_id = ?").run(+!!parsedBody.autocreate, parsedBody.guild_id)
-	return null // 204
+as.router.post("/api/autocreate", defineToggle("autocreate", {
+	table: "guild_active",
+	after(event, guild_id) {
+		// If showing a partial page due to incomplete setup, need to refresh the whole page to show the alternate version
+		const spaceID = select("guild_space", "space_id", {guild_id}).pluck().get()
+		if (!spaceID) {
+			if (getRequestHeader(event, "HX-Request")) {
+				setResponseHeader(event, "HX-Refresh", "true")
+			} else {
+				return sendRedirect(event, "", 302)
+			}
+		}
+	}
 }))
 
-as.router.post("/api/privacy-level", defineEventHandler(async event => {
-	const parsedBody = await readValidatedBody(event, schema.privacyLevel.parse)
-	const session = await useSession(event, {password: reg.as_token})
-	if (!(session.data.managedGuilds || []).includes(parsedBody.guild_id)) throw createError({status: 403, message: "Forbidden", data: "Can't change settings for a guild you don't have Manage Server permissions in"})
+as.router.post("/api/url-preview", defineToggle("url_preview"))
 
-	const i = levels.indexOf(parsedBody.level)
-	assert.notEqual(i, -1)
-	db.prepare("UPDATE guild_space SET privacy_level = ? WHERE guild_id = ?").run(i, parsedBody.guild_id)
-	await createSpace.syncSpaceFully(parsedBody.guild_id) // this is inefficient but OK to call infrequently on user request
-	return null // 204
+as.router.post("/api/presence", defineToggle("presence", {
+	after() {
+		setPresence.guildPresenceSetting.update()
+	}
+}))
+
+as.router.post("/api/privacy-level", defineToggle("privacy_level", {
+	transform(value) {
+		assert(value)
+		const i = ["invite", "link", "directory"].indexOf(value)
+		assert.notEqual(i, -1)
+		return i
+	},
+	async after(event, guildID) {
+		const createSpace = getCreateSpace(event)
+		await createSpace.syncSpaceFully(guildID) // this is inefficient but OK to call infrequently on user request
+	}
 }))

src/web/routes/guild-settings.test.js

@@ -0,0 +1,96 @@
+// @ts-check
+
+const tryToCatch = require("try-to-catch")
+const {router, test} = require("../../../test/web")
+const {select} = require("../../passthrough")
+const {MatrixServerError} = require("../../matrix/mreq")
+
+test("web autocreate: checks permissions", async t => {
+	const [error] = await tryToCatch(() => router.test("post", "/api/autocreate", {
+		body: {
+			guild_id: "66192955777486848"
+		}
+	}))
+	t.equal(error.data, "Can't change settings for a guild you don't have Manage Server permissions in")
+})
+
+
+test("web autocreate: turns off autocreate and does htmx page refresh when guild not linked", async t => {
+	const event = {}
+	await router.test("post", "/api/autocreate", {
+		sessionData: {
+			managedGuilds: ["66192955777486848"]
+		},
+		body: {
+			guild_id: "66192955777486848",
+			// autocreate is false
+		},
+		headers: {
+			"hx-request": "true"
+		},
+		event
+	})
+	t.equal(event.node.res.getHeader("hx-refresh"), "true")
+	t.equal(select("guild_active", "autocreate", {guild_id: "66192955777486848"}).pluck().get(), 0)
+})
+
+test("web autocreate: turns on autocreate and issues 302 when not using htmx", async t => {
+	const event = {}
+	await router.test("post", "/api/autocreate", {
+		sessionData: {
+			managedGuilds: ["66192955777486848"]
+		},
+		body: {
+			guild_id: "66192955777486848",
+			autocreate: "yes"
+		},
+		event
+	})
+	t.equal(event.node.res.getHeader("location"), "")
+	t.equal(select("guild_active", "autocreate", {guild_id: "66192955777486848"}).pluck().get(), 1)
+})
+
+test("web privacy level: checks permissions", async t => {
+	const [error] = await tryToCatch(() => router.test("post", "/api/privacy-level", {
+		body: {
+			guild_id: "112760669178241024",
+			privacy_level: "directory"
+		}
+	}))
+	t.equal(error.data, "Can't change settings for a guild you don't have Manage Server permissions in")
+})
+
+test("web privacy level: updates privacy level", async t => {
+	let called = 0
+	await router.test("post", "/api/privacy-level", {
+		sessionData: {
+			managedGuilds: ["112760669178241024"]
+		},
+		body: {
+			guild_id: "112760669178241024",
+			privacy_level: "directory"
+		},
+		createSpace: {
+			async syncSpaceFully(guildID) {
+				called++
+				t.equal(guildID, "112760669178241024")
+				return ""
+			}
+		}
+	})
+	t.equal(called, 1)
+	t.equal(select("guild_space", "privacy_level", {guild_id: "112760669178241024"}).pluck().get(), 2) // directory = 2
+})
+
+test("web presence: updates presence", async t => {
+	await router.test("post", "/api/presence", {
+		sessionData: {
+			managedGuilds: ["112760669178241024"]
+		},
+		body: {
+			guild_id: "112760669178241024"
+			// presence is on by default - turn it off
+		}
+	})
+	t.equal(select("guild_space", "presence", {guild_id: "112760669178241024"}).pluck().get(), 0)
+})

src/web/routes/guild.js

@@ -1,24 +1,33 @@
 // @ts-check
 
+const DiscordTypes = require("discord-api-types/v10")
 const assert = require("assert/strict")
 const {z} = require("zod")
-const {H3Event, defineEventHandler, sendRedirect, useSession, createError, getValidatedQuery, readValidatedBody} = require("h3")
+const {H3Event, defineEventHandler, sendRedirect, createError, getValidatedQuery, readValidatedBody, setResponseHeader} = require("h3")
 const {randomUUID} = require("crypto")
 const {LRUCache} = require("lru-cache")
 const Ty = require("../../types")
 const uqr = require("uqr")
 
-const {discord, as, sync, select} = require("../../passthrough")
+const {id: botID} = require("../../../addbot")
+const {discord, as, sync, select, from, db} = require("../../passthrough")
 /** @type {import("../pug-sync")} */
 const pugSync = sync.require("../pug-sync")
 /** @type {import("../../d2m/actions/create-space")} */
 const createSpace = sync.require("../../d2m/actions/create-space")
+/** @type {import("../auth")} */
+const auth = require("../auth")
+/** @type {import("../../discord/utils")} */
+const utils = sync.require("../../discord/utils")
 const {reg} = require("../../matrix/read-registration")
 
 const schema = {
 	guild: z.object({
 		guild_id: z.string().optional()
 	}),
+	qr: z.object({
+		guild_id: z.string().optional()
+	}),
 	invite: z.object({
 		mxid: z.string().regex(/@([^:]+):([a-z0-9:-]+\.[a-z0-9.:-]+)/),
 		permissions: z.enum(["default", "moderator", "admin"]),
@@ -43,10 +52,31 @@ function getAPI(event) {
 const validNonce = new LRUCache({max: 200})
 
 /**
- * @param {string} guildID
- * @param {Ty.R.Hierarchy[]} rooms
+ * Modifies the input, removing items that don't pass the filter. Returns the items that didn't pass.
+ * @param {T[]} xs
+ * @param {(x: T, i?: number) => any} fn
+ * @template T
+ * @returns T[]
  */
-function getChannelRoomsLinks(guildID, rooms) {
+function filterTo(xs, fn) {
+	/** @type {T[]} */
+	const filtered = []
+	for (let i = xs.length-1; i >= 0; i--) {
+		const x = xs[i]
+		if (!fn(x, i)) {
+			filtered.unshift(x)
+			xs.splice(i, 1)
+		}
+	}
+	return filtered
+}
+
+/**
+ * @param {DiscordTypes.APIGuild} guild
+ * @param {Ty.R.Hierarchy[]} rooms
+ * @param {string[]} roles
+ */
+function getChannelRoomsLinks(guild, rooms, roles) {
 	function getPosition(channel) {
 		let position = 0
 		let looking = channel
@@ -58,37 +88,83 @@ function getChannelRoomsLinks(guildID, rooms) {
 		return position
 	}
 
-	let channelIDs = discord.guildChannelMap.get(guildID)
+	let channelIDs = discord.guildChannelMap.get(guild.id)
 	assert(channelIDs)
 
 	let linkedChannels = select("channel_room", ["channel_id", "room_id", "name", "nick"], {channel_id: channelIDs}).all()
-	let linkedChannelsWithDetails = linkedChannels.map(c => ({channel: discord.channels.get(c.channel_id), ...c})).filter(c => c.channel)
+	let linkedChannelsWithDetails = linkedChannels.map(c => ({channel: discord.channels.get(c.channel_id), ...c}))
+	let removedUncachedChannels = filterTo(linkedChannelsWithDetails, c => c.channel)
 	let linkedChannelIDs = linkedChannelsWithDetails.map(c => c.channel_id)
 	linkedChannelsWithDetails.sort((a, b) => getPosition(a.channel) - getPosition(b.channel))
 
 	let unlinkedChannelIDs = channelIDs.filter(c => !linkedChannelIDs.includes(c))
-	let unlinkedChannels = unlinkedChannelIDs.map(c => discord.channels.get(c)).filter(c => c && [0, 5].includes(c.type))
+	/** @type {DiscordTypes.APIGuildChannel[]} */ // @ts-ignore
+	let unlinkedChannels = unlinkedChannelIDs.map(c => discord.channels.get(c))
+	let removedWrongTypeChannels = filterTo(unlinkedChannels, c => c && [0, 5].includes(c.type))
+	let removedPrivateChannels = filterTo(unlinkedChannels, c => {
+		const permissions = utils.getPermissions(roles, guild.roles, botID, c["permission_overwrites"])
+		return utils.hasPermission(permissions, DiscordTypes.PermissionFlagsBits.ViewChannel)
+	})
 	unlinkedChannels.sort((a, b) => getPosition(a) - getPosition(b))
 
 	let linkedRoomIDs = linkedChannels.map(c => c.room_id)
-	let unlinkedRooms = rooms.filter(r => !linkedRoomIDs.includes(r.room_id) && !r.room_type)
+	let unlinkedRooms = [...rooms]
+	let removedLinkedRooms = filterTo(unlinkedRooms, r => !linkedRoomIDs.includes(r.room_id))
+	let removedWrongTypeRooms = filterTo(unlinkedRooms, r => !r.room_type)
 	// https://discord.com/developers/docs/topics/threads#active-archived-threads
 	// need to filter out linked archived threads from unlinkedRooms, will just do that by comparing against the name
-	unlinkedRooms = unlinkedRooms.filter(r => r.name && !r.name.match(/^\[(🔒)?⛓️\]/))
+	let removedArchivedThreadRooms = filterTo(unlinkedRooms, r => r.name && !r.name.match(/^\[(🔒)?⛓️\]/))
 
-	return {linkedChannelsWithDetails, unlinkedChannels, unlinkedRooms}
+	return {
+		linkedChannelsWithDetails, unlinkedChannels, unlinkedRooms,
+		removedUncachedChannels, removedWrongTypeChannels, removedPrivateChannels, removedLinkedRooms, removedWrongTypeRooms, removedArchivedThreadRooms
+	}
 }
 
 as.router.get("/guild", defineEventHandler(async event => {
 	const {guild_id} = await getValidatedQuery(event, schema.guild.parse)
-	const session = await useSession(event, {password: reg.as_token})
-	const row = select("guild_space", ["space_id", "privacy_level"], {guild_id}).get()
+	const session = await auth.useSession(event)
+	const managed = await auth.getManagedGuilds(event)
+	const row = from("guild_active").join("guild_space", "guild_id", "left").select("space_id", "privacy_level", "autocreate").where({guild_id}).get()
 	// @ts-ignore
 	const guild = discord.guilds.get(guild_id)
 
 	// Permission problems
-	if (!guild_id || !guild || !session.data.managedGuilds || !session.data.managedGuilds.includes(guild_id)) {
-		return pugSync.render(event, "guild_access_denied.pug", {guild_id})
+	if (!guild_id || !guild || !managed.has(guild_id) || !row) {
+		return pugSync.render(event, "guild_access_denied.pug", {guild_id, row})
+	}
+
+	// Self-service guild that hasn't been linked yet - needs a special page encouraging the link flow
+	if (!row.space_id && row.autocreate === 0) {
+		const spaces = db.prepare("SELECT room_id, type, name, topic, avatar FROM invite LEFT JOIN guild_space ON invite.room_id = guild_space.space_id WHERE mxid = ? AND space_id IS NULL AND type = 'm.space'").all(session.data.mxid)
+		return pugSync.render(event, "guild_not_linked.pug", {guild, guild_id, spaces})
+	}
+
+	const roles = guild.members?.find(m => m.user.id === botID)?.roles || []
+
+	// Easy mode guild that hasn't been linked yet - need to remove elements that would require an existing space
+	if (!row.space_id) {
+		const links = getChannelRoomsLinks(guild, [], roles)
+		return pugSync.render(event, "guild.pug", {guild, guild_id, ...links, ...row})
+	}
+
+	// Linked guild
+	const api = getAPI(event)
+	const rooms = await api.getFullHierarchy(row.space_id)
+	const links = getChannelRoomsLinks(guild, rooms, roles)
+	return pugSync.render(event, "guild.pug", {guild, guild_id, ...links, ...row})
+}))
+
+as.router.get("/qr", defineEventHandler(async event => {
+	const {guild_id} = await getValidatedQuery(event, schema.qr.parse)
+	const managed = await auth.getManagedGuilds(event)
+	const row = from("guild_active").join("guild_space", "guild_id", "left").select("space_id", "privacy_level", "autocreate").where({guild_id}).get()
+	// @ts-ignore
+	const guild = discord.guilds.get(guild_id)
+
+	// Permission problems
+	if (!guild_id || !guild || !managed.has(guild_id) || !row) {
+		return pugSync.render(event, "guild_access_denied.pug", {guild_id, row})
 	}
 
 	const nonce = randomUUID()
@@ -101,19 +177,7 @@ as.router.get("/guild", defineEventHandler(async event => {
 	const svg = generatedSvg.replace(/viewBox="0 0 ([0-9]+) ([0-9]+)"/, `data-nonce="${nonce}" width="$1" height="$2" $&`)
 	assert.notEqual(svg, generatedSvg)
 
-	// Unlinked guild
-	if (!row) {
-		const links = getChannelRoomsLinks(guild_id, [])
-		return pugSync.render(event, "guild.pug", {guild, guild_id, svg, ...links})
-	}
-
-	// Linked guild
-	const api = getAPI(event)
-	const mods = await api.getStateEvent(row.space_id, "m.room.power_levels", "")
-	const banned = await api.getMembers(row.space_id, "ban")
-	const rooms = await api.getFullHierarchy(row.space_id)
-	const links = getChannelRoomsLinks(guild_id, rooms)
-	return pugSync.render(event, "guild.pug", {guild, guild_id, svg, mods, banned, ...links, ...row})
+	return svg
 }))
 
 as.router.get("/invite", defineEventHandler(async event => {
@@ -126,13 +190,13 @@ as.router.get("/invite", defineEventHandler(async event => {
 
 as.router.post("/api/invite", defineEventHandler(async event => {
 	const parsedBody = await readValidatedBody(event, schema.invite.parse)
-	const session = await useSession(event, {password: reg.as_token})
+	const managed = await auth.getManagedGuilds(event)
 	const api = getAPI(event)
 
 	// Check guild ID or nonce
 	if (parsedBody.guild_id) {
 		var guild_id = parsedBody.guild_id
-		if (!(session.data.managedGuilds || []).includes(guild_id)) throw createError({status: 403, message: "Forbidden", data: "Can't invite users to a guild you don't have Manage Server permissions in"})
+		if (!managed.has(guild_id)) throw createError({status: 403, message: "Forbidden", data: "Can't invite users to a guild you don't have Manage Server permissions in"})
 	} else if (parsedBody.nonce) {
 		if (!validNonce.has(parsedBody.nonce)) throw createError({status: 403, message: "Nonce expired", data: "Nonce means number-used-once, and, well, you tried to use it twice..."})
 		let ok = validNonce.get(parsedBody.nonce)
@@ -164,9 +228,10 @@ as.router.post("/api/invite", defineEventHandler(async event => {
 		( parsedBody.permissions === "admin" ? 100
 		: parsedBody.permissions === "moderator" ? 50
 		: 0)
-	await api.setUserPowerCascade(spaceID, parsedBody.mxid, powerLevel)
+	if (powerLevel) await api.setUserPowerCascade(spaceID, parsedBody.mxid, powerLevel)
 
 	if (parsedBody.guild_id) {
+		setResponseHeader(event, "HX-Refresh", true)
 		return sendRedirect(event, `/guild?guild_id=${guild_id}`, 302)
 	} else {
 		return sendRedirect(event, "/ok?msg=User has been invited.", 302)

src/web/routes/guild.test.js

@@ -1,67 +1,102 @@
 // @ts-check
 
 const tryToCatch = require("try-to-catch")
-const {test} = require("supertape")
-const {router} = require("../../../test/web")
+const {router, test} = require("../../../test/web")
 const {MatrixServerError} = require("../../matrix/mreq")
 
 let nonce
 
 test("web guild: access denied when not logged in", async t => {
-	const content = await router.test("get", "/guild?guild_id=112760669178241024", {
+	const html = await router.test("get", "/guild?guild_id=112760669178241024", {
 		sessionData: {
 		},
 	})
-	t.match(content, /You need to log in to manage your servers./)
+	t.has(html, "You need to log in to manage your servers.")
 })
 
 test("web guild: asks to select guild if not selected", async t => {
-	const content = await router.test("get", "/guild", {
+	const html = await router.test("get", "/guild", {
 		sessionData: {
+			userID: "1",
 			managedGuilds: []
 		},
 	})
-	t.match(content, /Select a server from the top right corner to continue./)
+	t.has(html, "Select a server from the top right corner to continue.")
 })
 
 test("web guild: access denied when guild id messed up", async t => {
-	const content = await router.test("get", "/guild?guild_id=1", {
+	const html = await router.test("get", "/guild?guild_id=1", {
 		sessionData: {
+			userID: "1",
 			managedGuilds: []
 		},
 	})
-	t.match(content, /the selected server doesn't exist/)
+	t.has(html, "the selected server doesn't exist")
+})
+
+test("web qr: access denied when guild id messed up", async t => {
+	const html = await router.test("get", "/qr?guild_id=1", {
+		sessionData: {
+			userID: "1",
+			managedGuilds: []
+		},
+	})
+	t.has(html, "the selected server doesn't exist")
 })
 
 test("web invite: access denied with invalid nonce", async t => {
-	const content = await router.test("get", "/invite?nonce=1")
-	t.match(content, /This QR code has expired./)
+	const html = await router.test("get", "/invite?nonce=1")
+	t.match(html, /This QR code has expired./)
 })
 
 
 
 test("web guild: can view unbridged guild", async t => {
-	const content = await router.test("get", "/guild?guild_id=66192955777486848", {
+	const html = await router.test("get", "/guild?guild_id=66192955777486848", {
 		sessionData: {
 			managedGuilds: ["66192955777486848"]
-		},
-		api: {
-			async getStateEvent(roomID, type, key) {
-				return {}
-			},
-			async getMembers(roomID, membership) {
-				return {chunk: []}
-			},
-			async getFullHierarchy(roomID) {
-				return []
-			}
 		}
 	})
-	t.match(content, /<h1[^<]*Function &amp; Arg/)
+	t.has(html, `<h1 class="s-page-title--header">Function &amp; Arg</h1>`)
 })
 
-test("web guild: can view bridged guild", async t => {
-	const content = await router.test("get", "/guild?guild_id=112760669178241024", {
+test("web guild: unbridged self-service guild prompts log in to matrix", async t => {
+	const html = await router.test("get", "/guild?guild_id=665289423482519565", {
+		sessionData: {
+			managedGuilds: ["665289423482519565"]
+		}
+	})
+	t.has(html, `You picked self-service mode`)
+	t.has(html, `You need to log in with Matrix first`)
+})
+
+test("web guild: unbridged self-service guild asks to be invited", async t => {
+	const html = await router.test("get", "/guild?guild_id=665289423482519565", {
+		sessionData: {
+			mxid: "@user:example.org",
+			managedGuilds: ["665289423482519565"]
+		}
+	})
+	t.has(html, `On Matrix, invite <`)
+})
+
+test("web guild: unbridged self-service guild shows available spaces", async t => {
+	const html = await router.test("get", "/guild?guild_id=665289423482519565", {
+		sessionData: {
+			mxid: "@cadence:cadence.moe",
+			managedGuilds: ["665289423482519565"]
+		}
+	})
+	t.has(html, `<strong>Data Horde</strong>`)
+	t.has(html, `<li>here is the space topic</li>`)
+	t.has(html, `<img class="s-avatar--image" src="https://bridge.example.org/download/matrix/cadence.moe/TLqQOsTSrZkVKwBSWYTZNTrw">`)
+	t.notMatch(html, /<strong>some room<\/strong>/)
+	t.notMatch(html, /<strong>somebody else's space<\/strong>/)
+})
+
+
+test("web guild: can view bridged guild when logged in with discord", async t => {
+	const html = await router.test("get", "/guild?guild_id=112760669178241024", {
 		sessionData: {
 			managedGuilds: ["112760669178241024"]
 		},
@@ -77,14 +112,42 @@ test("web guild: can view bridged guild", async t => {
 			}
 		}
 	})
-	t.match(content, /<h1[^<]*Psychonauts 3/)
-	nonce = content.match(/data-nonce="([a-f0-9-]+)"/)?.[1]
+	t.has(html, `<h1 class="s-page-title--header">Psychonauts 3</h1>`)
+})
+
+test("web guild: can view bridged guild when logged in with matrix", async t => {
+	const html = await router.test("get", "/guild?guild_id=112760669178241024", {
+		sessionData: {
+			mxid: "@cadence:cadence.moe"
+		},
+		api: {
+			async getStateEvent(roomID, type, key) {
+				return {}
+			},
+			async getMembers(roomID, membership) {
+				return {chunk: []}
+			},
+			async getFullHierarchy(roomID) {
+				return []
+			}
+		}
+	})
+	t.has(html, `<h1 class="s-page-title--header">Psychonauts 3</h1>`)
+})
+
+test("web qr: generates nonce", async t => {
+	const html = await router.test("get", "/qr?guild_id=112760669178241024", {
+		sessionData: {
+			managedGuilds: ["112760669178241024"]
+		}
+	})
+	nonce = html.match(/data-nonce="([a-f0-9-]+)"/)?.[1]
 	t.ok(nonce)
 })
 
 test("web invite: page loads with valid nonce", async t => {
-	const content = await router.test("get", `/invite?nonce=${nonce}`)
-	t.match(content, /Invite a Matrix user/)
+	const html = await router.test("get", `/invite?nonce=${nonce}`)
+	t.has(html, "Invite a Matrix user")
 })
 
 
@@ -130,7 +193,7 @@ test("api invite: can invite with valid nonce", async t => {
 					return {membership: "leave"}
 				},
 				async inviteToRoom(roomID, mxidToInvite, mxid) {
-					t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+					t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 					called++
 				},
 				async setUserPowerCascade(roomID, mxid, power) {
@@ -163,7 +226,7 @@ test("api invite: can invite to a moderated guild", async t => {
 		router.test("post", `/api/invite`, {
 			body: {
 				mxid: "@cadence:cadence.moe",
-				permissions: "default",
+				permissions: "admin",
 				guild_id: "112760669178241024"
 			},
 			sessionData: {
@@ -175,11 +238,11 @@ test("api invite: can invite to a moderated guild", async t => {
 					throw new MatrixServerError({errcode: "M_NOT_FOUND", error: "Event not found or something"})
 				},
 				async inviteToRoom(roomID, mxidToInvite, mxid) {
-					t.equal(roomID, "!jjWAGMeQdNrVZSSfvz:cadence.moe")
+					t.equal(roomID, "!jjmvBegULiLucuWEHU:cadence.moe")
 					called++
 				},
 				async setUserPowerCascade(roomID, mxid, power) {
-					t.equal(power, 0)
+					t.equal(power, 100) // moderator
 					called++
 				}
 			}
@@ -205,14 +268,10 @@ test("api invite: does not reinvite joined users", async t => {
 				async getStateEvent(roomID, type, key) {
 					called++
 					return {membership: "join"}
-				},
-				async setUserPowerCascade(roomID, mxid, power) {
-					t.equal(power, 0)
-					called++
 				}
 			}
 		})
 	)
 	t.notOk(error)
-	t.equal(called, 2)
+	t.equal(called, 1)
 })

src/web/routes/info.js

@@ -0,0 +1,65 @@
+// @ts-check
+
+const {z} = require("zod")
+const {defineEventHandler, getValidatedQuery, H3Event} = require("h3")
+const {as, from, sync, select} = require("../../passthrough")
+
+/** @type {import("../../m2d/converters/utils")} */
+const mUtils = sync.require("../../m2d/converters/utils")
+
+/**
+ * @param {H3Event} event
+ * @returns {import("../../matrix/api")}
+ */
+function getAPI(event) {
+	/* c8 ignore next */
+	return event.context.api || sync.require("../../matrix/api")
+}
+
+const schema = {
+	message: z.object({
+		message_id: z.string().regex(/^[0-9]+$/)
+	})
+}
+
+as.router.get("/api/message", defineEventHandler(async event => {
+	const api = getAPI(event)
+
+	const {message_id} = await getValidatedQuery(event, schema.message.parse)
+	const metadatas = from("event_message").join("message_channel", "message_id").join("channel_room", "channel_id").where({message_id})
+		.select("event_id", "event_type", "event_subtype", "part", "reaction_part", "room_id", "source").and("ORDER BY part ASC, reaction_part DESC").all()
+
+	if (metadatas.length === 0) {
+		return new Response("Message not found", {status: 404, statusText: "Not Found"})
+	}
+
+	const events = await Promise.all(metadatas.map(metadata =>
+		api.getEvent(metadata.room_id, metadata.event_id).then(raw => ({
+			metadata: Object.assign({sender: raw.sender}, metadata),
+			raw
+		}))
+	))
+
+	/* c8 ignore next */
+	const primary = events.find(e => e.metadata.part === 0) || events[0]
+	const mxid = primary.metadata.sender
+	const source = primary.metadata.source === 0 ? "matrix" : "discord"
+
+	let matrix_author = undefined
+	if (source === "matrix") {
+		matrix_author = select("member_cache", ["displayname", "avatar_url", "mxid"], {room_id: primary.metadata.room_id, mxid}).get()
+		if (!matrix_author) {
+			try {
+				matrix_author = await api.getProfile(mxid)
+			} catch (e) {
+				matrix_author = {}
+			}
+		}
+		if (!matrix_author.displayname) matrix_author.displayname = mxid
+		if (matrix_author.avatar_url) matrix_author.avatar_url = mUtils.getPublicUrlForMxc(matrix_author.avatar_url)
+		else matrix_author.avatar_url = null
+		matrix_author["mxid"] = mxid
+	}
+
+	return {source, matrix_author, events}
+}))

src/web/routes/info.test.js

@@ -0,0 +1,219 @@
+// @ts-check
+
+const assert = require("assert/strict")
+const {router, test} = require("../../../test/web")
+
+test("web info: returns 404 when message doesn't exist", async t => {
+	const res = await router.test("get", "/api/message?message_id=1")
+	assert(res instanceof Response)
+	t.equal(res.status, 404)
+})
+
+test("web info: returns data for a matrix message and profile", async t => {
+	let called = 0
+	const raw = {
+		type: "m.room.message",
+		room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe",
+		sender: "@cadence:cadence.moe",
+		content: {
+			msgtype: "m.text",
+			body: "testing :heart_pink: :heart_pink: ",
+			format: "org.matrix.custom.html",
+			formatted_body: "testing <img data-mx-emoticon=\"\" src=\"mxc://cadence.moe/AyAhnRNjWyFhJYTRibYwQpvf\" alt=\":heart_pink:\" title=\":heart_pink:\" height=\"32\" vertical-align=\"middle\" /> <img data-mx-emoticon=\"\" src=\"mxc://cadence.moe/AyAhnRNjWyFhJYTRibYwQpvf\" alt=\":heart_pink:\" title=\":heart_pink:\" height=\"32\" vertical-align=\"middle\" />"
+		},
+		origin_server_ts: 1739312945302,
+		unsigned: {
+			membership: "join",
+			age: 10063702303
+		},
+		event_id: "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk",
+		user_id: "@cadence:cadence.moe",
+		age: 10063702303
+	}
+	const res = await router.test("get", "/api/message?message_id=1339000288144658482", {
+		api: {
+			// @ts-ignore - returning static data when method could be called with a different typescript generic
+			async getEvent(roomID, eventID) {
+				called++
+				t.equal(roomID, "!qzDBLKlildpzrrOnFZ:cadence.moe")
+				t.equal(eventID, "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk")
+				return raw
+			},
+			async getProfile(mxid) {
+				called++
+				t.equal(mxid, "@cadence:cadence.moe")
+				return {
+					displayname: "okay 🤍 yay 🤍"
+				}
+			}
+		}
+	})
+	t.deepEqual(res, {
+		source: "matrix",
+		matrix_author: {
+			displayname: "okay 🤍 yay 🤍",
+			avatar_url: null,
+			mxid: "@cadence:cadence.moe"
+		},
+		events: [{
+			metadata: {
+			  event_id: "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk",
+			  event_subtype: "m.text",
+			  event_type: "m.room.message",
+			  part: 0,
+			  reaction_part: 0,
+			  room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe",
+			  sender: "@cadence:cadence.moe",
+			  source: 0
+			},
+			raw
+		}]
+	})
+	t.equal(called, 2)
+})
+
+test("web info: returns data for a matrix message without profile", async t => {
+	let called = 0
+	const raw = {
+		type: "m.room.message",
+		room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe",
+		sender: "@cadence:cadence.moe",
+		content: {
+			msgtype: "m.text",
+			body: "testing :heart_pink: :heart_pink: ",
+			format: "org.matrix.custom.html",
+			formatted_body: "testing <img data-mx-emoticon=\"\" src=\"mxc://cadence.moe/AyAhnRNjWyFhJYTRibYwQpvf\" alt=\":heart_pink:\" title=\":heart_pink:\" height=\"32\" vertical-align=\"middle\" /> <img data-mx-emoticon=\"\" src=\"mxc://cadence.moe/AyAhnRNjWyFhJYTRibYwQpvf\" alt=\":heart_pink:\" title=\":heart_pink:\" height=\"32\" vertical-align=\"middle\" />"
+		},
+		origin_server_ts: 1739312945302,
+		unsigned: {
+			membership: "join",
+			age: 10063702303
+		},
+		event_id: "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk",
+		user_id: "@cadence:cadence.moe",
+		age: 10063702303
+	}
+	const res = await router.test("get", "/api/message?message_id=1339000288144658482", {
+		api: {
+			// @ts-ignore - returning static data when method could be called with a different typescript generic
+			async getEvent(roomID, eventID) {
+				called++
+				t.equal(roomID, "!qzDBLKlildpzrrOnFZ:cadence.moe")
+				t.equal(eventID, "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk")
+				return raw
+			}
+		}
+	})
+	t.deepEqual(res, {
+		source: "matrix",
+		matrix_author: {
+			displayname: "@cadence:cadence.moe",
+			avatar_url: null,
+			mxid: "@cadence:cadence.moe"
+		},
+		events: [{
+			metadata: {
+			  event_id: "$51gH61p_eJc2RylOdE2lAr4-ogP7dS0WJI62lCFzBvk",
+			  event_subtype: "m.text",
+			  event_type: "m.room.message",
+			  part: 0,
+			  reaction_part: 0,
+			  room_id: "!qzDBLKlildpzrrOnFZ:cadence.moe",
+			  sender: "@cadence:cadence.moe",
+			  source: 0
+			},
+			raw
+		}]
+	})
+	t.equal(called, 1)
+})
+
+test("web info: returns data for a discord message", async t => {
+	let called = 0
+	const raw1 = {
+		type: "m.room.message",
+		sender: "@_ooye_accavish:cadence.moe",
+		content: {
+			"m.mentions": {},
+			msgtype: "m.text",
+			body: "brony music mentioned on wikipedia's did you know and also unrelated cat pic"
+		},
+		origin_server_ts: 1749377203735,
+		unsigned: {
+			membership: "join",
+			age: 119
+		},
+		event_id: "$AfrB8hzXkDMvuoWjSZkDdFYomjInWH7jMBPkwQMN8AI",
+		room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
+	}
+	const raw2 = {
+		type: "m.room.message",
+		sender: "@_ooye_accavish:cadence.moe",
+		content: {
+			"m.mentions": {},
+			msgtype: "m.image",
+			url: "mxc://cadence.moe/ABOMymxHcpVeecHvmSIYmYXx",
+			external_url: "https://bridge.cadence.moe/download/discordcdn/112760669178241024/1381212840710504448/image.png",
+			body: "image.png",
+			filename: "image.png",
+			info: {
+				mimetype: "image/png",
+				w: 966,
+				h: 368,
+				size: 166060
+			}
+		},
+		origin_server_ts: 1749377203789,
+		unsigned: {
+			membership: "join",
+			age: 65
+		},
+		event_id: "$43baKEhJfD-RlsFQi0LB16Zxd8yMqp0HSVL00TDQOqM",
+		room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe"
+	}
+	const res = await router.test("get", "/api/message?message_id=1381212840957972480", {
+		api: {
+			// @ts-ignore - returning static data when method could be called with a different typescript generic
+			async getEvent(roomID, eventID) {
+				called++
+				t.equal(roomID, "!kLRqKKUQXcibIMtOpl:cadence.moe")
+				if (eventID === raw1.event_id) {
+					return raw1
+				} else {
+					assert(eventID === raw2.event_id)
+					return raw2
+				}
+			}
+		}
+	})
+	t.deepEqual(res, {
+		source: "discord",
+		matrix_author: undefined,
+		events: [{
+			metadata: {
+			  event_id: "$AfrB8hzXkDMvuoWjSZkDdFYomjInWH7jMBPkwQMN8AI",
+			  event_subtype: "m.text",
+			  event_type: "m.room.message",
+			  part: 0,
+			  reaction_part: 1,
+			  room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe",
+			  sender: "@_ooye_accavish:cadence.moe",
+			  source: 1
+			},
+			raw: raw1
+		}, {
+			metadata: {
+				event_id: "$43baKEhJfD-RlsFQi0LB16Zxd8yMqp0HSVL00TDQOqM",
+				event_subtype: "m.image",
+				event_type: "m.room.message",
+				part: 1,
+				reaction_part: 0,
+				room_id: "!kLRqKKUQXcibIMtOpl:cadence.moe",
+				sender: "@_ooye_accavish:cadence.moe",
+				source: 1
+			},
+			raw: raw2
+		}]
+	})
+	t.equal(called, 2)
+})