makepad/apps/desklink/PROTOCOL.md

Desk Link protocol, version 7

Desk Link is how a viewer (this app) sees and drives a desktop (the window manager's desk_link server, apps/wm/src/desk_link/). It runs between machines on one LAN only: no relays, no NAT traversal, no discovery service. The one implementation of the format is libs/desk_link, used by both ends.

Transport and security

• TCP, default port 8790, wrapped in TLS on each OS's own TLS stack (makepad_network::tls).
• The desktop presents a self-signed certificate (~/.makepad/wm/desk-link/tls). The viewer pins it ssh-style in ~/.makepad/ai-hub/known_nodes (host:port). A changed certificate is reported loudly and the connection goes on: the fleet proof below is what protects the desktop.
• The desktop serves only when all of these hold. Otherwise nothing listens:
  • its settings enable it (~/.makepad/wm/desk-link.conf, enabled = on);
  • the bind address is loopback, a private LAN address or 0.0.0.0;
  • its fleet key (fleet.key, the AI hub's <cache>/fleet) loads.
• Peer allowlist. Each server serves only the addresses in its allow_peers setting (the viewer machines). The list is required: an empty or missing list means the server does not start, so it never runs open to the whole LAN on a credential alone.
  • Anything else is closed before TLS, and the address is checked again after authentication. The peer must also be a LAN address (loopback, RFC 1918, link-local, IPv6 ULA).
  • Refusals are logged with the address only, at most once a minute per address.
  • Five failed credentials within a minute ban the address for 15 minutes. Sessions are capped at 8, with at most 4 per address.
  • IP pinning assumes the viewer machines have a DHCP reservation, so their address does not change.
• Authentication. The viewer's first message carries a fleet proof: MKC2 <id>.<role>.<expiry>.<HMAC-SHA256(secret, "mkfleet2 proof|" + certificate sha256 hex)>.
  • The secret comes from the viewer's credential (~/.makepad/ai-hub/client.credential).
  • The proof is bound to the certificate the desktop presented, so a proof captured by anything else is useless.
  • The desktop requires role lan, an unexpired credential, and an id not on its revoked list.
  • Anything else is answered Refused and closed.

Framing

Every message is:

type    u8
length  u32 LE        payload bytes (at most 16 MiB; more closes the connection)
payload

Fixed-size fields are little-endian. str and bytes are u32 LE length + bytes (str = UTF-8). bool is a u8 (0 or 1).

Viewer -> desktop

type	name	payload
1	Hello	version u16, name str, authorization str, screen i32 (-1 = the desktop's default), paused bool, caps u32 (1 = the viewer's decoder takes 4:4:4: H.264 High 4:4:4 Predictive, and with 4 H.265 Format range extensions 4:4:4; 2 = view-only: its Window never decides the size of a desktop that follows its viewer, e.g. a probe; 4 = it decodes H.265)
2	Pause	— no pictures for this session; pings continue
3	Resume	— pictures again, starting with a key frame
4	SelectScreen	index u32 (into Welcome.screens)
5	RequestKeyframe	— (sent by the viewer after a decode error)
7	Refresh	hz u16: the viewer's display refresh, measured from its vsync (0 = not known). Sent after Hello and on every change.
8	Window	width u32, height u32, scale u16: the viewer's picture area in device pixels and its scale (hundredths, 200 = 2.00x). Sent once it has held still for 250 ms, after the Welcome and after every resize. A desktop whose size follows its viewer (a virtual display set to follow) takes this size and scale from the active viewer (the watching viewer that sent input last), so the picture is shown 1:1.
10	Key	down bool, keycode u32 (makepad KeyCode index), mods u8 (1 shift, 2 control, 4 alt, 8 logo)
11	Pointer	x u16, y u16: position over the stream picture, 0..=65535 on each axis
12	Button	down bool, button u8 (0 left, 1 right, 2 middle, 3 back, 4 forward)
13	Scroll	dx i32, dy i32 in 1/120 notches; +dy = wheel away from the user
14	Gamepad	slot u8, buttons u32, left_x i16, left_y i16, right_x i16, right_y i16, left_trigger u8, right_trigger u8
15	GamepadGone	slot u8
20	Ping	client_ns u64 (the viewer's clock)
21	Stats	frame u32, decode_us u32, present_us u32: about once a second; frame the last picture shown and present_us its decode-end to on screen; decode_us a picture's decode (its slowest stream's), the mean over the pictures since the last Stats (version 7 viewers before this rule sent the last picture's)
22	Clipboard	text str (the desktop's clipboard takes it)
24	Encoding	chroma u8, codec u8, preset u8, mbps u16, fps u16: how this viewer wants the stream encoded, 0 = auto each. chroma 1 4:4:4, 2 4:2:0; codec 1 H.264, 2 H.265; preset 1 sharp (the encoder's best pictures per bit), 2 fast (its fastest encode); mbps the most the stream may take (all tiles; 0 = no cap of its own: the desktop streams at most the lowest cap of its watching viewers); fps pictures a second (0 = its display's refresh). Sent after the Hello and on every change. The active viewer's choices decide the stream (the watching viewer that sent input last, view-only viewers last); the decoders of every watching viewer (Hello.caps) and the desktop's settings bound them.

Gamepad buttons: bit 0 A, 1 B, 2 X, 3 Y, 4 left shoulder, 5 right shoulder, 6 select, 7 start, 8 home, 9 left thumb, 10 right thumb, 11-14 d-pad up/down/left/right. Sticks run −32767..32767, up positive (makepad's convention); triggers 0..255.

Desktop -> viewer

type	name	payload
100	Welcome	version u16, session u32, screens (n u32, then per screen: name str, x u32, y u32, width u32, height u32), screen u32, stream_w u32, stream_h u32, input bool, input_note str, server str
101	Refused	reason str (then the desktop closes)
102	Video	frame u32, key bool, built_ns u64, encoded_ns u64, sent_ns u64, stream u8, parts u8, x0 u32, y0 u32, codec u8 (1 H.264, 2 H.265), data bytes (see Streams)
103	Cursor	x u16, y u16 (over the stream picture), shape u8, visible bool
104	Pong	client_ns u64 (echoed), server_ns u64
105	Stream	screen u32, stream_w u32, stream_h u32 (the stream shows another screen now)
106	Clipboard	text str
107	Bye	reason str (then the desktop closes)
108	Rate	fps u16, kbps u32, limit u8, full_chroma bool, codec u8, preset u8, streams u8, why str: the pictures a second the desktop streams at now (the lowest of its settings' cap (fps, default 120), its streamed screen's refresh and every watching viewer's Refresh; a still desktop sends fewer), the stream's rate (kbit/s, all tiles; 0 = by picture size), what holds the rate (0 starting, 1 nothing: rising, 2 the link, 3 the encoder, 4 a cap, 5 a fixed setting), whether it is 4:4:4, its codec (1 H.264, 2 H.265), the encoder's preset (1 sharp, 2 fast), the encoder streams a picture is cut into, and why: what the stream is and why, in words (e.g. "fast: the encoder was behind at sharp"). Sent after Welcome and on every change.

Screens are in desktop device pixels. input = false means viewer input is not injected (for example, no /dev/uinput access); input_note says why.

Video

• Codec. H.265 when the desktop's settings allow it (codec = auto, the default), its GPU encodes it and every watching viewer's Hello says it decodes it (caps 4), and the active viewer's Encoding does not ask for H.264: Main (4:2:0 8-bit) or Format range extensions 4:4:4 8-bit. Else H.264 High, 4:2:0 8-bit, or High 4:4:4 Predictive (4:4:4 8-bit, CAVLC). 4:4:4 when the desktop's settings ask for it (chroma = 444, the default) and every watching viewer's Hello says it decodes it (caps 1). BT.709 limited range (written in the SPS VUI), Annex-B access units (one picture each). Each Video names its codec; a change of codec starts with a key frame in the new one.
  • Every key frame (IDR) starts with its parameter sets (H.264: SPS and PPS; H.265: VPS, SPS and PPS).
  • There are no B frames. Every P frame references the picture before it, and output order is decode order (POC type 2).
  • The VUI says max_num_reorder_frames = 0 and max_dec_frame_buffering = 1, so decoders output each picture as soon as it is decoded.
• GOP. Open-ended: an IDR comes at the start, after Resume, after RequestKeyframe and after a screen change.
• Resolution. The picture is the selected screen at its own output size. Width is rounded down to a multiple of 4 and height to a multiple of 2; the coded size is padded to macroblocks and cropped in the SPS. The viewer never renegotiates resolution: it scales the picture to its window, aspect-locked and letterboxed, or shows it 1:1. It maps input back through the same placement.
• Damage-driven. A picture is made only when the desktop's own frame changed (not for a cursor move), at most one per 1/fps seconds (fps = 60 by default). A change that comes too soon is encoded when it is due. An idle desktop sends nothing.
• Rate control. VBR at one rate for the whole stream, split over the tiles by area. A fixed kbps setting, or adaptive: it starts at 250 Mbit/s, goes down below what got out when a viewer falls behind (link-bound), and creeps back up when clean, with no ceiling but the desktop's max_mbps setting (none by default) and the viewers' Encoding caps. When the encoder cannot keep a picture's time (its slowest tile takes most of it and frames wait for it), that size gives up encoder work one step a second, at the same picture rate: sharp in as few streams as the size allows, then a stream per encode engine (bands encoded at once), then the fast preset, and 4:2:0 last, only after three seconds still behind at fast and only when the active viewer's chroma is auto. It steps back after a while well clear of the picture's time. When a watching viewer's decode of a picture (Stats.decode_us) takes 90% of a picture's time or more, the size streams in more H.265 bands (enough for each to decode in 70% of it, at most twice as many a step and 8 in all; the viewer decodes the bands at once), and the rate stays: that second's backlog is the decoder's. A step that does not cut the decode time by a fifth goes back and is the last. The bands stay until a viewer leaves. Rate tells the viewers each change.
• Backpressure. The desktop never drops an encoded picture: the next one references it. While a watching viewer has more than 2 pictures unsent, it makes none. The newest picture goes out once the viewer has caught up.

Streams (tiles and interleaving)

H.265 is one stream as large as the encoder takes (NVIDIA: 8192 wide; no seams, one decoder) while one hardware decoder keeps up: at most 1.3 Gpixel a second per stream (an Apple M4 decodes 7680x2160 4:4:4 in 7.9 ms alone, in two bands at once in 6.7 ms). Larger is cut into bands from the first picture (7680x2160 at 120: two bands; 7680x4320 at 120: four), more when a viewer's measured decode asks (above), and when one engine cannot keep up (the effort steps above) at least one band per encode engine, encoded at once. Bands are whole macroblock rows, at least 64 rows, and carry parts, x0, y0 as tiles do.

A desktop whose GPU has several hardware encode engines encodes one H.264 stream per engine; each stream is a stream of its own (its own SPS and PPS, its own decoder on the viewer):

• Tiles (the default: the lowest latency). Each picture is cut into columns x bands tiles encoded at once, the engines in turn: the fewest tiles that give every engine the same share, that fit the encoder's largest picture (NVIDIA's H.264: 4096 wide, so a 7680-wide desktop has two columns or more) and that a decoder takes at the stream rate (H.264 level 5.2: 2,073,600 macroblocks a second per stream); tiles = CxR in the desktop's settings picks them. Cuts are whole 16-pixel macroblocks except the last column and band, and cover the picture exactly once. Every part of picture frame carries parts = the tile count and its tile's first column x0 and row y0; the tile's size is its stream's SPS size. Tiles of two pictures may be in flight at once on the desktop; each picture's parts are still sent back to back. The parts of one picture are sent back to back, and the viewer shows a picture only once it has all its parts (never a torn picture).
• Interleave (the most pictures per second, at one picture's latency). The engines take whole pictures in turn: picture frame comes from stream frame % streams with parts = 1, x0 = y0 = 0 (a picture wider than the encoder takes is cut into columns instead: tiles).
• The desktop's settings pick the split (split = tiles | interleave | off).
• A key frame request makes every stream start again from a key frame.
• One engine (or split = off): one stream, stream 0, parts 1, x0 0, y0 0 (or columns, as above).

Cursor

The cursor is not in the video. The desktop sends Cursor whenever the pointer's position over the streamed screen or its shape changes, apart from the pictures, so it never waits for one. shape indexes this table; an unknown index means the default arrow:

0 Hidden  1 Default  2 Crosshair  3 Hand  4 Arrow  5 Move  6 Text  7 Wait  8 Help  9 NotAllowed
10 Grab  11 Grabbing  12 Progress  13 ContextMenu  14 Cell  15 VerticalText  16 Alias  17 Copy
18 NoDrop  19 AllScroll  20 ZoomIn  21 ZoomOut  22 NResize  23 NeResize  24 EResize  25 SeResize
26 SResize  27 SwResize  28 WResize  29 NwResize  30 NsResize  31 NeswResize  32 EwResize
33 NwseResize  34 ColResize  35 RowResize

The viewer gives its own OS cursor that shape while its pointer is over the picture, so the local pointer has no added latency. It draws a cursor sprite at the desktop's position only when the pointer is moved on the other side.

Input

The desktop injects input as local devices (Linux: uinput), for the session's life:

• an absolute pointer over the whole desktop, with five buttons and both wheels;
• a keyboard;
• one gamepad per viewer slot.

Its own input stack treats them like plugged-in hardware: key chords, modifiers, focus routing to hosted apps and Wayland clients, and gamepads in every app. Every held key and button is released when the session ends. The wheel is passed as whole notches; parts of a notch carry over to the next scroll.

Latency

All server times are on the desktop's clock. The viewer keeps an offset from its pings, taken from the smallest round trip of the session, NTP-style:

offset = server_ns − (client_ns + rtt / 2)

The figures, for each picture shown:

figure	measured as
encode	encoded_ns − built_ns: from the start of the desktop's frame build to the bitstream being ready (render + NV12 pack + hardware encode + readout)
network	received − (sent_ns − offset): from the desktop's socket write to the viewer's read
decode	the hardware decoder's time for the picture
present	from decode end to the frame that draws the picture
total	the sum: from the desktop's frame build to the viewer's frame

The viewer shows these in its overlay and sends Stats once a second.

Sessions

A viewer keeps every desktop in its list connected. Those it does not show are paused: no pictures, and a ping each second keeps their latency current. Showing another desktop is Pause to the old one and Resume to the new one, which answers with a key frame: one round trip plus one encode. It needs no new TLS handshake.

The desktop serves any number of sessions. They watch the same screen, and the stream follows the last SelectScreen.