cadence/out-of-your-element
cadence/out-of-your-element
4
10
Fork 7
Code Issues 15 Pull requests 3 Releases 29 Activity

heatsync stability fixes #98

Manually merged
cadence merged 1 commit from l1s5/out-of-your-element:heatsync-fixes into main 2026-10-26 11:25:52 +00:00
Conversation 1 Commits 1 Files changed 1 +5 -1
l1s5 commented 2026-10-24 19:37:07 +00:00
Contributor
Copy link

Hi!

I'm coming from a hackerspace where we are running the bridge in production on a NixOS machine, and several module-wide pieces of state have been causing some relatively incomprehensible errors messages.
The crux of the issue seems to be that the content in the nix store might have its ctime bumped at any point, which triggers a heatsync reload. Going over and registering all these into heatsync seems to remove the problems.

Example 1 (note that line numbers might be off due to other patches) 
  Cannot read properties of undefined (reading 'Icons')
      at eval (eval at wrap (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/pug-runtime/wrap.js:6:10), <anonymous>:642:52)
      at template (eval at wrap (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/pug-runtime/wrap.js:6:10), <anonymous>:997:7)
      at /nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/src/web/pug-sync.js:58:12
      at async Object.handler (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2024:19)
      at async Server.toNodeHandle (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2323:7)
Example 2 (same here) 
TypeError: Cannot read properties of undefined (reading 'guilds')
    at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/src/web/routes/stats.js:68:27)
    ... 6 lines matching cause stack trace ...
    at HTTPParser.parserOnHeadersComplete (node:_http_common:125:17) {
  cause: TypeError: Cannot read properties of undefined (reading 'guilds')
      at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/src/web/routes/stats.js:68:27)
      at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2270:34)
      at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2024:31)
      at Server.toNodeHandle (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2323:17)
      at Server.emit (node:events:514:28)
      at Server.emit (node:domain:489:12)
      at parserOnIncoming (node:_http_server:1293:12)
      at HTTPParser.parserOnHeadersComplete (node:_http_common:125:17)

I think this PR covers most of it, but I'm not completely sure we found all of them.

There's also a commit disabling reloads on ctime-only bumps, so that any new pieces of non-synced state won't affect NixOS setups anymore (for better or worse).

Hi! I'm coming from a hackerspace where we are running the bridge in production on a NixOS machine, and several module-wide pieces of state have been causing some relatively incomprehensible errors messages. The crux of the issue seems to be that the content in the nix store might have its `ctime` bumped at any point, which triggers a heatsync reload. Going over and registering all these into heatsync seems to remove the problems. <details> <summary>Example 1 (note that line numbers might be off due to other patches)</summary> ``` Cannot read properties of undefined (reading 'Icons') at eval (eval at wrap (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/pug-runtime/wrap.js:6:10), <anonymous>:642:52) at template (eval at wrap (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/pug-runtime/wrap.js:6:10), <anonymous>:997:7) at /nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/src/web/pug-sync.js:58:12 at async Object.handler (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2024:19) at async Server.toNodeHandle (/nix/store/h53sfx08n9nh55ccnfh6iv49vahhgqk7-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2323:7) ``` </details> <details> <summary>Example 2 (same here)</summary> ``` TypeError: Cannot read properties of undefined (reading 'guilds') at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/src/web/routes/stats.js:68:27) ... 6 lines matching cause stack trace ... at HTTPParser.parserOnHeadersComplete (node:_http_common:125:17) { cause: TypeError: Cannot read properties of undefined (reading 'guilds') at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/src/web/routes/stats.js:68:27) at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2270:34) at Object.handler (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2024:31) at Server.toNodeHandle (/nix/store/nmrhlf98p8fcc28ync12gvyfyswqfkrx-out-of-your-element-3.7.0/share/ooye/node_modules/h3/dist/index.cjs:2323:17) at Server.emit (node:events:514:28) at Server.emit (node:domain:489:12) at parserOnIncoming (node:_http_server:1293:12) at HTTPParser.parserOnHeadersComplete (node:_http_common:125:17) ``` </details> I think this PR covers most of it, but I'm not completely sure we found all of them. There's also a commit disabling reloads on `ctime`-only bumps, so that any new pieces of non-synced state won't affect NixOS setups anymore (for better or worse). 
Several pieces of module-wide state was being reset on reloads caused by
heatsync, leaving the bridge in a weird state. This commit goes over a
bunch of modules and wraps most state in `sync.remember` constructors.
`fs.watchFile()` calls its listener on all changes to any file attribute,
including `ctime`. Hardlinking a file changes its `ctime`, which is
what `nix store optimize` does when it deduplicates identical files,
causing an unnecessary reload in production environments.

This change makes it so we completely ignore ctime changes, and only
reload when mtime/size changes.
cadence manually merged commit e898258a69 into main 2026-10-26 11:25:52 +00:00
cadence commented 2026-10-26 11:25:57 +00:00
Owner
Copy link

Thank you!

Thank you!
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backfill
properly integrated backfill feature performed on room creation

  
blocking
critical issue that blocks release of milestone

  
design needed
must prototype and discuss to find the best way forward - suggestions appreciated!

  
moderately approachable
moderate investigation, deduction, and adaptability required - or covers multiple areas of code

  
very approachable
low mental complexity, few concepts, focused scope, no investigation

  
web
No labels
backfill
blocking
design needed
moderately approachable
very approachable
web
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cadence/out-of-your-element!98
No description provided.