225 Commits

  • refac: limit open collaborative documents per user (#33864)
    The limit on open collaborative documents now applies to all of a user's connections together instead of each connection separately.
  • refac: apply the Notes switch to live note editing (#33797)
    Live note editing now follows the instance's Notes switch and the Notes permission, like the rest of the Notes feature.
  • refac: restrict collaborative document rooms to notes (#33752)
    Collaborative editing now only opens note documents. Other document ids are ignored.
  • perf: chat access checks no longer load the whole conversation (#33667)
    Typing indicators, sending a message, shared links, comments and file access all check whether the user may access the chat, and each check loaded the full chat, images included, only to look at who owns it and which folder it is in. On large chats that delayed every other request on the same server process by up to a few hundred milliseconds per check. The check now reads just those few fields, and who gets access is unchanged.
    
    Measured on SQLite, best of 5 runs, sizes are the chat as stored in the database:
    
    | Chat | Checked for | Before | After |
    | --- | --- | --- | --- |
    | 7 MB, text only | the chat owner | 34.6 ms | 0.36 ms |
    | 7 MB, text only | a user with access through a shared folder | 35.1 ms | 2.58 ms |
    | 57 MB, with images | the chat owner | 249.8 ms | 0.72 ms |
    | 57 MB, with images | a user with access through a shared folder | 249.0 ms | 2.80 ms |
  • perf: faster filter lookups on chat requests when Redis is enabled (#33668)
    With Redis enabled, each chat request and each title or tag generation looks up which pipeline filters apply, and every lookup parsed the full model list stored in Redis from scratch. Each worker now keeps the parsed list and parses it again only when the model list changes. The lookup returns the same filters as before, at the cost of about 3 MB of memory per worker for 1000 models.
    
    Measured per lookup against a local Redis; the larger list size in each pair comes from models with profile images:
    
    | Models | Model list size | Before | After |
    | --- | --- | --- | --- |
    | 200 | 0.17 MB | 0.42 ms | 0.05 ms |
    | 200 | 2.2 MB | 1.16 ms | 0.05 ms |
    | 1000 | 0.83 MB | 2.43 ms | 0.06 ms |
    | 1000 | 11 MB | 6.21 ms | 0.10 ms |
  • fix: notes over 2 MB open empty and cannot be edited (#33618)
    A note over 2 MB, which one pasted screenshot can reach because images are embedded in the note, opened as an empty page and discarded what you typed. Live editing of notes had its own 2 MB limit, smaller than what the server accepts in one transfer, so it refused to load such notes. That limit now matches the server's 16 MB transfer limit, so any note the server can receive opens and saves again.
    
    Fixes #33606
  • perf: renaming a folder or deleting all chats takes seconds when many users are online (#32888)
    * perf: renaming a folder or deleting all chats takes seconds when many users are online
    
    Renaming, sharing, moving or deleting a folder, and deleting all chats, each made the server recheck access for every chat that any online user had open in a browser tab, one database lookup at a time, before the request finished, and every one of those open chats reloaded its contents afterwards. Any user can do these, so with a few thousand people online every such click took several seconds and reloaded everyone's open chat. Owners keep access to their own chats (a role change or account deletion already disconnects them), so the recheck now skips owners entirely, with no lookup and no reload, and only checks people viewing someone else's chat and chats that no longer exist. Anyone who lost access is still removed from the chat as before.
    
    Postgres 16, one open chat per online user, 2% of them viewing someone else's chat, time for a folder rename (the other actions above run the same recheck):
    
    | Online users | No Redis, before | No Redis, after | Redis, before | Redis, after |
    | --- | --- | --- | --- | --- |
    | 1,000 | 2.2 s | 0.12 s | 2.5 s | 0.23 s |
    | 3,000 | 6.9 s | 0.60 s | 7.7 s | 0.78 s |
    
    * refac: collect the open chats with a plain loop
    
    Same behaviour, the owner lookup now reads more plainly.
    
    * refac: shorten the owner comment in the chat access sweep
  • fix: new websocket connections hang until restart after a Redis reconnect (#33009)
    Since 0.12.3, with the Redis websocket manager, WEBSOCKET_REDIS_ROOM_CHANNELS on (the default) and retry_on_timeout in WEBSOCKET_REDIS_OPTIONS (as in the documented example), an instance could stop accepting new browser connections until it was restarted. Before accepting a browser, the instance asks Redis to confirm its subscriptions. If its Redis connection dropped while that confirmation was still on its way, the Redis client reconnected silently and the confirmation was lost, so that browser and every later one waited forever. The instance now asks Redis again after the reconnect, so the waiting browser connects as soon as Redis is back.
    
    Fixes #33008
  • perf: changing permissions on a large group is slow (#32883)
    When a group change affects its members (new permissions, parent group or default models, or deleting the group), the server makes every member's browser reconnect so it picks up the new permissions without a page reload. To find each member's connections it read through the whole list of connected users again for every single member, so saving a 2,000-member group with 3,000 people online spent almost 9 seconds on this with Redis as the websocket manager, and about 1 second without. It now finds all members' connections in one pass.
    
    Changing one permission on a 2,000-member group, time the save spends making members reconnect:
    
    | Online users | Redis, before | Redis, after | No Redis, before | No Redis, after |
    | --- | --- | --- | --- | --- |
    | 1,000 | 2.8 s | 0.15 s | 0.09 s | <0.01 s |
    | 3,000 | 8.7 s | 0.28 s | 1.2 s | <0.01 s |
  • fix: cannot connect through Redis Sentinels that require a password (#32241)
    When the Redis Sentinels are protected with a password, every Redis operation failed with "No master found", because the password in REDIS_URL was used for the Redis server itself but never for the Sentinels. The Sentinels now get the same credentials, both for REDIS_URL and for WEBSOCKET_REDIS_URL. Sentinels without a password keep working, and Sentinel credentials set by hand in WEBSOCKET_REDIS_OPTIONS still take precedence.
    
    Fixes #32240
  • refac: bound collaborative document buffers (#31603)
    Collaborative document buffers now keep a size budget per document and a limit on open documents per connection, only store well-formed updates and are keyed by the document id unchanged.
  • fix: Direct Connection replies are saved scrambled or empty (#31979)
    * fix: Direct Connection replies are saved scrambled or empty
    
    With a Direct Connection the browser tab forwards the model's reply to the server piece by piece. Since the server started checking the tab's session again for every one of those pieces, the pieces could overtake each other while the checks ran, so the saved reply came out in the wrong order, or empty when the end of the stream arrived first. Pieces from one tab are now handled one after another in the order they arrived, and each one is still checked.
    
    Fixes #31953
    
    * fix: check Direct Connection reply pieces side by side while keeping them in order
    
    Handling one tab's reply pieces strictly one after another also made each piece wait for the previous piece's session check, so a fast 2000-piece reply took 12 to 22% longer to save than without the ordering. Each piece's check now starts as soon as it arrives and only the hand-over waits its turn, so replies save as fast as before and still in order. Once a check fails, for example after a sign-out, no later piece from that tab gets through either.
  • fix: large pastes into a note are lost after a reload (#31893)
    Pasting a large block of text (around 150 KB or more) into a note made the note's live connection drop and reconnect, and the text was never saved, so a reload showed the note without it. Each edit message carries the whole note in several formats, which pushed it past the server's 1 MB cap on a single message. That cap is now 16 MiB, the same size the server already accepts for a websocket message, so notes with up to about 2 MB of text save again.
    
    Fixes #26140
  • refac: apply the Notes permission to live note editing (#31552)
    Opening a note for live collaborative editing now follows the same Notes permission as the rest of the Notes feature.
  • fix: apply the Channels permission to real-time channel messages (#31578)
    Real-time channel messages are now only delivered to users who have the Channels permission set in the admin user permission settings.