53 Commits

• refac
  Timothy Jaeryang Baek · 2026-11-02 08:48:51 +09:00
  d99ec1c02c
• fix: requests hang or fail with 503 after pages are closed while loading (#32617)

  Closing a page while the server was still sending it data (a file's processing status, a chat export) could interrupt a database read halfway. On SQLite the connection used for it was then never given back, and a later unrelated request could hang forever when that connection was finally cleaned up; once enough were lost, every request needing the database waited 30 seconds and failed with a 503. The connection is now kept and rolled back instead of being thrown away inside the interrupted request, and the session is closed once instead of twice, as either alone still lost connections. Closing 600 chat exports and 300 file status streams mid-load on SQLite used to lose 60 (the whole pool in that test) and 56 connections, now none; Postgres lost none before or after.
  
  Fixes #32609

  Classic298 · 2026-10-21 03:23:05 +09:00
  2355b8a369
• fix: non-English text is missed by searches and counted six times against size limits (#31615)

  With ENABLE_ORJSON off (the default), non-English letters were saved to the database as escape codes, so "Ü" was stored as \u00dc. Searches that ignore upper and lower case compare against that saved text, so they missed any match that differs only in the case of a non-English letter: filtering models by the tag "Überblick" found nothing on Postgres, and searching automations for "отчёт" missed a prompt containing "Отчёт" on SQLite and Postgres. The 100,000 character size limit for user and chat variables counted the escape codes too, so Cyrillic or Chinese variables were refused as too large (or chat variables silently came out empty in the system prompt) at about a sixth of that size. Non-English text is now saved as written, which is how it is already saved with ENABLE_ORJSON on, so nothing changes for those instances, and the limit counts real characters. Anything saved before this keeps the escape codes until it is next edited.

  Classic298 · 2026-09-30 19:19:29 +04:00
  321a24dfea
• refac: build the SQLite LIKE regex from wildcard-split segments (#30393)

  Each '%'-separated segment of the LIKE pattern is now matched with an atomic group.

  Classic298 · 2026-09-23 23:55:57 -04:00
  d9dff09c30
• refac
  Timothy Jaeryang Baek · 2026-09-06 17:13:32 -04:00
  d27aa72ab4
• refac
  Timothy Jaeryang Baek · 2026-08-30 21:36:17 -04:00
  49aab7451c
• refac
  Timothy Jaeryang Baek · 2026-08-30 12:09:34 -04:00
  26f37426b7
• fix: apply RDS IAM token auth to the pgvector engine (#27754)

  With `DATABASE_ENABLE_IAM_TOKEN_AUTH=true` and `VECTOR_DB=pgvector`, startup failed at vector store initialisation with `fe_sendauth: no password supplied`, so the two features could not be used together.
  
  `PgvectorClient` builds its own engine and never got the `do_connect` listener that refreshes the RDS IAM token, and the `ScopedSession` branch that would have reused the instrumented main engine is unreachable because `PGVECTOR_DB_URL` defaults to `DATABASE_URL` and is therefore never falsy.
  
  The pgvector engine now goes through `enable_iam_token_auth()` like the main and Alembic engines. Since a token authenticates exactly one host/port/user, that function now attaches the listener only to engines pointing at the same target, so a `PGVECTOR_DB_URL` aimed at a separate database keeps the password from its own URL instead of having it overwritten; the skip is logged with both identities.
  
  Fixes #27752

  Classic298 · 2026-08-24 07:35:19 -04:00
  baeb2dfb83
• perf: route native JSON columns through JSONCodec instead of stdlib json (#28396)

  JSONField serializes with JSONCodec, but columns declared as SQLAlchemy's own JSON
  type go through the engine's serializer instead, and no engine set one. That left
  Chat.chat - the largest blob the app stores - on stdlib json.dumps/loads no matter
  what ENABLE_ORJSON was set to, while the rest of the app used the codec. SQLAlchemy
  invokes it once per write and once per read, so every chat read and write paid a
  full stdlib pass over the whole conversation on top of whatever the caller did.
  
  Both engine constructors are now wrapped so the codec is wired in by default and
  cannot be missed by a call site that forgets it; an explicit json_serializer still
  wins. The 10 create_engine/create_async_engine calls in this module go through the
  wrappers. Vector-store engines (pgvector, mariadb, opengauss) are separate databases
  and are left alone.
  
  Serializing and deserializing chat-shaped blobs, median of 11 runs:
  
  | chat blob | write | read |
  | --- | --- | --- |
  | 600 msgs (2.8 MB) | 10.1 -> 1.7 ms | 8.4 -> 3.7 ms |
  | 3000 msgs (14.2 MB) | 51.9 -> 8.0 ms | 48.5 -> 27.8 ms |
  | 6000 msgs (28.5 MB) | 105.9 -> 29.5 ms | 112.7 -> 80.5 ms |
  
  With ENABLE_ORJSON off JSONCodec is stdlib json, so this is a no-op until the flag
  is set - the change cannot regress a default deployment.
  
  With it on, a round-trip probe through a native JSON column returns objects equal to
  the stdlib ones on all 12 shapes tried: ASCII, CJK, emoji, astral-plane, unicode
  keys, null bytes, lone surrogates, floats, ints above 2**63 and 2**64, line
  separators, empty and deeply nested. Stored text changes for non-ASCII, which is
  written as raw UTF-8 rather than backslash-uXXXX escapes and is correspondingly
  smaller. Nothing queries that text by escape except two Postgres safety filters in
  chats.py, and both still hold: a null byte is escaped identically by both codecs,
  and the title filter reads a text column rather than JSON. The ->> and json_extract
  searches decode the string before matching, so escaping cannot reach them.
  
  Two differences are inherent to JSONCodec and already apply to every JSONField
  column: ints beyond 2**64-1 come back as float, and NaN/Infinity serialize to null
  rather than the bare literals stdlib emits - the latter being invalid JSON that a
  Postgres json column rejects today. Neither shape occurs in chat blobs. Alembic
  builds its own engine and stays on stdlib, which is fine in both directions since
  each codec reads the other's output.
  
  
  Claude-Session: https://claude.ai/code/session_014BXoM6QiFJKisxcxKAXii8
  
  Co-authored-by: Claude <noreply@anthropic.com>

  Classic298 · 2026-08-10 19:39:49 -05:00
  8d6a7c8308
• refac
  Timothy Jaeryang Baek · 2026-07-31 17:41:14 -04:00
  bb0f898b43
• perf: cut per-request database session overhead (#27385)

  Two independent sources of fixed per-request cost:
  
  The async SQLite engine was created with pool_pre_ping=True. A pre-ping guards against server connections dropped by timeouts or restarts, which cannot happen to a local SQLite file; each ping still costs a hop into the aiosqlite worker thread plus a SELECT 1 on every connection checkout, and with session sharing off a single request checks out a connection for every model-layer call it makes. The Postgres engines keep their pre-ping, where it is actually protective.
  
  CommitSessionMiddleware unconditionally ran ScopedSession.commit() plus remove() after every HTTP request. The scoped registry instantiates a session on first access, so on the vast majority of requests (which never touch the sync session, per the middleware's own docstring) this built a Session, opened and committed an empty transaction and tore everything down for nothing. The middleware now checks ScopedSession.registry.has() first: requests that used the sync session are committed and removed exactly as before, on success and on the rollback path alike, and idle requests skip the machinery entirely.
  
  Benchmark (real SQLite database):
  
  | metric | before | after |
  | --- | --- | --- |
  | user row fetch incl. session + connection checkout | 681 us | 514 us |
  | idle-request sync session work (create + empty commit + teardown) | 12.3 us | 0.26 us |
  
  The first row saves per model-layer call, not per request: a request making five DB calls saves the checkout ping five times.
  
  Functionally verified: normal reads and writes work with pre-ping off; an idle request through the middleware leaves no sync session behind; a request that uses the sync session still gets committed and removed.

  Classic298 · 2026-07-23 17:49:48 -05:00
  d5f099a5d4
• refac
  Timothy Jaeryang Baek · 2026-06-29 12:40:20 -05:00
  c0c6c2181a
• chore: format
  Timothy Jaeryang Baek · 2026-06-01 13:56:55 -07:00
  6fce92aa12
• refac
  Timothy Jaeryang Baek · 2026-05-21 16:25:25 +04:00
  1acfbb6755
• refac
  Timothy Jaeryang Baek · 2026-05-21 15:29:49 +04:00
  d8b5b9fa79
• refac
  Timothy Jaeryang Baek · 2026-05-21 14:01:57 +04:00
  260ead64da
• refac: modernize type annotations (PEP 604 / PEP 585)
  Timothy Jaeryang Baek · 2026-05-12 17:10:15 +09:00
  6d0295588e
• refac: remove dead peewee connection wrappers
  Timothy Jaeryang Baek · 2026-05-12 04:14:19 +09:00
  c79ff81f55
• refac: deprecate peewee migration layer

  The Alembic init migration (7e5b5dc7342b) already creates the
  equivalent schema. Peewee migrations are no longer needed for
  any version >= 0.3.6.

  Timothy Jaeryang Baek · 2026-05-12 03:45:59 +09:00
  2c2d06c31b
• chore: format
  Timothy Jaeryang Baek · 2026-05-09 15:25:27 +09:00
  7bcc0e2e5c
• refac
  Timothy Jaeryang Baek · 2026-05-09 02:38:08 +09:00
  7eaecbad5a
• chore: format
  Timothy Jaeryang Baek · 2026-04-24 18:48:21 +09:00
  8ff7ff459b
• refac
  Timothy Jaeryang Baek · 2026-04-24 18:20:10 +09:00
  3d1e355df7
• refac
  Timothy Jaeryang Baek · 2026-04-24 16:17:46 +09:00
  678c44c7cd
• chore: format
  Timothy Jaeryang Baek · 2026-04-21 15:52:00 +09:00
  6cc799b1bb
• refac
  Timothy Jaeryang Baek · 2026-04-21 14:58:28 +09:00
  c4aac0415c
• refac
  Timothy Jaeryang Baek · 2026-04-20 18:47:53 +09:00
  b645b0dc23
• refac
  Timothy Jaeryang Baek · 2026-04-17 11:27:58 +09:00
  7cc7b367dc
• refac
  Timothy Jaeryang Baek · 2026-04-12 19:41:02 -05:00
  26b8ca5b5e
• chore: format
  Timothy Jaeryang Baek · 2026-04-12 18:12:59 -05:00
  25898116ea
• refac: async db
  Timothy Jaeryang Baek · 2026-04-12 14:22:11 -05:00
  27169124f2
• refac
  Timothy Jaeryang Baek · 2026-03-17 17:58:01 -05:00
  de3317e26b
• fix: use NullPool for SQLCipher engine to prevent segfault (#22273)

  The SQLCipher engine used a dummy sqlite:// URL with a creator function,
  which caused SQLAlchemy to auto-select SingletonThreadPool. This pool
  non-deterministically closes in-use connections when thread count exceeds
  pool_size (default 5), leading to use-after-free segfaults (exit code 139)
  in the native sqlcipher3 C library during multi-threaded operations like
  user signup.
  
  Now defaults to NullPool (each operation creates/closes its own connection)
  for maximum safety with the native C extension. Also respects the
  DATABASE_POOL_SIZE setting: if explicitly set >0, QueuePool is used with
  the configured pool parameters, matching the behavior of other DB paths.
  
  Fixes #22258

  Classic298 · 2026-03-06 14:04:10 -06:00
  04fae8b357
• refac/fix: DATABASE_ENABLE_SESSION_SHARING env var
  Timothy Jaeryang Baek · 2026-01-10 00:16:04 +04:00
  8646aebaab
• refac: get_db_context
  Timothy Jaeryang Baek · 2026-01-05 03:46:40 +04:00
  2bb13d5dbc
• refac/enh: db session sharing
  Timothy Jaeryang Baek · 2025-12-29 00:21:18 +04:00
  b1d0f00d8c
• refac/enh: db session sharing
  Timothy Jaeryang Baek · 2025-12-28 22:00:44 +04:00
  2041ab483e
• enh: ENABLE_DB_MIGRATIONS
  Timothy Jaeryang Baek · 2025-12-23 10:14:54 +04:00
  9824f0e333
• chore: format
  Timothy Jaeryang Baek · 2025-12-21 23:34:08 +04:00
  01e88c6ac2
• fix: Fix handling of absolute paths for SQLCipher databases (#20074)

  * sequential
  
  * zero default
  
  * fix
  
  * fix: preserve absolute paths in sqlite+sqlcipher URLs
  
  Previously, the connection logic incorrectly stripped the leading slash
  from `sqlite+sqlcipher` paths, forcibly converting absolute paths
  (e.g., `sqlite+sqlcipher:////app/data.db`) into relative paths
  (which became `app/data.db`). This caused database initialization failures
  when using absolute paths, such as with Docker volume mounts.
  This change removes the slash-stripping logic, ensuring that absolute
  path conventions (starting with `/`) are respected while maintaining
  support for relative paths (which do not start with `/`).

  Classic298 · 2025-12-21 09:18:20 -05:00
  b3904b6ecb
• chore/perf: Remove old SRC level log env vars with no impact (#20045)

  * Update openai.py
  
  * Update env.py
  
  * Merge pull request open-webui#19030 from open-webui/dev (#119)
  
  Co-authored-by: Tim Baek <tim@openwebui.com>
  Co-authored-by: Claude <noreply@anthropic.com>
  
  ---------
  
  Co-authored-by: Tim Baek <tim@openwebui.com>
  Co-authored-by: Claude <noreply@anthropic.com>

  Classic298 · 2025-12-20 08:16:14 -05:00
  823b9a6dd9
• refactor: format

  Signed-off-by: Adam Tao <tcx4c70@gmail.com>

  Adam Tao · 2025-08-10 22:28:31 +08:00
  7bd7559bfe
• feat(db): Add DATABASE_ENABLE_SQLITE_WAL to enable SQLite WAL

  Signed-off-by: Adam Tao <tcx4c70@gmail.com>

  Adam Tao · 2025-08-10 22:28:31 +08:00
  b23abcbfe5
• chore: format
  Timothy Jaeryang Baek · 2025-08-09 23:57:35 +04:00
  77189664c2
• feat: Implement SQLCipher support for database encryption

  - Added sqlcipher3 dependency to requirements.txt for SQLCipher integration.
  - Modified database connection handling in wrappers.py to support encrypted SQLite databases using the new sqlite+sqlcipher:// URL protocol.
  - Updated db.py to handle SQLCipher URLs for SQLAlchemy connections.
  - Enhanced Alembic migration environment to support SQLCipher URLs.

  Adam M. Smith · 2025-07-31 23:21:35 +00:00
  c9a4bc18f4
• refac
  Timothy Jaeryang Baek · 2025-06-21 19:12:43 +04:00
  aef0ad2d10
• Change default db pooling
  René Pfitzner · 2025-06-21 14:01:57 +01:00
  63d99abf41
• chore: format
  Timothy Jaeryang Baek · 2025-06-20 20:32:23 +04:00
  17b9a81504
• Provide more detailed information to the user upon database connection failure

  modified:   backend/open_webui/internal/db.py

  Taehong Gu · 2025-06-20 14:13:43 +09:00
  c7ef6025af
• Merge branch 'open-webui:main' into main
  MadsLang · 2025-01-13 08:28:13 +01:00
  d4a26f8031