Commit Graph

448 Commits (perf/postgres-queue)

Author SHA1 Message Date
Thomas Way 8c0c8a8d0e
feat: Use postgres as a queue
We've been keen to try this for a while as it means we can remove redis as a
dependency, which makes Immich easier to setup and run.

This replaces bullmq with a bespoke postgres queue. Jobs in the queue are
processed either immediately via triggers and notifications, or eventually if a
notification is missed.
2025-04-30 20:43:52 +01:00
Jason Rasmussen 526c02297c
refactor: stream queue migration (#17997) 2025-04-30 16:23:13 +00:00
Thomas 3ce353393a
chore(server): don't insert embeddings if the model has changed (#17885)
* chore(server): don't insert embeddings if the model has changed

We're moving away from the heuristic of waiting for queues to complete. The job
which inserts embeddings can simply check if the model has changed before
inserting, rather than attempting to lock the queue.

* more robust dim size update

* use check constraint

* index command cleanup

* add create statement

* update medium test, create appropriate extension

* new line

* set dimension size when running on all assets

* why does it want braces smh

* take 2

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2025-04-29 14:23:01 -04:00
Eli Gao 48bcbee6ed
feat(server): JXL previews from DNG 1.7+ (#17861)
* feat(server): JXL previews from RAW

* refactor(server): use var name assumedExtractedFormat for clarity

* test(server): fix existing media.extract() returning JPEG

* chore(openapi): regen

* style(server): lint

* fix(server): ignore undefined decode orientation

* fix(server): correct orientation assignment in media decode options

* test(server): unit tests of JXL-encoded DNG

* refactor(server): return buffer and format from mediaRepository.extract()

* chore(open-api): regen

* refactor

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2025-04-28 18:18:46 -04:00
Jason Rasmussen 1b5fc9c665
feat: notifications (#17701)
* feat: notifications

* UI works

* chore: pr feedback

* initial fetch and clear notification upon logging out

* fix: merge

---------

Co-authored-by: Alex Tran <alex.tran1502@gmail.com>
2025-04-28 10:36:14 -04:00
Jason Rasmussen ad272333db
refactor: user avatar color (#17753) 2025-04-28 08:54:51 -05:00
Jason Rasmussen 1d610ad9cb
refactor: database connection parsing (#17852) 2025-04-24 12:58:29 -04:00
Jason Rasmussen 1b5e981a45
fix: failing ci checks (#17810) 2025-04-23 10:59:54 -04:00
Jason Rasmussen 56a4aa9ffe
refactor: email repository (#17746) 2025-04-21 12:53:37 -04:00
Daniel Dietzler 52ae06c119
refactor: remove album entity, update types (#17450) 2025-04-18 23:10:34 +02:00
Jason Rasmussen 6474a78b8b
feat: initial kysely migration file (#17678) 2025-04-17 17:38:47 -04:00
Jason Rasmussen e275f2d8b3
feat: add foreign key indexes (#17672) 2025-04-17 14:41:06 -04:00
Daniel Dietzler 8b38f8a58d
fix: do not select album in time bucket query (#17662) 2025-04-16 17:52:22 -04:00
Daniel Dietzler 586a7a173b
refactor: handle detect faces job query (#17660) 2025-04-16 22:52:54 +02:00
Jason Rasmussen 8cefa0b84b
refactor: migrate some e2e to medium (#17640) 2025-04-16 14:59:08 -04:00
Daniel Dietzler f50e5d006c
refactor: dedicated queries for asset jobs (#17652) 2025-04-16 14:08:49 -04:00
Jason Rasmussen 8da5f21fcf
refactor: medium tests (#17634) 2025-04-15 15:54:23 -04:00
Jason Rasmussen b710ad36f3
feat: upgrade kysely (#17630)
* feat: upgrade kysely

* chore: pr feedback
2025-04-15 13:26:56 -04:00
Jason Rasmussen 17e720440d
refactor: new asset-job repository (#17622)
* refactor: new asset-job repository

* fix: broken medium tests on main
2025-04-15 10:24:51 -04:00
Jason Rasmussen f189c7b101
refactor: medium tests (#17599) 2025-04-15 08:53:14 -04:00
Mert c3d10c5be2
refactor(server): non-nullable file metadata (#17598) 2025-04-15 12:03:31 +01:00
Jason Rasmussen b2753103c6
chore: remove unused logger (#17593) 2025-04-14 15:01:49 -04:00
Daniel Dietzler a373034629
refactor: migrate stacks (#17559)
chore: migrate stacks
2025-04-12 08:33:35 -04:00
Mert 25f2b9602f
refactor(server): remove face, person and face search entities (#17535)
* remove face, person and face search entities

update tests and mappers

check if face relation exists

update sql

unused imports

* pr feedback

generate sql, remove unused imports
2025-04-11 14:44:45 -04:00
Jason Rasmussen 584e5894bf
refactor: user factories instead of stubs (#17540) 2025-04-11 11:53:37 -04:00
Jason Rasmussen 75c83cb704
refactor: metadata stub (#17532) 2025-04-10 21:58:55 +02:00
Jason Rasmussen eaa0e07329
refactor: asset files entity (#17527) 2025-04-10 13:26:27 -04:00
Daniel Dietzler 7a1e8ce6d8
chore: remove exif entity (#17499) 2025-04-10 12:36:29 -04:00
Jason Rasmussen 8aea07b750
refactor: album user entity (#17524) 2025-04-10 11:53:21 -04:00
Zack Pollard 94dba29298
refactor: remove user entity (#17498) 2025-04-10 10:53:21 -04:00
Jason Rasmussen 206545356d
refactor: metadata entity (#17492) 2025-04-09 11:45:30 -04:00
Jason Rasmussen 8943ec23ba
refactor: more database types (#17490) 2025-04-09 10:24:38 -04:00
Jason Rasmussen cf2c0260a6
refactor: activity item (#17470)
* refactor: activity item

* fix query

* qualified columns

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2025-04-09 08:35:20 -04:00
Jason Rasmussen 4794eeca88
refactor: database types (#17468) 2025-04-08 12:40:03 -04:00
Alex e5ca79dd44
refactor: remove session entity (#17466)
* refactor: remove session entity

* fix: test

* update sql

* remote export
2025-04-08 16:04:07 +00:00
Jason Rasmussen b6c5a03533
refactor: remove tag entity (#17462) 2025-04-08 10:52:54 -04:00
Jason Rasmussen fdbe6d649f
refactor: remove smart search entity (#17447)
refactor: smart search entity
2025-04-08 09:56:45 -04:00
Jason Rasmussen e7a5b96ed0
feat: extension, triggers, functions, comments, parameters management in sql-tools (#17269)
feat: sql-tools extension, triggers, functions, comments, parameters
2025-04-07 15:12:12 -04:00
Abhinav Valecha b621281351
feat(server): Avoid face match with people born after file creation #4743 (#16918)
* feat(server): Avoid face matching with people born after file creation date (#4743)

* lint

* add medium tests for facial recognition

---------

Co-authored-by: Alex <alex.tran1502@gmail.com>
2025-04-02 10:37:26 -05:00
Mert d911b76c08
fix(server): use stat instead of exifinfo for file date metadata (#17311)
* use stat instead of filecreatedate

* update tests

* unused import
2025-04-01 17:24:07 -05:00
Mert 59e5c82569
fix(server): full-size images not migrated or deleted correctly (#17308)
* fix file path logic

* update tests

* add empty array fallback just in case for now
2025-04-01 18:11:46 -04:00
Eli Gao 5c80e8734b
feat: original-sized previews for non-web-friendly images (#14446)
* feat(server): extract full-size previews from RAW images

* feat(web): load fullsize preview for RAW images when zoomed in

* refactor: tweaks for code review

* refactor: rename "converted" preview/assets to "fullsize"

* feat(web/server): fullsize preview for non-web-friendly images

* feat: tweaks for code review

* feat(server): require ASSET_DOWNLOAD premission for fullsize previews

* test: fix types and interfaces

* chore: gen open-api

* feat(server): keep only essential exif in fullsize preview

* chore: regen openapi

* test: revert unnecessary timeout

* feat: move full-size preview config to standalone entry

* feat(i18n): update en texts

* fix: don't return fullsizePath when disabled

* test: full-size previews

* test(web): full-size previews

* chore: make open-api

* feat(server): redirect to preview/original URL when fullsize thumbnail not available

* fix(server): delete fullsize preview image on thumbnail regen after fullsize preview turned off

* refactor(server): AssetRepository.deleteFiles with Kysely

* fix(server): type of MediaRepository.writeExif

* minor simplification

* minor styling changes and condensed wording

* simplify

* chore: reuild open-api

* test(server): fix media.service tests

* test(web): fix photo-viewer test

* fix(server):  use fullsize image when requested

* fix file path extension

* formatting

* use fullsize when zooming back out or when "display original photos" is enabled

* simplify condition

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>
2025-03-31 13:24:28 -04:00
Zack Pollard d613f15606
test: fix flaky user handle delete check medium test (#17253)
we can't run specifically the handleUserDeleteCheck tests concurrently due to one of the tests modifying the config in the shared database
if run concurrently you can get race conditions where the other tests pick up the change, even with resetting the config in the beforeEach
therefore the test that checks a delete actually happens, fails
there are many ways to solve this, disabling concurrency for the suite, forcing sequential tests for just handleUserDeleteCheck, increasing the delete test deletedAt to more than the custom duration tests deleteDelay
I applied all three of these. You could also force all the user tests to run in their own databases, but that feels overkill
2025-03-31 13:19:57 +01:00
PathToLife 09f4476f97
feat: improve performance for GET /api/album & /api/album/:id (#17124)
* fix(server) optimize number of sql calls for GET /api/albums

remove unnecessary join for getMetadataForIds
remove separate call to getLastUpdatedAssetForAlbumId

* fix(server) remove unnecessary getLastUpdatedAssetForAlbumId call for GET /api/album/:id

also remove getLastUpdatedAssetForAlbumId query as it is no longer referenced

* fix(server): correct lastModifiedAssetTimestamp return type + formatting and typing

* chore(server): address type issue with tests found via npm:check

tests & lint still pass before this commit.
2025-03-31 11:28:41 +00:00
Daniel Dietzler 238c151ac3
chore: finish migrating eslint config files; bump unicorn (#17200) 2025-03-31 12:18:25 +01:00
Jason Rasmussen 55a3c30664
feat: kysely migrations (#17198) 2025-03-29 09:26:24 -04:00
Jason Rasmussen 4b4bcd23f4
feat: schema diff sql tools (#17116) 2025-03-28 10:40:09 -04:00
Mert 75df8fc10e
refactor(server): bulk update exif (#17109)
* bulk update exif

* update sql

* update tests

* check job queeuing in test
2025-03-25 21:24:24 +00:00
Mert 9f46ba8eb4
fix(server): set pixel format when scaling and not tonemapping (#16932)
set pixel format when scaling and not tonemapping
2025-03-18 11:42:09 -05:00
Jason Rasmussen 6a40aa83b7
refactor: better types for getList and getDeletedAfter (#16926) 2025-03-17 15:32:12 -04:00