Extract Metadata has inconsistent behaviour

[dangarmol]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

I have started a fresh Immich instance, mounting my photo library via NFS (read-only), database on local SSD and /data on local HDD and most jobs seem to be running fine, except for the "Extract Metadata" job. I read that there was an issue with high concurrency, so I limited it to only 2.

On first container boot, the job seemed stuck after a while and the assets remaining didn't really go down, but I let it run for about a day before restarting the container.
The second time I started the container, the number of assets that needed metadata extraction started growing from 0 up to ~650k, although my library only has roughly ~460k assets, which Immich correctly reports on the External Libraries page. After a while, the number stopped growing but it never started going down. I let it run for several hours, but the number didn't move. I clicked the button to Clear the job, and Waiting has gone down to 0, but Active is stuck at 2, so I can't start a new job unless I restart the container.

I haven't restarted the containers in case it can be helpful for troubleshooting.

The OS that Immich Server is running on

Ubuntu 24.04.2 inside Proxmox VM

Version of Immich Server

v2.3.1

Version of Immich Mobile App

Irrelevant

Platform with the issue

• Server
• Web
• Mobile

Device make and model

VM with 24 cores and 80GB RAM

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
      - syno-nfs-homes:/syno-homes:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://docs.immich.app/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8@sha256:81db6d39e1bba3b3ff32bd3a1b19a6d69690f94a3954ec131277b9a26b95b3aa
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always

volumes:
  model-cache:
  syno-nfs-homes:
    driver: local
    driver_opts:
      type: nfs
      o: addr=<redacted>,nolock
      device: :<redacted>

Your .env content

UPLOAD_LOCATION=/storage/immich
DB_DATA_LOCATION=./postgres

TZ=<redacted>
IMMICH_VERSION=v2

DB_PASSWORD=<redacted>

DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Fresh Immich install
2. Add read-only external library (NFS) with ~460k assets
3. Make all jobs run for the library assets to be added to the timeline
4. Extract Metadata behaves erratically, reports too many assets after container restart, gets stuck and is unable to clear running jobs

Relevant log output

I searched the log for errors but none are related to the Extract Metadata job. I'll be happy to report back if you give me something to search for in the logs or elsewhere. Thanks!

Additional information

No response

[bo0tzz]

Does your UPLOAD_LOCATION have any overlap at all with the external library?

[dangarmol]

No, they're even on separate devices. /data is on local HDD and external library is mounted via NFS from another machine. I also have no uploads via the Immich app, only use the read-only external library.

[alextran1502]

Can you go to Administration > Settings > Logs and increase the logs level to debug and see if there is any other information from the server logs?

[dangarmol]

I've enabled it and don't see anything useful there. Mostly ffmpeg debug logs from the video transcoding job, but nothing related to this issue. Can you give me a keyword to search for in the logs?

Also interestingly, I don't see any debug logs there for Generate Thumbnails despite the job definitely consuming images, as the number is going down. Is this expected?

[dangarmol]

Restarted the container about 4 hours ago and all jobs have been running for a while, but the debug logs only show entries for the Metadata Sidecar like:

DEBUG [Microservices:MetadataService] Sidecar check found old=null, new=null will do nothing for asset 1d138891-1400-4d13-a286-f86eaf166610: path/to/file.jpg

There is no mention of the Extract Metadata job in the logs, and the number of Waiting items keeps steadily going up (around 350k now), whilst all other jobs are going down.

[dangarmol]

Btw, the issue I referenced above and why I limited the concurrency of this job to 2: #23890

Could my issue have something to do with that other one?

[skatsubo]

Reproduction steps
Add read-only external library (NFS) with ~460k assets
Make all jobs run for the library assets to be added to the timeline

Could you clarify the last line "Make all jobs run"? Was it a simple "Scan" from the External Libraries page or something else?

Hypothetical scenario:
I trigger Scan of external library. It works in batches. It adds newly discovered disk assets to the database. Then it adds a batch of jobs for metadata extraction for these assets. And so on.
If at this time I also start Metadata Extraction manually then it will query the database for all new (already discovered but not yet processed) assets. Next, it will create metadata extraction jobs for these assets.
So some assets will have 2 jobs in the metadata extraction queue. Hence the number of waiting jobs can be greater than the number of assets.
(This is based on a quick glance at library.service.ts, in particular handleSyncFiles(). I did not test this scenario on real data.)

Btw, the issue I referenced above and why I limited the concurrency of this job to 2: #23890
Could my issue have something to do with that other one?

It might be related if you ever seen "BatchCluster has ended" in logs. Typically if it happens then there will be a lot of such errors.
On the other hand, "BatchCluster has ended" does not block execution of jobs. It makes them fail fast. So the number of waiting jobs would be going down quickly if I'm not mistaken.

I don't see any debug logs there for Generate Thumbnails despite the job definitely consuming images, as the number is going down. Is this expected?

Right, this is expected.
If there are thumbnail generation errors they will be printed in logs as Errors or Warnings.
If everything goes smooth then nothing in logs at any level. In the next Immich release it will be printing each thumbnail generation attempt at Verbose level.

There is no mention of the Extract Metadata job in the logs

If there are metadata extraction errors they will be printed in logs as Errors or Warnings.
If everything is super smooth then only Verbose logging level will show activity; it will be really verbose as all extracted exif tags are printed at Verbose level.
If metadata extraction jobs/queue are stuck then there will be nothing at any logging level, huh.

---

Some more troubleshooting steps.

1. Jobs/queues JSON
   In your browser, while being logged in as an admin, go to https://my.immich.app/api/jobs. Post its JSON output. You can also monitor its values over time.

2. Redis info

docker exec immich_redis redis-cli INFO KEYSPACE
docker exec immich_redis ls -l /data

3. Optional: backup Redis for possible future investigation (there is only transient job/queue state in Redis, nothing critical).

docker exec immich_redis redis-cli save
docker cp immich_redis:/data/dump.rdb redis-dump.rdb

4. Purge state of all jobs/queue related to metadata extraction from Redis:

docker exec immich_redis sh -c 'redis-cli --scan --pattern "immich_bull:metadataExtraction:*" | xargs redis-cli DEL'

See if it improves things upon re-running metadata extraction.

5. Optional: nuke Redis. This removes all jobs/queue data.

docker exec immich_redis redis-cli FLUSHALL
# or
# docker compose down redis && docker compose up -d redis

Then trigger Scan of your external library.

---

If you are curious to see which files are in the queue of different jobs

Immich uses BullMQ to manage queues/jobs. BullMQ persists data in Redis.

1. If you prefer UI you can add a bull-board container to the compose stack, smth like

  bullboard:
    image: venatum/bull-board:latest
    environment:
      REDIS_HOST: redis
      BULL_PREFIX: immich_bull
    ports:
      - "3000:3000"

2. For CLI I have a little bit intimidating script to export Redis content to a text file:
   (You can replace --pattern '*' with --pattern 'immich_bull:metadataExtraction:*' to limit it to a particular queue)

docker exec -i immich_redis bash << 'EOF' > redis.txt
two_lines_to_kv='{ sub(/^[0-9]+\) /, "") ; if (NR % 2) { key = $0 } else { print key ": " $0 } }'
xrange_to_csv='/^[0-9]+-[0-9]+$/ { if (NR > 1) print output ; output = "id=" $0 ; next } { if (!key) { key = $0 } else { output = output ", " key "=" $0 ; key = "" } } END { print output }'
redis-cli --scan --pattern '*' | sort | while read -r key; do
  type=$(redis-cli type "$key")
  echo -e "\n==== $key ($type) ===="  
  case $type in
    "string") redis-cli get "$key" ;;
    "list")   redis-cli lrange "$key" 0 -1 ;;
    "set")    redis-cli smembers "$key" ;;
    "zset")   redis-cli zrange "$key" 0 -1 withscores | awk "$two_lines_to_kv" ;;
    "hash")   redis-cli hgetall "$key" | awk "$two_lines_to_kv" ;;
    "stream") redis-cli xrange "$key" - + | awk "$xrange_to_csv" ;;
    *)        echo "Unsupported: $type" ;;
  esac
done
EOF

Although with thousands of jobs it can take long and its output will be large.

[dangarmol]

Hi @skatsubo, kudos for your detailed explanation, really appreciate it!

What I meant by making all jobs run was that I clicked on all jobs on the "Missing" or equivalent buttons, so that assets with missing info would start processing. I was not aware that the right way of doing it was just to rescan the external library and let the jobs be created automatically. Perhaps this is something that you could somehow improve from the UX perspective, so that the same asset cannot be queued twice, since it was natural (at least for me) to make all the jobs run manually and not expect them to conflict with each other.

I guess this explains the job count being higher than the asset count in the whole library. I've now flushed Redis by restarting the containers and I'm letting the jobs run by rescanning the library as you suggested. I'll report back if I see the issue of the job being stuck again.

I saw the "BatchCluster has ended" log originally, but not anymore with concurrency set to 2.

As a side note, I'm now wondering what my timeline will look like given the order of jobs described in the docs: https://docs.immich.app/administration/jobs-workers/#job-processing-order
Most of my assets have already completed the Thumbnail Generation, Video Transcoding and other tasks before going through the Metadata Extraction because it was stuck and I triggered all other jobs manually. So now I guess I'll have a somewhat broken library because I have large numbers of Live Photos that are saved as HEIC+MOV files and these have been processed independently, or perhaps I'll have a bunch of useless assets in my /data mount? As an example, a MOV file that was part of a Live Photo probably shouldn't have had a thumbnail generated, since it will never be displayed.