Task delete_missing_photos getting an error soon after starting

[dotanm]

It was already discussed in #672 , but it was closed.

🐛 Bug Report

• [V] 📁 I've Included a ZIP file containing my librephotos log files
• [V] ❌ I have looked for similar issues (including closed ones)

📝 Description of issue:

I'm starting to delete missing photos, and the interface is stuck on "waiting" and "running", while in the ownphotos.log file there's an error:

2024-09-20 07:50:34,127 : views.py : _delete_missing_photos : 324 : ERROR : An Error occurred
Traceback (most recent call last):
  File "/code/api/views/views.py", line 321, in _delete_missing_photos
    delete_missing_photos(request.user, job_id)
  File "/code/api/autoalbum.py", line 213, in delete_missing_photos
    for missing_photo in missing_photos:
  File "/usr/local/lib/python3.12/dist-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.12/dist-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.12/dist-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/psycopg/cursor.py", line 93, in execute
    self._conn.wait(
  File "/usr/local/lib/python3.12/dist-packages/psycopg/connection.py", line 394, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/psycopg/waiting.py", line 346, in wait_poll
    fileevs = poll.poll(interval)
              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/gevent/select.py", line 339, in poll
    result.event.wait(timeout=timeout)
  File "src/gevent/event.py", line 163, in gevent._gevent_cevent.Event.wait
  File "src/gevent/_abstract_linkable.py", line 521, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait
  File "src/gevent/_abstract_linkable.py", line 487, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 490, in gevent._gevent_c_abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 442, in gevent._gevent_c_abstract_linkable.AbstractLinkable._AbstractLinkable__wait_to_be_notified
  File "src/gevent/_abstract_linkable.py", line 451, in gevent._gevent_c_abstract_linkable.AbstractLinkable._switch_to_hub
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
greenlet.GreenletExit

In my case there are over 400,000 missing photos.

Yes, there were a lot of duplicates, moves and deletes going around, but I really can't start the process all over again. There are too many items that are already defined.

🔁 How can we reproduce it:

Try deleting a relatively high number of missing photos. Probably a few dozens of thousands.

Please provide additional information:

• 💻 Operating system: Ubuntu 22.04
• ⚙ Architecture (x86 or ARM): x86
• 🔢 Librephotos version: 2024w36
• 📸 Librephotos installation method: Docker
  • 🐋 If Docker or Kubernets, provide docker-compose image tag:
• 📁 How is you picture library mounted: Local file system NTFS
  librephotos-logs-2024-09-20.zip

[dotanm]

Is there perhaps any manual way of running this process? With queries, maybe?

If possible, then users will at least be able to run queries with low number of records each time, until all the database is clean.

[gjgress]

Also having the same issue. Dev branch of the container doesn't help either. Workarounds would be greatly appreciated