Skip to content

1. Replace Sequential Loop with ThreadPoolExecutor #176

@osundwajeff

Description

@osundwajeff

File: geest/core/tasks/study_area_processing_task.py
Method: create_and_save_grid()

Steps:

  • Import ThreadPoolExecutor, as_completed from concurrent.futures
  • Import threading for lock
  • Get worker count from settings: grid_creation_workers (default 4)
  • Create thread-safe lock: threading.Lock()
  • Collect all chunks to process (filter out type="undefined")
  • Create ThreadPoolExecutor with max_workers
  • Submit all chunk tasks as futures
  • Store futures in dict mapping to task and chunk index
  • Iterate completed futures with as_completed()
  • For each completion:
    • Get result (raises exception if failed)
    • Acquire lock and write to GeoPackage
    • Update progress counter (thread-safe)
    • Handle exceptions per chunk (log, don't fail entire process)

Error Handling:

  • Catch individual chunk failures
  • Log warning with chunk index
  • Increment error count
  • Continue processing remaining chunks

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions