Skip to content

'cache_task_completion = true' causes luigi to ignore tasks' completion #3364

@telzhov

Description

@telzhov

Hi there,

Let's try this simple example:

from luigi import Task, LocalTarget, build

class A(Task):
    def output(self):
        return LocalTarget('A.txt')

    def run(self):
        print('++++++++++ Running A.run()')
        # That's all, don't create anything!

class B(Task):
    def requires(self):
        return A()

    def output(self):
        return LocalTarget('B.txt')

    def run(self):
        print(f'++++++++++ Running B.run() (despite that A\'s completeness is '
              f'{self.requires().complete()})')

        with self.output().open('w'):
            pass

build([B()], log_level='ERROR')

build() produces the following error:

++++++++++ Running A.run()

ERROR: [pid 19152] Worker Worker(salt=2366140394, workers=6, ...) failed    B()
Traceback (most recent call last):
  File ".../.pyenv/versions/3.11.12/lib/python3.11/site-packages/luigi/worker.py", line 195, in run
    raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: A__99914b932b (A.txt)

Indeed, that's correct since A.run() doesn't create any target

However, if I add in the /etc/luigi/luigi.cfg

[worker]
cache_task_completion = true

then suprisingly no errors occured:

++++++++++ Running A.run()
++++++++++ Running B.run() (despite that A's completeness is False)

Not sure if this is the correct behavior, according to documentation:

cache_task_completion
[...] When set to true, completion checks are cached so that tasks declared as complete once are not checked again. Defaults to false.

In the snippet above A().complete() evaluates to False, so I guess A() couldn't be marked as completed

Could you help me to solve this problem? Thank you!

--
PS checked for python 3.9.21, 3.11.12, 3.13.4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions