-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Open
Description
Hi there,
Let's try this simple example:
from luigi import Task, LocalTarget, build
class A(Task):
def output(self):
return LocalTarget('A.txt')
def run(self):
print('++++++++++ Running A.run()')
# That's all, don't create anything!
class B(Task):
def requires(self):
return A()
def output(self):
return LocalTarget('B.txt')
def run(self):
print(f'++++++++++ Running B.run() (despite that A\'s completeness is '
f'{self.requires().complete()})')
with self.output().open('w'):
pass
build([B()], log_level='ERROR')
build() produces the following error:
++++++++++ Running A.run()
ERROR: [pid 19152] Worker Worker(salt=2366140394, workers=6, ...) failed B()
Traceback (most recent call last):
File ".../.pyenv/versions/3.11.12/lib/python3.11/site-packages/luigi/worker.py", line 195, in run
raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
RuntimeError: Unfulfilled dependency at run time: A__99914b932b (A.txt)
Indeed, that's correct since A.run() doesn't create any target
However, if I add in the /etc/luigi/luigi.cfg
[worker]
cache_task_completion = true
then suprisingly no errors occured:
++++++++++ Running A.run()
++++++++++ Running B.run() (despite that A's completeness is False)
Not sure if this is the correct behavior, according to documentation:
cache_task_completion
[...] When set to true, completion checks are cached so that tasks declared as complete once are not checked again. Defaults to false.
In the snippet above A().complete() evaluates to False, so I guess A() couldn't be marked as completed
Could you help me to solve this problem? Thank you!
--
PS checked for python 3.9.21, 3.11.12, 3.13.4
Metadata
Metadata
Assignees
Labels
No labels