Implement error handling in ExecutionUpsertHandler to avoid unnecessary retries

[coderabbitai]

Background

Currently, when the Bidder encounters errors while updating execution state, these errors are propagated to the ExecutionUpsertHandler. While the event watcher has a retry mechanism (3 retries before skipping), we need more sophisticated error handling to:

• Avoid unnecessary retries for unrecoverable errors
• Log appropriate error messages
• Publish relevant metrics for monitoring

Links

• Related Issue: Implement error handling when updating execution state in Bidder #4684
• Discussion: async execution handling #4683 (comment)
• Code Reference:

  bacalhau/pkg/compute/watchers/executor_watcher.go

  Line 24 in f9fdcd5

  func (h *ExecutionUpsertHandler) HandleEvent(ctx context.Context, event watcher.Event) error {

Requirements

1. Implement error handling logic in ExecutionUpsertHandler
2. Add appropriate logging for different error scenarios
3. Add metrics to track error occurrences and types
4. Document the error handling strategy

Requested by @wdbaruni

[linear]

ENG-618 Implement error handling in ExecutionUpsertHandler to avoid unnecessary retries