[Feature]: Add the ability to dry run health checks #390
Description
Prerequisites
- I searched existing issues
Feature Summary
We want the ability to run new health checks in dry mode to allow operators to be able to evaluate the health checks in their environment without impacting their cluster. Currently the only way to dry run a health check is to disable it from the FQ rules, but this will still update the node conditions / events which can result in confusion for the end users, and we'd like to avoid that.
Problem/Use Case
Ability to run health checks in dry run mode so that will not write/update any resource on the cluster.
Proposed Solution
Introduce a dry run / audit flag in the health event. If this flag is set, then the health event should only be persisted to the data store and exported as metrics. All modules should be default drop all health events that have the dry run / audit flag set irrespective of their configuration.
Component
Health Monitor