[ISSUE-5705] Support for degraded health #5741

seonWKim · 2024-06-08T07:38:36Z

Motivation:

Add support for degraded health.

Modifications:

Add HealthStatus enum which can represent the health status of a server
Add healthStatus() method in HealthChecker for fine-grained representation of server's status
Add degradedResponse which shows that the server's status is being degraded.

I think adding new health check service which returns HealthStatus(instead of boolean) might be an alternate option. But because this PR is before any reviews, I've just implemented feature on top of HealthCheckService.

Result:

Users can now receive notification for degraded server status like below

{"healthy":true, "degraded":true}

Closes Support for degraded health #5705. (If this resolves the issue.)
Users can now identify degraded server status in addtion to healthy and unhealthy.

github-actions · 2024-06-08T07:52:52Z

🔍 Build Scan® (commit: `6ca0b1d`)

Job name	Status	Build Scan®
build-ubicloud-standard-16-jdk-8	✅	https://ge.armeria.dev/s/lydpldl4eh3km
build-ubicloud-standard-16-jdk-21-snapshot-blockhound	✅	https://ge.armeria.dev/s/3rxzpens63yjg
build-ubicloud-standard-16-jdk-17-min-java-17-coverage	❌ (failure)	https://ge.armeria.dev/s/4dj24f2uh6p6i
build-ubicloud-standard-16-jdk-17-min-java-11	✅	https://ge.armeria.dev/s/m6xuygdb2mwz6
build-ubicloud-standard-16-jdk-17-leak	✅	https://ge.armeria.dev/s/rcoxjfsqsvvrc
build-ubicloud-standard-16-jdk-11	✅	https://ge.armeria.dev/s/nbsohiudkzl3y
build-macos-latest-jdk-21	✅	https://ge.armeria.dev/s/srdl2sqsf4a2y

trustin

Thanks, @seonWKim! 🙇

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/SettableHealthStatusChecker.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

jrhee17

Looks good overall 👍 I think some client-side changes will be needed, but I prefer I do this to ensure compatibility with xDS-side requirements. Left some minor comments 🙇

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

trustin

Almost there!

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/DefaultHealthCheckUpdateHandler.java

…bleHealthChecker`

…dates

Co-authored-by: jrhee17 <[email protected]>

…cify HealthStatus

- Add javadoc on `HealthStatus` - Checks for IF-NONE-MATCH header for specific `HealthStatus` - Add `HealthStatusUpdateHandler`

jrhee17

Left some more nit comments 👍

core/src/main/java/com/linecorp/armeria/server/healthcheck/DefaultHealthCheckUpdateHandler.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckUpdateListener.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthChecker.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/SettableHealthChecker.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/ScheduledHealthChecker.java

- Add @UnstableApi

jrhee17

👍 👍 👍

minwoox · 2025-02-12T03:36:23Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckUpdateResult.java

+    /**
+     * Tells {@link HealthCheckService} to mark the {@link Server} as 'under maintenance'.
+     */
+    UNDER_MAINTENANCE,


Can we just use MAINTENANCE?

…le-health-checker

minwoox

Hi, @seonWKim!
Sorry about the late review.
I've pushed some fixes and left some comments to discuss. 😉

minwoox · 2025-02-12T06:24:52Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

+    /**
+     * The {@link Server} is under maintenance and unable to serve requests.
+     */
+    UNDER_MAINTENANCE(100, false);


Better to put UNDER_MAINTENANCE after STOPPING? We also need to update the Javadoc of priority to explain the value means.

minwoox · 2025-02-12T06:29:20Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

+                    this.pendingResponses.stream()
+                                         .filter(res -> {
+                                             if (res.pollHealthStatus) {
+                                                 return res.interestedStatus != healthStatus;


Shouldn't it filter when

the healthStatus is changed from its previous healthStatus

res.interestedStatus == healthStatus;

seonWKim · 2025-02-25T12:43:34Z

Hi, @minwoox :)

Thanks for the review. I'll take a look

seonWKim requested review from ikhoon, jrhee17, minwoox and trustin as code owners June 8, 2024 07:38

seonWKim mentioned this pull request Jun 8, 2024

Support for degraded health #5705

Open

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from f20ed57 to 665ec13 Compare June 8, 2024 08:20

trustin reviewed Jun 8, 2024

View reviewed changes

trustin added the new feature label Jun 8, 2024

seonWKim commented Jun 9, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java Outdated Show resolved Hide resolved

seonWKim commented Jun 9, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java Show resolved Hide resolved

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from 52fdd7b to aed0a93 Compare June 14, 2024 01:24

jrhee17 reviewed Jun 14, 2024

View reviewed changes

trustin reviewed Jun 19, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java Show resolved Hide resolved

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java Outdated Show resolved Hide resolved

seonWKim commented Jun 22, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/DefaultHealthCheckUpdateHandler.java Outdated Show resolved Hide resolved

seonWKim and others added 15 commits June 22, 2024 11:32

Add support for degradable health status in HttpCheckService

d821502

Fix CpuHealthChecker and add test

866d72f

Remove SettableHealthStatusChecker and merge the features to `Setta…

f7db1d1

…bleHealthChecker`

Add healthStatusUpdated for more fine fine-grained health status up…

40fad94

…dates

Add UNDER_MAINTENANCE

a677109

Fix getResponse

4ac6a6f

Fix lint

eed2b0a

Fix test name

59184fc

Fix docs

8b5d8f9

Add (server) STOPPING status

7a4b880

Fix test

3af30d2

Apply suggestions from code review

4d780db

Co-authored-by: jrhee17 <[email protected]>

Apply comments

26a4979

Fix to code to consider backward compatibility and allow users to spe…

ebb4b6e

…cify HealthStatus

Apply comments

495ce58

- Add javadoc on `HealthStatus` - Checks for IF-NONE-MATCH header for specific `HealthStatus` - Add `HealthStatusUpdateHandler`

jrhee17 reviewed Jul 10, 2024

View reviewed changes

Apply comments

8489392

- Add @UnstableApi

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from a13308f to 74f702d Compare July 18, 2024 00:10

Modify HealthCHeckStatus to accept boolean and HealthStatus

2f610ee

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from 74f702d to 2f610ee Compare July 18, 2024 00:16

jrhee17 approved these changes Jul 19, 2024

View reviewed changes

ikhoon modified the milestones: 1.30.0, 1.31.0 Aug 2, 2024

ikhoon mentioned this pull request Aug 2, 2024

Improve xDS health check #5785

Merged

github-actions bot added the Stale label Sep 2, 2024

jrhee17 modified the milestones: 1.31.0, 1.32.0 Nov 5, 2024

github-actions bot removed the Stale label Nov 6, 2024

github-actions bot added the Stale label Dec 15, 2024

minwoox reviewed Feb 12, 2025

View reviewed changes

minwoox added 2 commits February 12, 2025 12:37

Merge branch 'main' into pr-5741-seonWKim-feature/ISSUE-5705-degradab…

c9d8885

…le-health-checker

More fix

6ca0b1d

minwoox reviewed Feb 12, 2025

View reviewed changes

github-actions bot removed the Stale label Feb 16, 2025

minwoox modified the milestones: 1.32.0, 1.33.0 Feb 24, 2025

github-actions bot added the Stale label Apr 11, 2025

ikhoon modified the milestones: 1.33.0, 1.34.0 Aug 1, 2025

github-actions bot removed the Stale label Aug 3, 2025

github-actions bot added the Stale label Sep 7, 2025

jrhee17 modified the milestones: 1.34.0, 1.35.0 Nov 24, 2025

github-actions bot removed the Stale label Nov 28, 2025

[ISSUE-5705] Support for degraded health #5741

Are you sure you want to change the base?

[ISSUE-5705] Support for degraded health #5741

Conversation

seonWKim commented Jun 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Build Scan® (commit: 6ca0b1d)

Uh oh!

trustin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrhee17 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trustin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrhee17 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jrhee17 left a comment

Choose a reason for hiding this comment

Uh oh!

minwoox Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

minwoox left a comment

Choose a reason for hiding this comment

Uh oh!

minwoox Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

minwoox Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

seonWKim commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

seonWKim commented Jun 8, 2024 •

edited

Loading

github-actions bot commented Jun 8, 2024 •

edited

Loading

🔍 Build Scan® (commit: `6ca0b1d`)