Nodes that haven't finish creation are abandoned and left for registration TTL (15m) when Karpenter restarts

[comtalyst]

Version

Karpenter Version: v1.6.3

Kubernetes Version: v1.32

Expected Behavior

• Karpenter decides to scale-up, start creating a node
• Karpenter restarts (e.g., due to configuration changes from cluster update, if using NAP)
  ...
• Node creation is completed, as if there is no restart

Actual Behavior

• Karpenter decides to scale-up, start creating a node
• Karpenter restarts (e.g., due to configuration changes from cluster update, if using NAP)
  ...
• Ongoing node creation is abandoned
• After registration TTL (15 minutes), the node is cleaned up
  • This means 15 minutes of down time, where user workload is left unschedulable
• Karpenter creates a new node

Steps to Reproduce the Problem

Restart Karpenter while the node is being created, per the timeline above.
For NAP, Karpenter may be restarted with cluster update. Node image upgrade on system pool is one of the operations that trigger restart.

Resource Specs and Logs

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[comtalyst]

The state where "Karpenter needs to wait for the first step/request to finish, then do the next" should cease to exist after AKS Machine API integration. Existing instance reuse logic should still cover other cases and handle them quickly enough. This should effectively solves this issue.

Karpenter restarting on cluster updates should also be less frequent, also after AKS Machine API integration and internal changes, as Karpenter itself should be less relying on cluster configurations directly.

[VictorAtPL]

hi @comtalyst , we're experiencing this issue and escalated it via Support some time ago. Great to see it's in the backlog.

Quick question — with NAP, Karpenter restarts seem to occur relatively frequently (e.g., during any cluster update). If we were to self-host Karpenter instead, and avoid restarting it, would the issue with waiting 15mins for pod get assigned to a node occur less frequently?

[comtalyst]

Generally, Karpenter restarts are required to ensure that new nodes contain updated configurations. E.g., if the cluster update changes bootstrapping token and Karpenter did not read it in, then new nodes won't join the cluster.
Self-hosted cluster admins are required to ensure Karpenter restarts accordingly, while NAP does that automatically.

Although, if you suspect that Karpenter (NAP) is restarting more than it needs to, do you mind showing us the cluster update configuration?
Also note the distinction between the issues of "frequent restart" (lower-level) and "upcoming nodes stuck upon restart" (user facing, and represented by this ticket).