Upgrade to etcd 3.6 recovers old/duplicate members from store

[thechristschn]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

After upgrading the first node (node3) of our 5-node etcd cluster from 3.5.25 to 3.6.6, etcd recovers duplicate members from the store.
They have different IDs, but the same name, peer address and client address.

Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.071990Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"fc2bc6ba22cf8949","recovered-remote-peer-urls":["https://node1:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072010Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"17f509730dfb2deb","recovered-remote-peer-urls":["https://node2:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072027Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"2c242739ba404a36","recovered-remote-peer-urls":["https://node1:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072050Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"4b53351423911bb0","recovered-remote-peer-urls":["https://node3:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072063Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"54fbb3e529b23215","recovered-remote-peer-urls":["https://node5:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072085Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"77edd50ec79a221a","recovered-remote-peer-urls":["https://node3:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072104Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"9dca8d575473b780","recovered-remote-peer-urls":["https://node4:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072116Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"cba918f4c53c7475","recovered-remote-peer-urls":["https://node2:2380"],"recovered-remote-peer-is-learner":false}

$ etcdctl --endpoints=https://node3:2379 member list -wtable
+------------------+---------+-------+--------------------+--------------------+------------+
|        ID        | STATUS  | NAME  |    PEER ADDRS      |   CLIENT ADDRS     | IS LEARNER |
+------------------+---------+-------+--------------------+--------------------+------------+
| 17f509730dfb2deb | started | node2 | https://node2:2380 | https://node2:2379 |      false |
| 2c242739ba404a36 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
| 4b53351423911bb0 | started | node3 | https://node3:2380 | https://node3:2379 |      false |
| 54fbb3e529b23215 | started | node5 | https://node5:2380 | https://node5:2379 |      false |
| 77edd50ec79a221a | started | node3 | https://node3:2380 | https://node3:2379 |      false |
| 9dca8d575473b780 | started | node4 | https://node4:2380 | https://node4:2379 |      false |
| cba918f4c53c7475 | started | node2 | https://node2:2380 | https://node2:2379 |      false |
| fc2bc6ba22cf8949 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
+------------------+---------+-------+--------------------+--------------------+------------+

Before the update there were only 5 members and so I would expect to only see the same 5 members after the upgrade.

This only happens if I execute etcdctl member list against the upgraded node. If I check against the current leader, member information is normal.

This results in a lot of failed to find remote peer in cluster warning messages. It also breaks our automation for the node upgrades, because we don't expect multiple members with the same name and client/peer address.

I suspect, that v2 and v3 member store are somehow inconsistent and etcd 3.6 load them differently. But I have no idea how to debug or verify that. I can only see those log messages:

Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067567Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:77edd50ec79a221a RaftAttributes:{PeerURLs:[https://node3:2380] IsLearner:false} Attributes:{Name:node3 ClientURLs:[https://node3:2379]}}"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067597Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:fc2bc6ba22cf8949 RaftAttributes:{PeerURLs:[https://node1:2380] IsLearner:false} Attributes:{Name:node1 ClientURLs:[https://node1:2379]}}"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067614Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:17f509730dfb2deb RaftAttributes:{PeerURLs:[https://node2:2380] IsLearner:false} Attributes:{Name:node2 ClientURLs:[https://node2:2379]}}"}

I can delete the wrong members with etcdctl member remove <id>. But as soon as I remove the upgraded node3 from the cluster and add it back again, the old entries also resurface.

If I remove node3 from the cluster again (with its working memberID), downgrade the binary back to 3.5.25 and add the node back again, those duplicate members don't show up.

We regularly redeploy all members of the cluster one after another. This means we remove the member with etcdctl member remove <id>, stop etcd, wipe the whole node on which etcd is running, add member again, start etcd. This process has also been done at least once with 3.5.25.

What did you expect to happen?

The list of members is the same as before the upgrade.

How can we reproduce it (as minimally and precisely as possible)?

Unfortunately, I have no idea how to reproduce this from scratch.

In my case it is:

• Stop etcd on node3
• Download 3.6.6 binaries
• Start etcd with new binaries on node3
• Check the output of etcdctl member list against node3 and see members that haven't been there before.

Anything else we need to know?

The cluster is many years old and has been maintained and upgraded by many people many times. Maybe some inconsistency started building up at some point?

Etcd version (please run commands below)

Before update:

$ etcd --version
etcd Version: 3.5.25
Git SHA: e2eff77
Go Version: go1.24.10
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.25
API version: 3.5

After update:

$ etcd --version
etcd Version: 3.6.6
Git SHA: d2809cf
Go Version: go1.24.10
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.6.6
API version: 3.6

Etcd configuration (command line flags or environment variables)

"--unsafe-no-fsync",
"--name=node3",
"--cert-file=/secrets/etcd_crt.pem",
"--key-file=/secrets/etcd_key.pem",
"--peer-cert-file=/secrets/etcd_crt.pem",
"--peer-key-file=/secrets/etcd_key.pem",
"--trusted-ca-file=/secrets/etcd_ca.pem",
"--peer-trusted-ca-file=/secrets/etcd_ca.pem",
"--initial-cluster=node1=https://node1:2380,node5=https://node5:2380,node4=https://node4:2380,node2=https://node2:2380,node3=https://node3:2380",
"--initial-advertise-peer-urls=https://node3:2380",
"--listen-peer-urls=https://10.234.32.194:2380",
"--listen-client-urls=https://10.234.32.194:2379",
"--advertise-client-urls=https://node3:2379",
"--client-cert-auth",
"--peer-client-cert-auth",
"--initial-cluster-token=cluster-name",
"--initial-cluster-state=existing",
"--data-dir=/var/lib/etcd",
"--listen-metrics-urls=http://10.234.32.194:2801",
"--logger=zap",
"--log-outputs=systemd/journal",
"--quota-backend-bytes=4294967296"

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Before the update

$ etcdctl member list -w table
+------------------+---------+-------+--------------------+--------------------+------------+
|        ID        | STATUS  | NAME  |  PEER ADDRS        |     CLIENT ADDRS   | IS LEARNER |
+------------------+---------+-------+--------------------+--------------------+------------+
| 2c242739ba404a36 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
| 4b53351423911bb0 | started | node3 | https://node3:2380 | https://node3:2379 |      false |
| 54fbb3e529b23215 | started | node5 | https://node5:2380 | https://node5:2379 |      false |
| 9dca8d575473b780 | started | node4 | https://node4:2380 | https://node4:2379 |      false |
| cba918f4c53c7475 | started | node2 | https://node2:2380 | https://node2:2379 |      false |
+------------------+---------+-------+--------------------+--------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
+--------------------+------------------+---------+---------+-----------+------------+-----------+-------------+--------------------+--------+
|    ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX  | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+-------------+--------------------+--------+
| https://node3:2379 | 4b53351423911bb0 |  3.5.25 |  3.0 GB |     false |      false |      2683 | 12882792186 |        12882792186 |        |
| https://node1:2379 | 2c242739ba404a36 |  3.5.25 |  3.0 GB |      true |      false |      2683 | 12882792188 |        12882792188 |        |
| https://node5:2379 | 54fbb3e529b23215 |  3.5.25 |  3.0 GB |     false |      false |      2683 | 12882792190 |        12882792190 |        |
| https://node4:2379 | 9dca8d575473b780 |  3.5.25 |  3.0 GB |     false |      false |      2683 | 12882792191 |        12882792191 |        |
| https://node2:2379 | cba918f4c53c7475 |  3.5.25 |  3.0 GB |     false |      false |      2683 | 12882792192 |        12882792192 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+-------------+--------------------+--------+

After the update:

$ etcdctl --endpoints=https://node3:2379 member list -wtable
+------------------+---------+-------+--------------------+--------------------+------------+
|        ID        | STATUS  | NAME  |    PEER ADDRS      |   CLIENT ADDRS     | IS LEARNER |
+------------------+---------+-------+--------------------+--------------------+------------+
| 17f509730dfb2deb | started | node2 | https://node2:2380 | https://node2:2379 |      false |
| 2c242739ba404a36 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
| 4b53351423911bb0 | started | node3 | https://node3:2380 | https://node3:2379 |      false |
| 54fbb3e529b23215 | started | node5 | https://node5:2380 | https://node5:2379 |      false |
| 9dca8d575473b780 | started | node4 | https://node4:2380 | https://node4:2379 |      false |
| cba918f4c53c7475 | started | node2 | https://node2:2380 | https://node2:2379 |      false |
| fc2bc6ba22cf8949 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
+------------------+---------+-------+--------------------+--------------------+------------+

$ etcdctl --cert=/secrets/etcd_crt.pem --key=/secrets/etcd_key.pem --cacert=/secrets/etcd_ca.pem --endpoints=https://node1:2379 member list -wtable
+------------------+---------+-------+--------------------+--------------------+------------+
|        ID        | STATUS  | NAME  |    PEER ADDRS      |    CLIENT ADDRS    | IS LEARNER |
+------------------+---------+-------+--------------------+--------------------+------------+
| 2c242739ba404a36 | started | node1 | https://node1:2380 | https://node1:2379 |      false |
| 4b53351423911bb0 | started | node3 | https://node3:2380 | https://node3:2379 |      false |
| 54fbb3e529b23215 | started | node5 | https://node5:2380 | https://node5:2379 |      false |
| 9dca8d575473b780 | started | node4 | https://node4:2380 | https://node4:2379 |      false |
| cba918f4c53c7475 | started | node2 | https://node2:2380 | https://node2:2379 |      false |
+------------------+---------+-------+--------------------+--------------------+------------+

Relevant log output

Nov 24 08:38:12 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:12.716424Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.716519Z","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["/opt/bin/etcd","--unsafe-no-fsync","--name=node3","--cert-file=/secrets/etcd_crt.pem","--key-file=/secrets/etcd_key.pem","--peer-cert-file=/secrets/etcd_crt.pem","--peer-key-file=/secrets/etcd_key.pem","--trusted-ca-file=/secrets/etcd_ca.pem","--peer-trusted-ca-file=/secrets/etcd_ca.pem","--initial-cluster=node1=https://node1:2380,node5=https://node5:2380,node4=https://node4:2380,node2=https://node2:2380,node3=https://node3:2380","--initial-advertise-peer-urls=https://node3:2380","--listen-peer-urls=https://10.234.32.194:2380","--listen-client-urls=https://10.234.32.194:2379","--advertise-client-urls=https://node3:2379","--client-cert-auth","--peer-client-cert-auth","--initial-cluster-token=cluster-name","--initial-cluster-state=existing","--data-dir=/var/lib/etcd","--listen-metrics-urls=http://10.234.32.194:2801","--logger=zap","--log-outputs=systemd/journal","--quota-backend-bytes=4294967296"]}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.716605Z","caller":"etcdmain/etcd.go:107","msg":"server has already been initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:12.716652Z","caller":"embed/config.go:1209","msg":"Running http and grpc server on single port. This is not recommended for production."}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.716690Z","caller":"embed/etcd.go:138","msg":"configuring peer listeners","listen-peer-urls":["https://10.234.32.194:2380"]}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.716718Z","caller":"embed/etcd.go:544","msg":"starting with peer TLS","tls-info":"cert = /secrets/etcd_crt.pem, key = /secrets/etcd_key.pem, client-cert=, client-key=, trusted-ca = /secrets/etcd_ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.719799Z","caller":"embed/etcd.go:146","msg":"configuring client listeners","listen-client-urls":["https://10.234.32.194:2379"]}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.719983Z","caller":"embed/etcd.go:323","msg":"starting an etcd server","etcd-version":"3.6.6","git-sha":"d2809cf","go-version":"go1.24.10","go-os":"linux","go-arch":"amd64","max-cpu-set":6,"max-cpu-available":32,"member-initialized":true,"name":"node3","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://node3:2380"],"listen-peer-urls":["https://10.234.32.194:2380"],"advertise-client-urls":["https://node3:2379"],"listen-client-urls":["https://10.234.32.194:2379"],"listen-metrics-urls":["http://10.234.32.194:2801"],"experimental-local-address":"","cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"feature-gates":"","initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","discovery-token":"","discovery-endpoints":"","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":true,"discovery-insecure-skip-tls-verify":false,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1,"v2-deprecation":"write-only"}
Nov 24 08:38:12 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:12.720490Z","logger":"bbolt","caller":"backend/backend.go:203","msg":"Opening db file (/var/lib/etcd/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: true, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: hashmap, ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 4724464025, PageSize: 0, NoSync: true, OpenFile: 0x0, Mlock: false, Logger: 0xc000384168}"}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.288918Z","logger":"bbolt","caller":"bbolt@v1.4.3/db.go:321","msg":"Opening bbolt db (/var/lib/etcd/member/snap/db) successfully"}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.289070Z","caller":"storage/backend.go:80","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"568.651853ms"}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.289127Z","caller":"etcdserver/bootstrap.go:220","msg":"restore consistentIndex","index":12882810844}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.617267Z","caller":"etcdserver/bootstrap.go:413","msg":"recovered v2 store from snapshot","snapshot-index":12882755267,"snapshot-size":"854 kB"}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.617387Z","caller":"storage/backend.go:108","msg":"Skipping snapshot backend","consistent-index":12882810844,"snapshot-index":12882755267}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.617446Z","caller":"etcdserver/bootstrap.go:232","msg":"recovered v3 backend","backend-size-bytes":2995113984,"backend-size":"3.0 GB","backend-size-in-use-bytes":1956429824,"backend-size-in-use":"2.0 GB"}
Nov 24 08:38:13 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:13.617655Z","caller":"etcdserver/bootstrap.go:90","msg":"Bootstrapping WAL from snapshot"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.065089Z","caller":"etcdserver/bootstrap.go:599","msg":"restarting local member","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","commit-index":12882810844}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.065169Z","caller":"etcdserver/bootstrap.go:94","msg":"bootstrapping cluster"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.065256Z","caller":"etcdserver/bootstrap.go:101","msg":"bootstrapping storage"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067567Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:77edd50ec79a221a RaftAttributes:{PeerURLs:[https://node3:2380] IsLearner:false} Attributes:{Name:node3 ClientURLs:[https://node3:2379]}}"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067597Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:fc2bc6ba22cf8949 RaftAttributes:{PeerURLs:[https://node1:2380] IsLearner:false} Attributes:{Name:node1 ClientURLs:[https://node1:2379]}}"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.067614Z","caller":"membership/cluster.go:605","msg":"Detected member only in v3store but missing in v2store","member":"{ID:17f509730dfb2deb RaftAttributes:{PeerURLs:[https://node2:2380] IsLearner:false} Attributes:{Name:node2 ClientURLs:[https://node2:2379]}}"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.071959Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.5"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.071990Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"fc2bc6ba22cf8949","recovered-remote-peer-urls":["https://node1:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072010Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"17f509730dfb2deb","recovered-remote-peer-urls":["https://node2:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072027Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"2c242739ba404a36","recovered-remote-peer-urls":["https://node1:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072050Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"4b53351423911bb0","recovered-remote-peer-urls":["https://node3:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072063Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"54fbb3e529b23215","recovered-remote-peer-urls":["https://node5:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072085Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"77edd50ec79a221a","recovered-remote-peer-urls":["https://node3:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072104Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"9dca8d575473b780","recovered-remote-peer-urls":["https://node4:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072116Z","caller":"membership/cluster.go:297","msg":"recovered/added member from store","cluster-id":"a99dd61e7f994020","local-member-id":"4b53351423911bb0","recovered-remote-peer-id":"cba918f4c53c7475","recovered-remote-peer-urls":["https://node2:2380"],"recovered-remote-peer-is-learner":false}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072131Z","caller":"membership/cluster.go:307","msg":"set cluster version from store","cluster-version":"3.5"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072151Z","caller":"etcdserver/bootstrap.go:109","msg":"bootstrapping raft"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072528Z","caller":"etcdserver/server.go:312","msg":"bootstrap successfully"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072834Z","logger":"raft","caller":"v3@v3.6.0/raft.go:1981","msg":"4b53351423911bb0 switched to configuration voters=(3180710365721807414 5427740336523910064 6123685915168748053 11370055615468779392 14675288300323632245)"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072889Z","logger":"raft","caller":"v3@v3.6.0/raft.go:897","msg":"4b53351423911bb0 became follower at term 2683"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.072908Z","logger":"raft","caller":"v3@v3.6.0/raft.go:493","msg":"newRaft 4b53351423911bb0 [peers: [2c242739ba404a36,4b53351423911bb0,54fbb3e529b23215,9dca8d575473b780,cba918f4c53c7475], term: 2683, commit: 12882810844, applied: 12882755267, lastindex: 12882810844, lastterm: 2683]"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.073130Z","caller":"auth/store.go:1135","msg":"simple token is not cryptographically signed"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.073209Z","caller":"mvcc/kvstore.go:334","msg":"restored last compact revision","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":12648476198}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734145Z","caller":"mvcc/kvstore.go:408","msg":"kvstore restored","current-rev":12648563909}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734392Z","caller":"storage/quota.go:116","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":4294967296,"quota-size":"4.3 GB"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734826Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734856Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734980Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735006Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb","remote-peer-urls":["https://node2:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735024Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.734970Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735039Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735042Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735140Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735155Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735158Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735178Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36","remote-peer-urls":["https://node1:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735191Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735074Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735018Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"17f509730dfb2deb"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735211Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735202Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735332Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735597Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735599Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735583Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735602Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735692Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215","remote-peer-urls":["https://node5:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735732Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735753Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735638Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735817Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735841Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a","remote-peer-urls":["https://node3:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735838Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735797Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735839Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735858Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"77edd50ec79a221a"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735861Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735934Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735999Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735998Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.735990Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736023Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780","remote-peer-urls":["https://node4:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736050Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736083Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736031Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736145Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736266Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736298Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475","remote-peer-urls":["https://node2:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736323Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736329Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736346Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736346Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736315Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736304Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736410Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736421Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736413Z","caller":"rafthttp/stream.go:168","msg":"started stream writer with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736394Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736472Z","caller":"rafthttp/stream.go:394","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736491Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"fc2bc6ba22cf8949","remote-peer-urls":["https://node1:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736539Z","caller":"etcdserver/server.go:589","msg":"starting etcd server","local-member-id":"4b53351423911bb0","local-server-version":"3.6.6","cluster-id":"a99dd61e7f994020","cluster-version":"3.5"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736638Z","caller":"etcdserver/server.go:499","msg":"starting initial election tick advance","election-ticks":10}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736721Z","caller":"embed/etcd.go:766","msg":"starting with client TLS","tls-info":"cert = /secrets/etcd_crt.pem, key = /secrets/etcd_key.pem, client-cert=, client-key=, trusted-ca = /secrets/etcd_ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736827Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736870Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736890Z","caller":"fileutil/purge.go:49","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.736962Z","caller":"embed/etcd.go:292","msg":"now serving peer/client/metrics","local-member-id":"4b53351423911bb0","initial-advertise-peer-urls":["https://node3:2380"],"listen-peer-urls":["https://10.234.32.194:2380"],"advertise-client-urls":["https://node3:2379"],"listen-client-urls":["https://10.234.32.194:2379"],"listen-metrics-urls":["http://10.234.32.194:2801"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.737010Z","caller":"embed/etcd.go:890","msg":"serving metrics","address":"http://10.234.32.194:2801"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.737073Z","caller":"embed/etcd.go:640","msg":"serving peer traffic","address":"10.234.32.194:2380"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.737110Z","caller":"embed/etcd.go:611","msg":"cmux::serve","address":"10.234.32.194:2380"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.742001Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.742043Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744292Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"cba918f4c53c7475","stream-type":"stream MsgApp v2"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744328Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744352Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744347Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"9dca8d575473b780","stream-type":"stream MsgApp v2"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744380Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744397Z","logger":"raft","caller":"v3@v3.6.0/node.go:370","msg":"raft.node: 4b53351423911bb0 elected leader 2c242739ba404a36 at term 2683"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744428Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"cba918f4c53c7475","stream-type":"stream Message"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744453Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744426Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"54fbb3e529b23215","stream-type":"stream MsgApp v2"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744464Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744477Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744495Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744556Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"2c242739ba404a36","stream-type":"stream MsgApp v2"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744580Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744602Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744581Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"9dca8d575473b780","stream-type":"stream Message"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744649Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"2c242739ba404a36","stream-type":"stream Message"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744669Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744660Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"9dca8d575473b780"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744622Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"4b53351423911bb0","to":"54fbb3e529b23215","stream-type":"stream Message"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744708Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744734Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"4b53351423911bb0"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.744747Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"4b53351423911bb0","remote-peer-id":"4b53351423911bb0","remote-peer-urls":["https://node3:2380"]}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.744763Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.744834Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.747881Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.748148Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"cba918f4c53c7475"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.748400Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.748406Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.750942Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"2c242739ba404a36"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.754117Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"4b53351423911bb0","remote-peer-id":"54fbb3e529b23215"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.756791Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.756863Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.789340Z","caller":"etcdserver/server.go:522","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"4b53351423911bb0","forward-ticks":8,"forward-duration":"800ms","election-ticks":10,"election-timeout":"1s","active-remote-members":4}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:14.840502Z","caller":"wal/wal.go:825","msg":"created a new WAL segment","path":"/var/lib/etcd/member/wal/0000000000000012-00000002ffe027b9.wal"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.853822Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.860135Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.958246Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:14 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:14.958337Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.054404Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.054601Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:15.149421Z","caller":"etcdserver/server.go:2201","msg":"triggering snapshot","local-member-id":"4b53351423911bb0","local-member-applied-index":12882765345,"local-member-snapshot-index":12882755267,"local-member-snapshot-count":10000,"snapshot-forced":false}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.152008Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.152507Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:15.154155Z","caller":"etcdserver/server.go:2246","msg":"saved snapshot to disk","snapshot-index":12882765345}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.262640Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.262738Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.357131Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.357209Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.454423Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.454510Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.553183Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"warn","ts":"2025-11-24T08:38:15.558522Z","caller":"rafthttp/http.go:415","msg":"failed to find remote peer in cluster","local-member-id":"4b53351423911bb0","remote-peer-id-stream-handler":"4b53351423911bb0","remote-peer-id-from":"4b53351423911bb0","cluster-id":"a99dd61e7f994020"}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:15.570018Z","caller":"etcdserver/server.go:2201","msg":"triggering snapshot","local-member-id":"4b53351423911bb0","local-member-applied-index":12882775347,"local-member-snapshot-index":12882765345,"local-member-snapshot-count":10000,"snapshot-forced":false}
Nov 24 08:38:15 node3 etcd[568390]: {"level":"info","ts":"2025-11-24T08:38:15.574744Z","caller":"etcdserver/server.go:2246","msg":"saved snapshot to disk","snapshot-index":12882775347}

[brainbug95]

I saw the same thing on our k3s based cluster when tying to upgrade from k3s v1.33.5+k3s1 which uses to v3.5.21-k3s1 to v1.34.1+k3s1 which uses v3.6.4-k3s3.

We also upgrade our cluster by replacing all nodes one by one. When I start an new server-node which should join the existing etcd-cluster it gets refused because the new server-node etcd sees ancient etcd-cluster members and therefor thinks the cluster is unhealthy and stops the joining process. This has kept me from upgrading so far.

[ahrtr]

I saw the same thing on our k3s based cluster when tying to upgrade from k3s v1.33.5+k3s1 which uses to v3.5.21-k3s1 to v1.34.1+k3s1 which uses v3.6.4-k3s3.

@brainbug95
3.5.21 is not safe to upgrade to 3.6.x. Please ensure you are on etcd 3.5.24+ before upgrading to etcd v3.6.6. Refer to https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue_followup/

After upgrading the first node (node3) of our 5-node etcd cluster from 3.5.25 to 3.6.6, etcd recovers duplicate members from the store.

@thechristschn Most likely the v3store isn't consistent with v2store.

To avoid long back and forth communication, is it possible to share the data of node3? If not, is it possible to share it to me (e.g. via google drive) only? My email is here

[ahrtr]

One workaround is to create a single-member cluster from the leader, and then add other members into the cluster one by one. @thechristschn can you try the workaround and feedback whether it work? (Before you try the workaround, please backup all the data and logs for debugging)

[thechristschn]

To avoid long back and forth communication, is it possible to share the data of node3?

@ahrtr unfortunately, I can't share the data just right away. I would have to put some effort into removing all relevant secret data somehow first. The etcd cluster is used for kubernetes.

One workaround is to create a single-member cluster from the leader, and then add other members into the cluster one by one.

What exactly do you mean by that?

I tried to remove all members except the leader, restarted etcd with --force-new-cluster and then added the other nodes back again, but the error persisted. But at some point I wasn't 100% sure if that flag really worked as I expected. Maybe I can try again (If that is what you are referring to). Should I try that still with the 3.5.25 binary or with 3.6 only?

We face this is our integration environment where I can have a downtime for testing, but for production we ideally need a solution without downtime.

[ahrtr]

@ahrtr unfortunately, I can't share the data just right away. I would have to put some effort into removing all relevant secret data somehow first. The etcd cluster is used for kubernetes.

Understood. That's the reason why I asked if it's possible to share the data to me only. Without data, it would mean that there will be a long back and forth inefficient communication.

restarted etcd with --force-new-cluster and then added the other nodes back again, but the error persisted.

Please do this when you are still on v3.5.25,

• add --force-new-cluster to the leader, and restart leader to create a single-member cluster. Once it's done, remove the flag, and restart the leader again;
• add other members to the cluster one by one
• at last, try to upgrade to v3.6.

[thechristschn]

It is reproducible even if I copy the data-dir of a node to my local machine and start etcd with --force-new-cluster there. It will still recover the old members if I add a second etcd instance with v3.6.6.

And this way I can delete all keys from etcd (followed by compaction and defrag to really get rid of them). I've sent @ahrtr cleaned data dir.

[ahrtr]

I had an online debug session with @thechristschn. Summary below.

Reason

Membership data is inconsistent between v2store and v3store:
v2store shows 5 members, but v3store shows 8. The 3 extra members in v3store have duplicate peerURLs matching existing members.

In etcd v3.5.x, v2store is the source of truth. In etcd v3.6.x, v3store is the source of truth. Therefore, the issue only appears when upgrading from v3.5 to v3.6.

Root Cause

The root cause is unclear. But it's very likely it's the same issue as #13196.

In v3.4.x, v2store is the source of truth for membership data. In etcd v3.5.0, etcd switched to v3store, which triggered the issue #13196 when upgrading from v3.4.14 to v3.5.0. The “fix” in #13348 simply reverted the source of truth back to v2store starting from v3.5.1, without addressing the actual underlying issue.

So the real bug was introduced somewhere in v3.4.x and has never been fully resolved.

Impact on v3.6

In v3.6, etcd uses v3store exclusively for membership data. As long as the cluster successfully upgrades to v3.6, it will not be affected.

The problem is specifically the upgrade path from v3.5.x to v3.6.x, which may fail due to the inconsistent membership data.

Next Steps

1. Identify the root cause

Ideally we should still find and fix the actual underlying issue, though it no longer affects v3.6 directly.

2. Auto sync membership data between v2store and v3store

We should ensure v2store and v3store remain consistent during bootstrap and after each ConfChange (We should ensure etcdserver have finished replaying the WAL records before performing the auto-sync). This fix should go into release-3.5 only, so users must upgrade to v3.5.26 (with the fix) before upgrading to v3.6.x.

@fuweid do you have bandwidth to take care of this?

3. Fix the --force-new-cluster issue

When users hit this issue, they should be able to recover by starting a single-member cluster with --force-new-cluster and then re-adding members. But currently, v3.6 incorrectly cleans up membership based on v2store, even though v3store is the source of truth. We should cleanup based on v3store instead.

This fix should go into main and be backported to release-3.6. I’ve already implemented it in
#20981; I just need to extract it into a separate PR and backport it. @ahrtr will take care of this.

EDIT (2025-12-01): I realized that fixing this separately in release-3.6 is quite difficult because it still bootstraps from v2snapshot. Addressing the issue in release-3.6 would be almost equivalent to backporting #20981 to 3.6, which is what I have been trying to avoid. We should be good as long as etcd supports auto-syncing membership data between v2store and v3store as mentioned above.

4. New Blog Post

@jberkus Unfortunately, it seems we may need to publish another blog post once items 2 and 3 are completed.

[thechristschn]

Thanks @ahrtr for the debugging session.

For documentation, here is the output of the member stores:

v3store, dumped with https://github.com/vmware/etcd-diagnosis v0.1.0:

$ etcd-diagnosis iterate-bucket var/lib/etcd members
key="fc2bc6ba22cf8949", value="{\"id\":18170835624306641225,\"peerURLs\":[\"https://node1:2380\"],\"name\":\"node1\",\"clientURLs\":[\"https://node1:2379\"]}"
key="eae796b82ba04374", value="{\"id\":16926663442245370740,\"peerURLs\":[\"https://node2:2380\"],\"name\":\"node2\",\"clientURLs\":[\"https://node2:2379\"]}"
key="ca5a658c0050ca30", value="{\"id\":14581078395540523568,\"peerURLs\":[\"https://node1:2380\"],\"name\":\"node1\",\"clientURLs\":[\"https://node1:2379\"]}"
key="95d2ffd07975c615", value="{\"id\":10795972527616542229,\"peerURLs\":[\"https://node5:2380\"],\"name\":\"node5\",\"clientURLs\":[\"https://node5:2379\"]}"
key="83eb5bd8eca494b6", value="{\"id\":9505792425736836278,\"peerURLs\":[\"https://node4:2380\"],\"name\":\"node4\",\"clientURLs\":[\"https://node4:2379\"]}"
key="77edd50ec79a221a", value="{\"id\":8641797519448875546,\"peerURLs\":[\"https://node3:2380\"],\"name\":\"node3\",\"clientURLs\":[\"https://node3:2379\"]}"
key="563f2d8062cb2ab0", value="{\"id\":6214736040231119536,\"peerURLs\":[\"https://node3:2380\"],\"name\":\"node3\",\"clientURLs\":[\"https://node3:2379\"]}"
key="17f509730dfb2deb", value="{\"id\":1726296421926907371,\"peerURLs\":[\"https://node2:2380\"],\"name\":\"node2\",\"clientURLs\":[\"https://node2:2379\"]}"

v2store, dumped with grep and jq:

$ grep -ao '{.*}' var/lib/etcd/member/snap/0000000000000a7d-0000000302949740.snap | jq '.Root.Children."0".Children.members'
{
  "Path": "/0/members",
  "CreatedIndex": 1,
  "ModifiedIndex": 1,
  "ExpireTime": "0001-01-01T00:00:00Z",
  "Value": "",
  "Children": {
    "563f2d8062cb2ab0": {
      "Path": "/0/members/563f2d8062cb2ab0",
      "CreatedIndex": 15437,
      "ModifiedIndex": 15437,
      "ExpireTime": "0001-01-01T00:00:00Z",
      "Value": "",
      "Children": {
        "attributes": {
          "Path": "/0/members/563f2d8062cb2ab0/attributes",
          "CreatedIndex": 15440,
          "ModifiedIndex": 15440,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"name\":\"node3\",\"clientURLs\":[\"https://node3:2379\"]}",
          "Children": null
        },
        "raftAttributes": {
          "Path": "/0/members/563f2d8062cb2ab0/raftAttributes",
          "CreatedIndex": 15437,
          "ModifiedIndex": 15437,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"peerURLs\":[\"https://node3:2380\"]}",
          "Children": null
        }
      }
    },
    "83eb5bd8eca494b6": {
      "Path": "/0/members/83eb5bd8eca494b6",
      "CreatedIndex": 15449,
      "ModifiedIndex": 15449,
      "ExpireTime": "0001-01-01T00:00:00Z",
      "Value": "",
      "Children": {
        "attributes": {
          "Path": "/0/members/83eb5bd8eca494b6/attributes",
          "CreatedIndex": 15451,
          "ModifiedIndex": 15451,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"name\":\"node4\",\"clientURLs\":[\"https://node4:2379\"]}",
          "Children": null
        },
        "raftAttributes": {
          "Path": "/0/members/83eb5bd8eca494b6/raftAttributes",
          "CreatedIndex": 15449,
          "ModifiedIndex": 15452,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"peerURLs\":[\"https://node4:2380\"]}",
          "Children": null
        }
      }
    },
    "95d2ffd07975c615": {
      "Path": "/0/members/95d2ffd07975c615",
      "CreatedIndex": 15443,
      "ModifiedIndex": 15443,
      "ExpireTime": "0001-01-01T00:00:00Z",
      "Value": "",
      "Children": {
        "attributes": {
          "Path": "/0/members/95d2ffd07975c615/attributes",
          "CreatedIndex": 15445,
          "ModifiedIndex": 15445,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"name\":\"node5\",\"clientURLs\":[\"https://node5:2379\"]}",
          "Children": null
        },
        "raftAttributes": {
          "Path": "/0/members/95d2ffd07975c615/raftAttributes",
          "CreatedIndex": 15443,
          "ModifiedIndex": 15446,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"peerURLs\":[\"https://node5:2380\"]}",
          "Children": null
        }
      }
    },
    "ca5a658c0050ca30": {
      "Path": "/0/members/ca5a658c0050ca30",
      "CreatedIndex": 15425,
      "ModifiedIndex": 15425,
      "ExpireTime": "0001-01-01T00:00:00Z",
      "Value": "",
      "Children": {
        "attributes": {
          "Path": "/0/members/ca5a658c0050ca30/attributes",
          "CreatedIndex": 15428,
          "ModifiedIndex": 15428,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"name\":\"node1\",\"clientURLs\":[\"https://node1:2379\"]}",
          "Children": null
        },
        "raftAttributes": {
          "Path": "/0/members/ca5a658c0050ca30/raftAttributes",
          "CreatedIndex": 15425,
          "ModifiedIndex": 15425,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"peerURLs\":[\"https://node1:2380\"]}",
          "Children": null
        }
      }
    },
    "eae796b82ba04374": {
      "Path": "/0/members/eae796b82ba04374",
      "CreatedIndex": 15431,
      "ModifiedIndex": 15431,
      "ExpireTime": "0001-01-01T00:00:00Z",
      "Value": "",
      "Children": {
        "attributes": {
          "Path": "/0/members/eae796b82ba04374/attributes",
          "CreatedIndex": 15434,
          "ModifiedIndex": 15434,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"name\":\"node2\",\"clientURLs\":[\"https://node2:2379\"]}",
          "Children": null
        },
        "raftAttributes": {
          "Path": "/0/members/eae796b82ba04374/raftAttributes",
          "CreatedIndex": 15431,
          "ModifiedIndex": 15431,
          "ExpireTime": "0001-01-01T00:00:00Z",
          "Value": "{\"peerURLs\":[\"https://node2:2380\"]}",
          "Children": null
        }
      }
    }
  }
}

v2store shows as expected the 5 members of the cluster.
v3store shows 8 members, the same 5 from v2store but also three additional members.

[jberkus]

Ack, ready to work on this once we know more.