Add server level feature gate

[siyuanfoundation]

What would you like to be added?

Add the Kubernetes style feature gate framework into etcd to gate future feature enhancement.
Users would be able to turn features on or off for the single server and query feature enablement in a consistent way.

This issue is for tracking KEP-4578.
Tasks in this KEP include

• Milestone 1

• server feature gate interface defined, and code copied from K8s into pkg/: featuregate: adds EtcdServer.FeatureEnabled interface. #18062.
• --feature-gates flag added: Add "server-feature-gates" flag. #18279, Add config file field for feature-gates flag. #18365.
• feature gate added to the server code, and used by a server level experimental feature: migrate experimental-stop-grpc-service-on-defrag flag to feature gate. #18359
• feature metrics added. Add Prometheus gauge metrics for server feature gate #19324

• Milestone 2

• e2e tests added for the feature gate equivalent of the selected experimental feature.
• robustness test scenarios added for the selected experimental feature(s).
• documentation to track all feature gates added to [etcd-io/website]: Add documentation to track all feature gates added. website#940.

• Milestone 3

• migrate all existing --experimental server level features to feature gates without removing the old flags.
  List of features to migrate:
  • experimental-stop-grpc-service-on-defrag: migrate experimental-stop-grpc-service-on-defrag flag to feature gate. #18359
  • experimental-enable-distributed-tracing: migrate --experimental-enable-distributed-tracing flag to feature gate #19050
  • experimental-initial-corrupt-check: KEP-4578: migrate experimental-initial-corrupt-check flag to feature gate. #18478
  • experimental-compact-hash-check-enabled: migrate experimental-compact-hash-check-enabled to feature gate. #19053
  • experimental-enable-lease-checkpoint and experimental-enable-lease-checkpoint-persist: migrate experimental-enable-lease-checkpoint and experimental-enable-lease-checkpoint-persist flag to feature gate. #19356
  • experimental-memory-mlock: migrate experimental-memory-mlock flag to feature gate #19061
  • experimental-txn-mode-write-with-shared-buffer: migrate experimental-txn-mode-write-with-shared-buffer flag to feature gate #19063

Why is this needed?

To make it easier to add and use new features in etcd.
Please refer to kubernetes/enhancements#4578 for more details.

[siyuanfoundation]

while kubernetes/enhancements#4610 is still in review, I think there is no much contention regarding server level feature gate. we can get started on the implementation.
/cc @stackbaek @henrybear327

[henrybear327]

Thanks @siyuanfoundation for the heads up! Happy to contribute to this feature :) Looking forward to it!

[serathius]

Can we wait for approvals for the KEP so the implementation done by contributors don't need to be updated when the design details are changed frustrating the contributors?

[siyuanfoundation]

/cc @YoyinZyc

[ahrtr]

Several remaining issues/tasks:

• The flag --experimental-set-member-localaddr is missed. It's added in 3.6 only, so we can just change it to feature gate. assigned to @henrybear327
  • Action: A PR is needed. Contribution is welcome. Thanks. Migrate --experimental-set-member-localaddr to using feature flag #19413
  • Action: Please also double check if we miss any other experimental flags
• All the deprecated fields are not correctly marked. The comment should begins with Deprecated:, refer to https://go.dev/wiki/Deprecated. The benefit is that IDEs will automatically draw a line through the field, so with better awareness to the users. assigned to @ivanvc
  • Action: A PR is needed. Contribution is welcome. Thanks. Done in Fix v3.6 deprecation notices #19401
• The deprecated comment/description isn't consistent across "help/flag description/field comment" etc. For example, for --experimental-txn-mode-write-with-shared-buffer, we only added "deprecated ..." in the help.go, but not in the field comment, nor the flag description. Fixed by @ivanvc in Fix more v3.6 deprecation comments #19408
  • Action: A PR is needed. Contribution is welcome. Thanks
• For embedded use cases, if an experimental flag was migrated to a non-experimental flag, users aren't able to set the experimental field any more, It won't work. For example

  etcd/tests/integration/tracing_test.go

  Line 59 in 6485d48

  cfg.ExperimentalEnableDistributedTracing = true

  Instead users have to set the related non-experimental field. It's a breaking change, we need to clearly document this.
  • Action: documentation
• For embedded use cases, if an experimental flag was migrated to the feature gate, users aren't able to set the experimental field any more, It won't work. It's also a breaking change. We need to document this, also we need to add document to users how to set it.
  • Action: documentation

cc @fuweid @jmhbnz @siyuanfoundation @henrybear327 @ivanvc @serathius

[henrybear327]

I can take on some tasks while the e2e PRs are being reviewed / merged!
Please assign it directly to me and I will submit the PR before midnight!

Thanks @ahrtr

[ahrtr]

@henrybear327 thx, can you deliver a PR for The flag --experimental-set-member-localaddr is missed firstly?

[henrybear327]

@henrybear327 thx, can you deliver a PR for The flag --experimental-set-member-localaddr is missed firstly?

Duly noted.

[ivanvc]

I'll take the second one:

All the deprecated fields are not correctly marked.

[ivanvc]

The deprecated comment/description isn't consistent across "help/flag description/field comment" etc. For example, for --experimental-txn-mode-write-with-shared-buffer, we only added "deprecated ..." in the help.go, but not in the field comment, nor the flag description.

I did another pass at deprecations and fixed the following:

• ExperimentalTxnModeWriteWithSharedBuffer
• ExperimentalStopGRPCServiceOnDefrag
• ExperimentalInitialCorruptCheck
• Standardized all deprecation messages in server/etcdmain/help.go

See PR: #19408

[ahrtr]

@henrybear327 thx, can you deliver a PR for The flag --experimental-set-member-localaddr is missed firstly?

Duly noted.

@henrybear327 are you able to deliver a PR to fix this today (asap)? thx.

[henrybear327]

@henrybear327 thx, can you deliver a PR for The flag --experimental-set-member-localaddr is missed firstly?

Duly noted.

@henrybear327 are you able to deliver a PR to fix this today (asap)? thx.

Yes, I am on it. Work is currently ongoing as I ran into some issues that I am debugging now!

[ahrtr]

@henrybear327 the migration of the missing experimental flag --experimental-set-member-localaddr has higher priority than the downgrade e2e test (#19399) to me.

[henrybear327]

@henrybear327 the migration of the missing experimental flag --experimental-set-member-localaddr has higher priority than the downgrade e2e test (#19399) to me.

Sorry for the delay, initial attempt is put up.