Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

etcd client starts retrying transient errors from the etcd cluster #105069

Merged
merged 1 commit into from Sep 16, 2021
Merged

etcd client starts retrying transient errors from the etcd cluster #105069

merged 1 commit into from Sep 16, 2021

Conversation

p0lyn0mial
Copy link
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

It makes the etcd client (used by the API server) retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72

This PR enables the unaryClientInterceptor in conjunction with the Prometheus interceptor. Previously it was simply overwritten by the Prometheus interceptor. As a result etcd client didn't attempt to retry certain errors.

The unaryClientInterceptor is important because it knows how to retry all sorts of errors from the etcd cluster. It will make the API server more resilient to failures - end users won't see certain errors.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

makes the etcd client (used by the API server) retry certain types of errors. The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@p0lyn0mial
etcd-client starts retrying transient errors from the etcd cluster
8317156 
This PR enables unaryClientInterceptor in conjunction with Prometheus interceptor.
Previously it was simply overwritten by the Prometheus interceptor.
As a result etcd client didn't attempt to retry certain errors.

The unaryClientInterceptor is important because it knows how to retry all sorts of errors from the etcd cluster. It will make the API server more resilient to failures -  end users won't see certain errors.
The full list of retriable (codes.Unavailable) errors can be found at https://github.com/etcd-io/etcd/blob/main/api/v3rpc/rpctypes/error.go#L72
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 16, 2021
@p0lyn0mial
Copy link
Contributor Author

/assign @hexfusion @sttts

@k8s-ci-robot
Copy link
Contributor

@p0lyn0mial: GitHub didn't allow me to assign the following users: hexfusion.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @hexfusion @sttts

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 16, 2021
@wojtek-t
Copy link
Member

/lgtm
/approve

Thanks for the fix!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 16, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: p0lyn0mial, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 16, 2021
@k8s-triage-robot
Copy link

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

  • The PR does have any do-not-merge/* labels
  • The PR does not have the needs-ok-to-test label
  • The PR is mergeable (does not have a needs-rebase label)
  • The PR is approved (has cncf-cla: yes, lgtm, approved labels)
  • The PR is failing tests required for merge

You can:

/retest

@k8s-ci-robot k8s-ci-robot merged commit 63e7ee4 into kubernetes:master Sep 16, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Sep 16, 2021
@caesarxuchao
Copy link
Member

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Sep 16, 2021
@openshift-ci-robot openshift-ci-robot mentioned this pull request Sep 22, 2021
Merged
@openshift-ci-robot openshift-ci-robot mentioned this pull request Oct 7, 2021
Merged
@openshift-ci-robot openshift-ci-robot mentioned this pull request Oct 29, 2021
Closed
This was referenced Nov 10, 2021
Merged
Merged
benluddy added a commit to benluddy/kubernetes that referenced this pull request Jan 20, 2022
@benluddy
Revert "Merge pull request kubernetes#105069 from p0lyn0mial/upstream…
ad25fd4 
…-etcd-client-retry"

This reverts commit 63e7ee4, reversing
changes made to af85e0e.
This was referenced Jun 2, 2022
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Jun 9, 2022
@k8s-ci-robot
Merge pull request #110356 from p0lyn0mial/automated-cherry-pick-of-#…
4bf8569 
…105069-upstream-release-1.21

Automated cherry pick of #105069: etcd-client starts retrying transient errors from the etcd
k8s-ci-robot added a commit that referenced this pull request Jun 9, 2022
@k8s-ci-robot
Merge pull request #110355 from p0lyn0mial/automated-cherry-pick-of-#…
9bad366 
…105069-upstream-release-1.22

Automated cherry pick of #105069: etcd-client starts retrying transient errors from the etcd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timothysc timothysc Awaiting requested review from timothysc

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

Assignees

@sttts sttts

@wojtek-t wojtek-t

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@p0lyn0mial @k8s-ci-robot @wojtek-t @k8s-triage-robot @caesarxuchao @sttts