Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lease manager limit max objects attached to a lease #98257

Merged
merged 1 commit into from Feb 1, 2021
Merged

lease manager limit max objects attached to a lease #98257

merged 1 commit into from Feb 1, 2021

Conversation

lingsamuel
Copy link
Contributor

@lingsamuel lingsamuel commented Jan 21, 2021 •
edited by wojtek-t

Signed-off-by: Ling Samuel lingsamuelgrace@gmail.com

What type of PR is this?

/kind feature

What this PR does / why we need it:

Details: #96836

Which issue(s) this PR fixes:

Part of #96836

Special notes for your reviewer:

Default value set to 10k per #97480 (comment)

Does this PR introduce a user-facing change?:

A lease can only attach up to 1k objects.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 21, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @lingsamuel. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jan 21, 2021
@lingsamuel
Copy link
Contributor Author

/cc @wojtek-t @mborsz

@k8s-ci-robot k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 21, 2021
@lingsamuel lingsamuel force-pushed the etcd-lease-max-count branch 2 times, most recently from 5541099 to 5bbb303 Compare January 21, 2021 08:42
@mborsz
Copy link
Member

mborsz commented Jan 21, 2021

Thanks for the PR!

#97480 (comment) I meant that we may want to play with '--lease-reuse-duration-seconds' value, before we introduce, but after thinking about that, with this PR we will solve it more precisely (after this PR we will create new lease either after 1m from the previous lease or 10k objects, which should spread. e.g. first lease from that comment into 6-7 smaller leases spread evenly in the first minute).

The value 10k looks reasonable to me (this is ~what we are seeing during the test when there are no performance problems).

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2021
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2021
@lingsamuel
Copy link
Contributor Author

@mborsz I fixed a wrong test, need a new lgtm label

@fedebongio
Copy link
Contributor

/assign @lavalamp
/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 21, 2021
@lavalamp
Copy link
Member

I would really prefer not to add another flag. Can we experiment and find a good value and then hard code it?

cc @deads2k @jpbetz @jingyih @wenjiaswe

@lavalamp
Copy link
Member

lavalamp commented Feb 1, 2021

/lgtm
/approve

This seems reasonable to me now. Thanks!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 1, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, lingsamuel, ptabor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 1, 2021
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit f812209 into kubernetes:master Feb 1, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 1, 2021
@wojtek-t
Copy link
Member

wojtek-t commented Feb 2, 2021

LGTM - I guess we can expose an option to configure it if needed later.

@ptabor ptabor mentioned this pull request Feb 2, 2021
Closed
@github-actions github-actions bot mentioned this pull request Feb 3, 2021
Open
@mborsz
Copy link
Member

mborsz commented Feb 3, 2021

Thanks for this change!

It reduced CPU usage of events etcd from ~10-15 cores to 1-2 cores:
image
Link

This is really impressive!

@wojtek-t
Copy link
Member

wojtek-t commented Feb 3, 2021

wow - this is amazing. I expected improvements, but definitely not that huge...

@dims
Copy link
Member

dims commented Feb 3, 2021

@mborsz @wojtek-t very cool!

@lingsamuel lingsamuel deleted the etcd-lease-max-count branch February 4, 2021 06:46
@ptabor ptabor mentioned this pull request Feb 4, 2021
Closed
@ptabor
Copy link
Contributor

ptabor commented Feb 4, 2021

Really nice. Do you have a chart that shows tail latency of e.g. api-server requests ?

@mborsz
Copy link
Member

mborsz commented Feb 4, 2021

There is some visible impact on e.g. events POST latency, but it's rather small:

image

@ptabor
Copy link
Contributor

ptabor commented Feb 4, 2021

Thank you. There is hope it will mitigate the huge unpredictive spikes.

@ptabor ptabor mentioned this pull request Mar 9, 2021
Closed
@mborsz mborsz mentioned this pull request Mar 10, 2021
Merged
@wojtek-t wojtek-t added kind/bug Categorizes issue or PR as related to a bug. and removed kind/feature Categorizes issue or PR as related to a new feature. labels Mar 12, 2021
This was referenced Mar 22, 2021
Merged
Merged
k8s-ci-robot added a commit that referenced this pull request Apr 8, 2021
@k8s-ci-robot
Merge pull request #100450 from mborsz/automated-cherry-pick-of-#97009-…
75aca05 
…#97480-#98257-upstream-release-1.19

[1.19] Automated cherry pick of fixes for "large leases overload event etcd" issue (96836)
k8s-ci-robot added a commit that referenced this pull request Apr 8, 2021
@k8s-ci-robot
Merge pull request #100084 from mborsz/automated-cherry-pick-of-#97009-…
ca5eb11 
…#97480-#98257-upstream-release-1.20

[1.20] Automated cherry pick of fixes for "large leases overload event etcd" issue (96836)
k8s-ci-robot added a commit that referenced this pull request Apr 8, 2021
@k8s-ci-robot
Merge pull request #100452 from mborsz/automated-cherry-pick-of-#97009-…
8e67804 
…#97480-#98257-upstream-release-1.18

[1.18] Automated cherry pick of fixes for "large leases overload event etcd" issue (96836)
@chaochn47 chaochn47 mentioned this pull request Mar 10, 2022
Closed
@marseel marseel mentioned this pull request Jun 2, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ptabor ptabor ptabor approved these changes

@mborsz mborsz Awaiting requested review from mborsz

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

@caesarxuchao caesarxuchao Awaiting requested review from caesarxuchao

@dims dims Awaiting requested review from dims

@lavalamp lavalamp Awaiting requested review from lavalamp

Assignees

@lavalamp lavalamp

@mborsz mborsz

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

None yet