Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix repeatedly aquire the inhibit lock #98088

Merged
merged 1 commit into from Jan 22, 2021
Merged

Fix repeatedly aquire the inhibit lock #98088

merged 1 commit into from Jan 22, 2021

Conversation

wzshiming
Copy link
Member

@wzshiming wzshiming commented Jan 15, 2021 •
edited by liggitt

What type of PR is this?
/kind bug
/sig node

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #98087

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

kubelet: Fix repeatedly acquiring the inhibit lock

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 15, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @wzshiming. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/kubelet labels Jan 15, 2021
@ehashman ehashman added this to Triage in SIG Node PR Triage Jan 15, 2021
@wzshiming
Copy link
Member Author

/assign @vishh

@bobbypage
Copy link
Member

/assign

@bobbypage
Copy link
Member

@wzshiming can you please provide some more info on how you got into the state where kubelet obtained more than one inhibit lock?

As far as I understand, Kubelet should only acquire the inhibit lock here on startup

kubernetes/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go

Line 132 in af46c47

err = m.aquireInhibitLock()

and here

kubernetes/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go

Lines 160 to 163 in af46c47

if isShuttingDown {
m.processShutdownEvent()
} else {
m.aquireInhibitLock()

if isShuttingDown == false (a pending shutdown was cancelled). However, the inhibitLock should be released by processShutdownEvent:

kubernetes/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go

Lines 251 to 252 in af46c47

m.dbusCon.ReleaseInhibitLock(m.inhibitLock)
klog.V(1).Infof("Shutdown manager completed processing shutdown event, node will shutdown shortly")

and processShutdownEvent is called in the loop above and should be a blocking call (i.e. I would expect before any shutdown events to be processed for processShutdownEvent to return, thus ensuring that aquireInhibitLock is only acquired if existing one is released).

@wzshiming
Copy link
Member Author

Run it many times gdbus emit --system --object-path /org/freedesktop/login1 --signal org.freedesktop.login1.Manager.PrepareForShutdown false.
aquireInhibitLock is called multiple times, causing the inhibitLock to leak.

@bobbypage
Copy link
Member

Run it many times gdbus emit --system --object-path /org/freedesktop/login1 --signal org.freedesktop.login1.Manager.PrepareForShutdown false.
aquireInhibitLock is called multiple times, causing the inhibitLock to leak.

Got it, thanks. That's because we call aquireInhibitLock if PrepareForShutdown false event is received with the assumption that a prior PrepareForShutdown == true event was sent in the past. But if you send it manually like that or due to whatever reason PrepareForShutdown == true, wasn't sent before we will incorrectly aquire another inhibit lock.

I think it should be fine to release the inhibit lock if it's already acquired as you proposed.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2021
@bobbypage
Copy link
Member

/ok-to-test
/lgtm
/triage accepted
/assign @mrunalp

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 21, 2021
@bobbypage bobbypage moved this from Triage to Needs Approver in SIG Node PR Triage Jan 21, 2021
@wzshiming
Copy link
Member Author

/test pull-kubernetes-e2e-gce-ubuntu-containerd

@mrunalp
Copy link
Contributor

mrunalp commented Jan 22, 2021

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mrunalp, wzshiming

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 22, 2021
@wzshiming
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 82ebcd1 into kubernetes:master Jan 22, 2021
SIG Node PR Triage automation moved this from Needs Approver to Done Jan 22, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Jan 22, 2021
@wzshiming wzshiming deleted the fix-inhibit-lock branch January 22, 2021 08:45
@wzshiming wzshiming mentioned this pull request Jan 25, 2021
Closed
6 tasks
@wzshiming wzshiming mentioned this pull request Feb 20, 2021
Merged
jingxu97 pushed a commit to jingxu97/kubernetes that referenced this pull request Mar 2, 2021
@wzshiming
kubelet: Fix repeatedly aquire the inhibit lock
18daff9 
Cherrypicking kubernetes#98088 from upstream

kubernetes#98088

Change-Id: Ieae2da50a21ec8c208449f9350c63cb2737fd845
k8s-ci-robot added a commit that referenced this pull request Mar 12, 2021
@k8s-ci-robot
Merge pull request #99255 from wzshiming/automated-cherry-pick-of-#98…
a1b0301 
…088-upstream-release-1.20

Automated cherry pick of #98088: Fix repeatedly aquire the inhibit lock
@bobbypage bobbypage mentioned this pull request Mar 18, 2021
Merged
13 tasks
@tossmilestone tossmilestone mentioned this pull request Jun 30, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krmayankk krmayankk Awaiting requested review from krmayankk

@odinuge odinuge Awaiting requested review from odinuge

Assignees

@mrunalp mrunalp

@bobbypage bobbypage

@vishh vishh

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node PR Triage
Done
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

GracefulNodeShutdown Repeatedly aquire the inhibit lock
5 participants
@wzshiming @k8s-ci-robot @bobbypage @mrunalp @vishh