Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apiserver: add --permit-address-sharing flag to listen with SO_REUSEADDR #93861

Merged
merged 1 commit into from Feb 13, 2021
Merged

apiserver: add --permit-address-sharing flag to listen with SO_REUSEADDR #93861

merged 1 commit into from Feb 13, 2021

Conversation

sttts
Copy link
Contributor

@sttts sttts commented Aug 10, 2020 •
edited

If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state.

This is useful when restarting kube-apiserver: sockets in TIME_WAIT can take up to some minutes depending on kernel settings to be released, stopping the new kube-apiserver instance to launch, even though the old one has long been terminated.

/kind feature

Add --permit-address-sharing flag to kube-apiserver to listen with SO_REUSEADDR. While allowing to listen on wildcard IPs like 0.0.0.0 and specific IPs in parallel, it avoid waiting for the kernel to release socket in TIME_WAIT state, and hence, considably reducing kube-apiserver restart times under certain conditions.

For reference, https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ/14388707#14388707 is a great discussion about the background of SO_REUSEADDR.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Aug 10, 2020
@sttts
Copy link
Contributor Author

sttts commented Aug 10, 2020

/assign @bparees @deads2k

@k8s-ci-robot k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 10, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2020
@sttts sttts mentioned this pull request Aug 10, 2020
Merged
@sttts
Copy link
Contributor Author

sttts commented Aug 10, 2020

/retest

@bparees
Copy link
Contributor

bparees commented Aug 10, 2020

it looks fine to me but there seems to be some debate within the apiserver team about the merits of this flag so i'm going to abstain from lgtm'ing it while that is negotiated.

@liggitt
Copy link
Member

liggitt commented Aug 11, 2020

how does this differ from #88893?

@bparees
Copy link
Contributor

bparees commented Aug 11, 2020

how does this differ from #88893?

REUSEADDR has different implications than REUSEPORT does, this is adding the former, not the latter.

https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ

@liggitt
Copy link
Member

liggitt commented Aug 11, 2020

ah, missed that

@sttts
Copy link
Contributor Author

sttts commented Aug 27, 2020

@liggitt any objections to add this?

@liggitt
Copy link
Member

liggitt commented Aug 27, 2020

It seems coherent with the SO_REUSEPORT option. Are there any risks we need to be aware of or call out for people using this option?

I am a little confused by the help text ("...This allows binding to ... specific IPs in parallel..."). Does that mean that the existing reuse-port option is not sufficient to let two servers both bind to 1.2.3.4:8080 at the same time?

@sttts
Copy link
Contributor Author

sttts commented Aug 29, 2020

@liggitt take a look at https://stackoverflow.com/a/14388707. The behaviour of this flag is anything but trivial. Compare my downstream (openshift#309) comment:

This allows to bind before a closed socket leave TIME_WAIT state.
Note: this does not allow another process to bind to the same addr/port in non TIME_WAIT state.

In other words, this cuts down the time only until the socket is released by the existing process after it has closed it. This avoids that the port is blocked up to minutes sometimes (depending on kernel setting afaik) although the process is long terminated.

@sttts
Copy link
Contributor Author

sttts commented Aug 29, 2020

If you have a good way to express that in one sentence for release notes, am open for suggestions. I fear that you have to read the actual kernel docs to really understand what's going on. Nevertheless the feature is without workaround if you really need it.

@liggitt
Copy link
Member

liggitt commented Aug 31, 2020

some verify errors, lgtm otherwise

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 29, 2020
@sttts
Copy link
Contributor Author

sttts commented Nov 30, 2020

/remove-lifecycle stale

@sttts sttts added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 12, 2021
@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Feb 12, 2021
@sttts
Copy link
Contributor Author

sttts commented Feb 12, 2021

Fixed the gofmt error.

@sttts
Copy link
Contributor Author

sttts commented Feb 12, 2021

/retest

@p0lyn0mial
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2021
aojea
aojea reviewed Feb 12, 2021
View reviewed changes
staging/src/k8s.io/apiserver/pkg/server/options/serving.go Outdated
@@ -203,6 +207,11 @@ func (s *SecureServingOptions) AddFlags(fs *pflag.FlagSet) {
fs.BoolVar(&s.PermitPortSharing, "permit-port-sharing", s.PermitPortSharing,
"If true, SO_REUSEPORT will be used when binding the port, which allows "+
"more than one instance to bind on the same address and port. [default=false]")

fs.BoolVar(&s.PermitAddressSharing, "permit-address-sharing", s.PermitPortSharing,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be the s.PermitPortSharing at the end of the line PermitAddressSharing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, good catch.

/hold

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

/hold cancel
/lgtm

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2021
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2021
@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Feb 12, 2021
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

5 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 7f083d3 into kubernetes:master Feb 13, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 13, 2021
@hexfusion hexfusion mentioned this pull request Feb 19, 2021
Merged
@pires
Copy link
Contributor

pires commented Feb 25, 2021

cc @jdef

@openshift-ci-robot openshift-ci-robot mentioned this pull request Mar 5, 2021
Closed
This was referenced Mar 9, 2021
Merged
Merged
@openshift-ci-robot openshift-ci-robot mentioned this pull request Mar 23, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aojea aojea aojea left review comments

@p0lyn0mial p0lyn0mial p0lyn0mial left review comments

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@wojtek-t wojtek-t Awaiting requested review from wojtek-t

Assignees

@bparees bparees

@deads2k deads2k

@p0lyn0mial p0lyn0mial

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

None yet
9 participants
@sttts @k8s-ci-robot @bparees @liggitt @fejta-bot @p0lyn0mial @pires @aojea @deads2k