Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prefer nominated node - IMPL #93179

Merged
merged 1 commit into from Jan 27, 2021
Merged

prefer nominated node - IMPL #93179

merged 1 commit into from Jan 27, 2021

Conversation

chendave
Copy link
Member

@chendave chendave commented Jul 17, 2020 •
edited

Performance evaluation based on the metric of scheduler_framework_extension_point_duration_seconds/Filter in the case of preemption when the feature gate is enabled, workloads configuration is 500Nodes, 2000InitPods and 500PodsToSchedule.

xref: #96258

indicator before (ms) after (ms) change
Average 0.46601329799999947 0.3219608919999995 -30.9%
Perc50 0.2828025477707007 0.1457317073170732 -48.4%
Perc90 1.2829493087557604 1.0933940774487472 -14.8%
Perc99 1.4672811059907833 1.366742596810934 -6.9%
  • backup for the legacy approach - component based config and skip the schedule cycle when the nominated node is not fit.
    legacy metric name is scheduling_algorithm_predicate_evaluation_seconds, workloads configuration is 500Nodes, 2000InitPods and 500PodsToSchedule.
indicator before (ms) after (ms) change
Average 0.40068560239999973 0.20466781999999958 -48.9%
Perc50 0.5357908272610373 0.5070993914807302 -5.3%
Perc90 0.9644234890698671 0.9127789046653144 -5.4%
Perc99 1.2937062937062935 0.9634888438133874 -25.5%

Signed-off-by: Dave Chen dave.chen@arm.com

What type of PR is this?
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #93013

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Overall, enable the feature of `PreferNominatedNode` will  improve the performance of scheduling where preemption might frequently happen, but in theory, enable the feature of `PreferNominatedNode`, the pod might not be scheduled to the best candidate node in the cluster.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Please use the following format for linking documentation:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 17, 2020
@chendave
Copy link
Member Author

/cc @Huang-Wei @alculquicondor @ahg-g

will try to build a benchmark for this later.

@fejta-bot
Copy link

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

@chendave
Copy link
Member Author

/retest

@chendave
Copy link
Member Author

@alculquicondor thanks for your review, let's see whether @Huang-Wei and @ahg-g has anything to say about this.

@chendave chendave force-pushed the preemption_improve branch 2 times, most recently from 2559161 to 8c4c9ce Compare July 20, 2020 06:47
@k8s-ci-robot
Copy link
Contributor

@chendave: The label(s) kind/scheduling cannot be applied, because the repository doesn't have them

In response to this:

/kind scheduling

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@chendave
Copy link
Member Author

/sig scheduling

@chendave
Copy link
Member Author

/retest

2 similar comments
@chendave
Copy link
Member Author

/retest

@chendave
Copy link
Member Author

/retest

@chendave
Copy link
Member Author

kubernetes/test/integration/scheduler_perf/config/performance-config.yaml

Lines 230 to 235 in 43fbe17

- template:
desc: Preemption
initPods:
- podTemplatePath: config/pod-low-priority.yaml
podsToSchedule:
podTemplatePath: config/pod-high-priority.yaml
might be the right benchmark needed here, but I found some issue there and currently is still investigating into it.

@Huang-Wei
Copy link
Member

@alculquicondor Suppose we go with this new option, should the SchedulerConfig version be bumped to v1beta2 in 1.20?

ahg-g
ahg-g reviewed Jan 21, 2021
View reviewed changes
pkg/scheduler/core/generic_scheduler.go Outdated Show resolved Hide resolved
pkg/scheduler/core/generic_scheduler.go Outdated Show resolved Hide resolved
pkg/scheduler/core/generic_scheduler.go Show resolved Hide resolved
pkg/scheduler/core/generic_scheduler.go Outdated Show resolved Hide resolved
@chendave
Copy link
Member Author

@ahg-g @alculquicondor thanks for the review and comments!

@chendave
Copy link
Member Author

/retest

alculquicondor
alculquicondor reviewed Jan 22, 2021
View reviewed changes
pkg/scheduler/core/generic_scheduler.go Outdated Show resolved Hide resolved
pkg/scheduler/core/generic_scheduler.go Outdated Show resolved Hide resolved
pkg/scheduler/core/generic_scheduler_test.go Outdated Show resolved Hide resolved
@ahg-g
Copy link
Member

ahg-g commented Jan 22, 2021

/approve
/hold

approving the feature gate and the core logic, will leave the lgtm to Aldo.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 22, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, chendave

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 22, 2021
@chendave
Copy link
Member Author

commit squashed, comments addressed!
@alculquicondor PTAL, thanks!

@chendave
Copy link
Member Author

/retest

@chendave
Prefer nominated node in the new scheduling cycle
6d800ff 
Signed-off-by: Dave Chen <dave.chen@arm.com>
@chendave
Copy link
Member Author

/retest

alculquicondor
alculquicondor reviewed Jan 26, 2021
View reviewed changes
Copy link
Member

@alculquicondor alculquicondor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 26, 2021
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 26, 2021
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@chendave
Copy link
Member Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 21d3c73 into kubernetes:master Jan 27, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Jan 27, 2021
@chendave chendave deleted the preemption_improve branch January 27, 2021 05:06
@github-actions github-actions bot mentioned this pull request Feb 3, 2021
Open
@chendave chendave mentioned this pull request Feb 22, 2021
Closed
@chendave
Copy link
Member Author

per the requirement, link this to the enhancement issue: kubernetes/enhancements#1923

edwardstudy
edwardstudy reviewed Mar 3, 2021
View reviewed changes
pkg/scheduler/core/generic_scheduler.go
klog.ErrorS(err, "Evaluation failed on nominated node", "pod", klog.KObj(pod), "node", pod.Status.NominatedNodeName)
}
// Nominated node passes all the filters, scheduler is good to assign this node to the pod.
if len(feasibleNodes) != 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chendave Just curious. Could we remove nominatedNode from allNodes quickly if this result is empty? After that, this node should not go Filters/Extenders twice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the original version was removing this node from the node list to avoid the evaluation again later, but this require some slice manipulation, which might be cost more than just evaluate it again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is true today, as we don't have too much default filtering list, and cost of the filtering is not dramatic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@soulxu soulxu soulxu left review comments

@alculquicondor alculquicondor alculquicondor left review comments

@edwardstudy edwardstudy edwardstudy left review comments

@ahg-g ahg-g ahg-g left review comments

@Huang-Wei Huang-Wei Awaiting requested review from Huang-Wei

@vishh vishh Awaiting requested review from vishh

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

Honor "NominatedNodeName" during the new scheduling cycle
8 participants
@chendave @fejta-bot @k8s-ci-robot @Huang-Wei @alculquicondor @ahg-g @soulxu @edwardstudy