Add AcceleratorStats to cri_stats_provider #96873

ruiwen-zhao · 2020-11-25T21:52:14Z

What type of PR is this?
/kind bug

What this PR does / why we need it:
Currently cadvisor_stats_provider provides AcceleratorStats but cri_stats_provider does not. As a result, when using cri_stats_provider, kubelet's Summary API does not have accelerator metrics. This PR tries to fix the discrepancy between the two providers.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
Note that this PR does not add any new metric collecting logic, but rather merely passes the AcceleratorStats collected by cadvisor to cri_stats_provider. I am planning to cherry-pick this change to release branches after merging it to master.

Does this PR introduce a user-facing change?:

AcceleratorStats will be available in the Summary API of kubelet when cri_stats_provider is used.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

ruiwen-zhao · 2020-11-25T21:54:19Z

/cc @SergeyKanzhelev @bobbypage @dashpole @dchen1107 @Random-Liu @pradvenkat

k8s-ci-robot · 2020-11-25T21:54:23Z

@ruiwen-zhao: GitHub didn't allow me to request PR reviews from the following users: pradvenkat.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @SergeyKanzhelev @bobbypage @dashpole @dchen1107 @Random-Liu @pradvenkat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ruiwen-zhao · 2020-11-25T21:56:42Z

/triage accepted

dashpole · 2020-11-26T00:12:17Z

pkg/kubelet/stats/cri_stats_provider.go

@@ -780,6 +780,25 @@ func removeTerminatedContainers(containers []*runtimeapi.Container) []*runtimeap
 func (p *criStatsProvider) addCadvisorContainerStats(
 	cs *statsapi.ContainerStats,
 	caPodStats *cadvisorapiv2.ContainerInfo,
+) {
+        if caPodStats.Spec.HasCustomMetrics {
+                cs.UserDefinedMetrics = cadvisorInfoToUserDefinedMetrics(caPodStats)


Just curious, why are we also adding un custom cAdvisor metrics here? Just for completeness?

Did you mean this question to addCadvisorContainerCPUAndMemoryStats or here?

why are we also adding un custom cAdvisor metrics here?

Yeah I am basically keeping the same as the existing addCadvisorContainerStats.

dashpole · 2020-11-26T00:12:21Z

pkg/kubelet/stats/cri_stats_provider_test.go:702:9: should omit 2nd value from range; this loop is equivalent to for i := range ...

dashpole · 2020-11-30T22:05:58Z

pkg/kubelet/stats/cri_stats_provider.go

+	cs *statsapi.ContainerStats,
+	caPodStats *cadvisorapiv2.ContainerInfo,
+) {
+	if caPodStats.Spec.HasCustomMetrics {


We should definitely omit custom metrics if we only want CPU and Memory metrics

Actually, it looks like this is just maintaining the existing behavior

dashpole · 2020-11-30T22:10:00Z

/lgtm
/approve
/priority important-soon

k8s-ci-robot · 2020-11-30T22:10:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dashpole, ruiwen-zhao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/stats/OWNERS~~ [dashpole]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ruiwen-zhao · 2020-11-30T22:22:04Z

/milestone v1.20

k8s-ci-robot · 2020-11-30T22:22:06Z

@ruiwen-zhao: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dchen1107 · 2020-12-01T18:48:59Z

/milestone v1.20

fejta-bot · 2020-12-01T21:56:18Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

hasheddan · 2020-12-01T22:51:37Z

/hold

This should be reviewed by @kubernetes/release-team-leads as we are currently in code / test freeze for v1.20.

ruiwen-zhao · 2020-12-01T23:24:31Z

/retest

SergeyKanzhelev · 2020-12-02T01:05:18Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

ruiwen-zhao · 2020-12-02T16:10:16Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

ruiwen-zhao · 2020-12-02T17:34:30Z

/retest

ruiwen-zhao · 2020-12-04T00:55:15Z

/retest

ruiwen-zhao · 2020-12-04T17:34:03Z

/retest

liggitt · 2020-12-09T14:49:24Z

Can the hold be dropped now that master is open for 1.21?

jeremyrickard · 2020-12-09T14:55:32Z

/hold cancel

Thanks for waiting while we got 1.20 out. Hold dropped @liggitt

k8s-ci-robot requested review from resouer and sjenning November 25, 2020 21:52

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 25, 2020

k8s-ci-robot requested review from dchen1107, Random-Liu, SergeyKanzhelev and bobbypage November 25, 2020 21:54

k8s-ci-robot requested a review from dashpole November 25, 2020 21:54

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 25, 2020

dashpole reviewed Nov 26, 2020

View reviewed changes

Add AcceleratorStats to cri_stats_provider

36b4ccd

ruiwen-zhao requested a review from dashpole November 30, 2020 21:00

dashpole reviewed Nov 30, 2020

View reviewed changes

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 30, 2020

k8s-ci-robot assigned dashpole Nov 30, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 30, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 30, 2020

k8s-ci-robot added this to the v1.20 milestone Dec 1, 2020

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 1, 2020

ruiwen-zhao mentioned this pull request Dec 2, 2020

Add AcceleratorStats to cri_stats_provider for 1.19 #97017

Merged

ruiwen-zhao mentioned this pull request Dec 2, 2020

[1.20.1] Add AcceleratorStats to cri_stats_provider for 1.20 #97018

Merged

liggitt changed the title ~~Add AcceleratorStats to cri_stats_provider~~ [1.20.1] Add AcceleratorStats to cri_stats_provider Dec 7, 2020

liggitt changed the title ~~[1.20.1] Add AcceleratorStats to cri_stats_provider~~ Add AcceleratorStats to cri_stats_provider Dec 9, 2020

liggitt modified the milestones: v1.20, v1.21 Dec 9, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 9, 2020

k8s-ci-robot merged commit a20aeb8 into kubernetes:master Dec 9, 2020

github-actions bot mentioned this pull request Dec 16, 2020

Week Ending December 13, 2020 dev-obs/actus#308

Open

pacoxu mentioned this pull request Dec 28, 2020

update known issue status for 1.20 changelog #97517

Merged

qiutongs mentioned this pull request Apr 8, 2021

"Add AcceleratorStats to cri_stats_provider for 1.18" #100905

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AcceleratorStats to cri_stats_provider #96873

Add AcceleratorStats to cri_stats_provider #96873

ruiwen-zhao commented Nov 25, 2020

ruiwen-zhao commented Nov 25, 2020

k8s-ci-robot commented Nov 25, 2020

ruiwen-zhao commented Nov 25, 2020

dashpole Nov 26, 2020

SergeyKanzhelev Nov 26, 2020

ruiwen-zhao Nov 30, 2020

dashpole commented Nov 26, 2020

dashpole Nov 30, 2020

dashpole Nov 30, 2020

dashpole commented Nov 30, 2020

k8s-ci-robot commented Nov 30, 2020

ruiwen-zhao commented Nov 30, 2020

k8s-ci-robot commented Nov 30, 2020

dchen1107 commented Dec 1, 2020

fejta-bot commented Dec 1, 2020

hasheddan commented Dec 1, 2020

ruiwen-zhao commented Dec 1, 2020

SergeyKanzhelev commented Dec 2, 2020

ruiwen-zhao commented Dec 2, 2020

ruiwen-zhao commented Dec 2, 2020

ruiwen-zhao commented Dec 4, 2020

ruiwen-zhao commented Dec 4, 2020

liggitt commented Dec 9, 2020

jeremyrickard commented Dec 9, 2020

Add AcceleratorStats to cri_stats_provider #96873

Add AcceleratorStats to cri_stats_provider #96873

Conversation

ruiwen-zhao commented Nov 25, 2020

ruiwen-zhao commented Nov 25, 2020

k8s-ci-robot commented Nov 25, 2020

ruiwen-zhao commented Nov 25, 2020

dashpole Nov 26, 2020

Choose a reason for hiding this comment

SergeyKanzhelev Nov 26, 2020

Choose a reason for hiding this comment

ruiwen-zhao Nov 30, 2020

Choose a reason for hiding this comment

dashpole commented Nov 26, 2020

dashpole Nov 30, 2020

Choose a reason for hiding this comment

dashpole Nov 30, 2020

Choose a reason for hiding this comment

dashpole commented Nov 30, 2020

k8s-ci-robot commented Nov 30, 2020

ruiwen-zhao commented Nov 30, 2020

k8s-ci-robot commented Nov 30, 2020

dchen1107 commented Dec 1, 2020

fejta-bot commented Dec 1, 2020

hasheddan commented Dec 1, 2020

ruiwen-zhao commented Dec 1, 2020

SergeyKanzhelev commented Dec 2, 2020

ruiwen-zhao commented Dec 2, 2020

ruiwen-zhao commented Dec 2, 2020

ruiwen-zhao commented Dec 4, 2020

ruiwen-zhao commented Dec 4, 2020

liggitt commented Dec 9, 2020

jeremyrickard commented Dec 9, 2020