Add migrated field to storage_operation_duration_seconds metric #99050

Jiawei0227 · 2021-02-13T01:40:11Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds a "migrated" field to the existing storage_operation_duration_seconds metrics that are being exposed by kube-controller-mananger and kubelet.
The migrated field will be used to determine if the underlying volume it is operating is a CSI migrated volume. It will help SRE to better monitor the status of the CSI migration feature.

This is implemented by extend the opComplete method to record an extra metric field.

Which issue(s) this PR fixes:

Partially #98279

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Add `migrated` field into `storage_operation_duration_seconds` metric

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig storage
/assign @msau42
/cc @verult

k8s-ci-robot · 2021-02-13T01:40:18Z

@Jiawei0227: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2021-02-16T17:45:25Z

pkg/volume/util/operationexecutor/operation_generator.go

@@ -227,7 +227,8 @@ func (og *operationGenerator) GenerateVolumesAreAttachedFunc(
 			}
 		}

-		return nil, nil
+		// It is hard to differentiate migrated status for all volumes for verify_volumes_are_attached_per_node


Do we actually do translation for the verify operations?

We do translation when generate the VolumeSpec. So it does not matter if it is verify operation.

verify is a bulk operation thus is difficult to differentiate. It will check attachedVolumes(which is a collection of VolumeSpec), and these VolumeSpecs, it can be partial CSI migrated specs and some real CSI specs. So we cannot tell the difference.

msau42 · 2021-02-16T17:52:50Z

pkg/volume/util/metrics.go

@@ -47,7 +49,7 @@ var storageOperationMetric = metrics.NewHistogramVec(
 		Buckets:        []float64{.1, .25, .5, 1, 2.5, 5, 10, 15, 25, 50, 120, 300, 600},
 		StabilityLevel: metrics.ALPHA,
 	},
-	[]string{"volume_plugin", "operation_name", "status"},
+	[]string{"volume_plugin", "operation_name", "status", "migrated"},


Can you enhance

kubernetes/test/e2e/storage/testsuites/base.go

Line 758 in 7604762

func getMigrationVolumeOpCounts(cs clientset.Interface, pluginName string) (opCounts, opCounts) {

to also check for the new field in the metric?

I checked the implementation of this here, some observation:

The idea of migration volume metrics checking is to compare the in-tree plugin metrics has not changed before/after the test execution. => which implies no in-tree operation is invoked.

This check does not utilize the corresponding CSI metrics at all. I think the reason might be

The test can be execute parallel and it would be hard to compare accurate result.

And also per the comments, some negative test cases might not emit any metrics.

This is done using the to-be-deprecated metric "storage_operation_status_count". So we need to replace it with the correct metric.

And it seems that my change of adding a migrated status does not fit in the picture as we are not checking the csi metrics at all. And even if we want to check it will be hard because the observation 2 I mentioned above.

Ok, maybe we need a todo to try to test this via unit tests.

msau42 · 2021-02-17T00:40:31Z

/lgtm
/approve

k8s-ci-robot · 2021-02-17T00:40:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jiawei0227, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/controller/volume/expand/OWNERS~~ [msau42]
~~pkg/controller/volume/persistentvolume/OWNERS~~ [msau42]
~~pkg/volume/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Jiawei0227 · 2021-02-17T01:05:44Z

/retest

Jiawei0227 · 2021-02-17T01:52:18Z

/retest

verult · 2021-02-17T02:18:57Z

pkg/volume/util/types/types.go

+			Err:      &context.DetailedErr,
+			Migrated: &context.Migrated,
+		}
+		c.Err = &detailedErr


do we need this line?

Nice catch! I will delete this in another PR

verult · 2021-02-17T02:45:36Z

pkg/controller/volume/persistentvolume/pv_controller.go

@@ -1558,7 +1559,7 @@ func (ctrl *PersistentVolumeController) provisionClaimOperation(

 	opComplete := util.OperationCompleteHook(plugin.GetPluginName(), "volume_provision")
 	volume, err = provisioner.Provision(selectedNode, allowedTopologies)
-	opComplete(&err)
+	opComplete(volumetypes.CompleteFuncParam{Err: &err})


Should the migrated field be part of provision and delete calls as well?

For CSI Migration, if a plugin has been migrated, then provision and delete will be handle by csi-provisioner. So any call happens here is a non-csi migration call

verult · 2021-02-17T02:50:08Z

pkg/volume/volume_linux.go

-		completeFunc(&err)
+		completeFunc(types.CompleteFuncParam{
+			Err: &err,
+		})


Maybe less important but do we also want to pass in migrated info here for consistency? How hard is it to get the migrated field here?

Hmm, I might be wrong but this seems not to be a migrated operation? That being said, there is no difference in in-tree/CSI for this operation. So by default it will be migrated:false

Add migrated field to storage_operation_duration_seconds metric

6a7222c

k8s-ci-robot assigned msau42 Feb 13, 2021

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Feb 13, 2021

k8s-ci-robot requested a review from verult February 13, 2021 01:40

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. sig/storage Categorizes an issue or PR as relevant to SIG Storage. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 13, 2021

Jiawei0227 added 3 commits February 12, 2021 17:50

update bazel

3d61b56

Fix typecheck

e64309c

Fix bazel build

9ec820a

msau42 reviewed Feb 16, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 17, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 17, 2021

verult reviewed Feb 17, 2021

View reviewed changes

k8s-ci-robot merged commit aa28a35 into kubernetes:master Feb 17, 2021

k8s-ci-robot added this to the v1.21 milestone Feb 17, 2021

verult reviewed Feb 17, 2021

View reviewed changes

Jiawei0227 mentioned this pull request Feb 19, 2021

CSI Migration Metrics #98279

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add migrated field to storage_operation_duration_seconds metric #99050

Add migrated field to storage_operation_duration_seconds metric #99050

Jiawei0227 commented Feb 13, 2021

k8s-ci-robot commented Feb 13, 2021

msau42 Feb 16, 2021

Jiawei0227 Feb 16, 2021 •

edited

msau42 Feb 16, 2021

Jiawei0227 Feb 17, 2021

msau42 Feb 17, 2021

msau42 commented Feb 17, 2021

k8s-ci-robot commented Feb 17, 2021

Jiawei0227 commented Feb 17, 2021

Jiawei0227 commented Feb 17, 2021

verult Feb 17, 2021

Jiawei0227 Feb 17, 2021

verult Feb 17, 2021

Jiawei0227 Feb 17, 2021

verult Feb 17, 2021

Jiawei0227 Feb 17, 2021

Add migrated field to storage_operation_duration_seconds metric #99050

Add migrated field to storage_operation_duration_seconds metric #99050

Conversation

Jiawei0227 commented Feb 13, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Feb 13, 2021

Choose a reason for hiding this comment

Jiawei0227 Feb 16, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Feb 17, 2021

k8s-ci-robot commented Feb 17, 2021

Jiawei0227 commented Feb 17, 2021

Jiawei0227 commented Feb 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jiawei0227 Feb 16, 2021 •

edited