Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get filesystem stats for files on Windows #99221

Merged
merged 1 commit into from Mar 10, 2021
Merged

Get filesystem stats for files on Windows #99221

merged 1 commit into from Mar 10, 2021

Conversation

jsturtevant
Copy link
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

Pod log stats for Windows doesn't work. With this update the stats for log files are included in summary:

http://localhost:8001/api/v1/nodes/2733k8s01000000/proxy/stats/summary (output truncated, two containers 1 with log output, 1 without)

 "pods": [
  {
   "podRef": {
    "name": "iis-1809-668c5b9c87-fqqml",
    "namespace": "default",
    "uid": "84ff19a1-f0bb-4ff9-bc0b-b291d0f86818"
   },
   "startTime": "2021-02-19T00:59:15Z",
   "containers": [
    {
     "name": "iis",
     "startTime": "2021-02-19T00:59:16Z",
     "cpu": {
      "time": "2021-02-19T01:14:27Z",
      "usageNanoCores": 4680917,
      "usageCoreNanoSeconds": 16203125000
     },
     "memory": {
      "time": "2021-02-19T01:14:27Z",
      "workingSetBytes": 59080704
     },
     "rootfs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 71303168,
      "inodesUsed": 0
     },
     "logs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 0,
      "inodesUsed": 0
     }
    }
   ],
  },
  {
   "podRef": {
    "name": "logger-f76f4d988-k5n9m",
    "namespace": "default",
    "uid": "6a3c95d2-27cf-4439-927e-724f069aee89"
   },
   "startTime": "2021-02-19T01:12:34Z",
   "containers": [
    {
     "name": "testwin",
     "startTime": "2021-02-19T01:13:06Z",
     "cpu": {
      "time": "2021-02-19T01:14:27Z",
      "usageNanoCores": 190263476,
      "usageCoreNanoSeconds": 19796875000
     },
     "memory": {
      "time": "2021-02-19T01:14:27Z",
      "workingSetBytes": 66772992
     },
     "rootfs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 37748736,
      "inodesUsed": 0
     },
     "logs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 172746066,
      "inodesUsed": 0
     }
    }

Which issue(s) this PR fixes:

Fixes #99173

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Pod Log stats for windows now reports metrics

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

/sig windows

@k8s-ci-robot k8s-ci-robot added the sig/windows Categorizes an issue or PR as relevant to SIG Windows. label Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

/priority important-soon
/milestone v1.21

@k8s-ci-robot k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 19, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 19, 2021
@k8s-ci-robot k8s-ci-robot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

Reviewing the logs I am also finding the following error which is also related to the changes with the host_stats_provider

E0218 23:30:13.293729    4360 cri_stats_provider.go:407] unable to fetch pod etc hosts stats: failed to get stats CreateFile \var\lib\kubelet\pods\830b35a0-d8f3-421f-afc7-2861ace4ff20/etc-hosts: The system cannot find the file specified.

/etc/hosts file isn't used in windows:

kubernetes/pkg/kubelet/kubelet_pods.go

Lines 144 to 150 in f49385d

// Kubernetes only mounts on /etc/hosts if:
// - container is not an infrastructure (pause) container
// - container is not already mounting on /etc/hosts
// - OS is not Windows
// Kubernetes will not mount /etc/hosts if:
// - when the Pod sandbox is being created, its IP is still unknown. Hence, PodIP will not have been set.
mountEtcHostsFile := len(podIPs) > 0 && runtime.GOOS != "windows"

pushing a fix for this as well

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 19, 2021
@jingxu97
Copy link
Contributor

is that possible to add a test for it?

@ehashman ehashman added this to Needs Reviewer in SIG Node PR Triage Feb 22, 2021
@k8s-ci-robot k8s-ci-robot added area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Feb 23, 2021
@erismaster
Copy link

Hi 👋 Bug Triage here, just a reminder that code freeze is end of day, tomorrow the 9th! PRs need to be merged in before then, or they will require an approved exception to go into the 1.21 milestone.

@jsturtevant
Copy link
Contributor Author

/retest

@marosset
Copy link
Contributor

marosset commented Mar 8, 2021

/test pull-kubernetes-e2e-aks-engine-azure-windows
/test pull-kubernetes-e2e-aks-engine-windows-contianerd

@marosset
Copy link
Contributor

marosset commented Mar 8, 2021

/test pull-kubernetes-e2e-aks-engine-windows-containerd

@jsturtevant
Copy link
Contributor Author

The flakes for tests pull-kubernetes-e2e-azure-file-windows-containerd and /test pull-kubernetes-e2e-azure-disk-windows should be resolved by #95018

@marosset
Copy link
Contributor

marosset commented Mar 8, 2021

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2021
@jsturtevant
Copy link
Contributor Author

/test pull-kubernetes-node-e2e

@jsturtevant
Copy link
Contributor Author

Without these updates the kubelet logs for Windows are really noisy with error messages. Hope we can get it in for into the release. Thanks!

/assign @jingxu97 @dashpole @saad-ali @ehashman

@jingxu97
Copy link
Contributor

jingxu97 commented Mar 9, 2021

/approve

@jingxu97
Copy link
Contributor

jingxu97 commented Mar 9, 2021

/assign @Random-Liu @yujuhong

@marosset
Copy link
Contributor

marosset commented Mar 9, 2021

/assign @saad-ali @dchen1107

@dchen1107
Copy link
Member

@saad-ali will take a look soon. Thanks!

saad-ali
saad-ali approved these changes Mar 10, 2021
View reviewed changes
Copy link
Member

@saad-ali saad-ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@jsturtevant
Copy link
Contributor Author

Thanks @saad-ali @jingxu97

@dchen1107 it looks like we need an approval from Kubelet owners as well. Would you be able to take a quick look?

@mrunalp
Copy link
Contributor

mrunalp commented Mar 10, 2021

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jingxu97, jsturtevant, mrunalp, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 10, 2021
@jsturtevant
Copy link
Contributor Author

pull-kubernetes-e2e-aks-engine-azure-windows is a known test issue with an open fix: #99860

looks like pull-kubernetes-integration is unrealted and has flake open: #98606
/test pull-kubernetes-integration

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Mar 10, 2021 •
edited

@jsturtevant: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-azure-file-windows-containerd b688199f18f76cd497524ed4f6c1cac9d98abeed link /test pull-kubernetes-e2e-azure-file-windows-containerd
pull-kubernetes-e2e-aks-engine-azure-windows b688199f18f76cd497524ed4f6c1cac9d98abeed link /test pull-kubernetes-e2e-aks-engine-azure-windows

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jsturtevant
Copy link
Contributor Author

/retest

@ehashman ehashman moved this from Needs Approver to Done in SIG Node PR Triage Mar 10, 2021
@k8s-ci-robot k8s-ci-robot merged commit 4b6e3e1 into kubernetes:master Mar 10, 2021
SIG-Windows automation moved this from In Review (v1.21) to Done (v1.21) Mar 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aravindhp aravindhp aravindhp left review comments

@marosset marosset marosset left review comments

@saad-ali saad-ali saad-ali approved these changes

@jingxu97 jingxu97 Awaiting requested review from jingxu97

Assignees

@dashpole dashpole

@ehashman ehashman

@Random-Liu Random-Liu

@saad-ali saad-ali

@yujuhong yujuhong

@aravindhp aravindhp

@marosset marosset

@jingxu97 jingxu97

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. sig/windows Categorizes an issue or PR as relevant to SIG Windows. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node CI/Test Board
Triage
SIG Node CI/Test Board
Archived in project
SIG Node PR Triage
Done
Milestone
v1.21
Development

Successfully merging this pull request may close these issues.

sig-windows - Unable to fetch container log stats: failed to get fsstats