Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kubeadm]: Remove the deprecated kube-dns as an option in kubeadm #99646

Merged
merged 3 commits into from Mar 4, 2021

Conversation

rajansandeep
Copy link
Contributor

@rajansandeep rajansandeep commented Mar 2, 2021

What type of PR is this?

/kind cleanup
/kind deprecation

What this PR does / why we need it:

This PR removes the deprecated kube-dns as an option in kubeadm

Which issue(s) this PR fixes:

Fixes kubernetes/kubeadm#1943

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 2, 2021
@k8s-ci-robot k8s-ci-robot added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 2, 2021
@neolit123
Copy link
Member

thanks for the PR @rajansandeep
as long as this passes CI were on schedule for the removal.

/approve
/triage accepted
/priority important-longterm
/milestone v1.21

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Mar 2, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Mar 2, 2021
@k8s-ci-robot k8s-ci-robot removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 2, 2021
@neolit123
Copy link
Member

/remove-priority important-longterm
/priority important-soon

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Mar 2, 2021
@neolit123
Copy link
Member

neolit123 commented Mar 2, 2021

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/99646/pull-kubernetes-dependencies/1366791520585256960

@@ -191,7 +191,6 @@ github.com/blang/semver
 ## explicit
 # github.com/boltdb/bolt => github.com/boltdb/bolt v1.3.1
 # github.com/caddyserver/caddy v1.0.3 => github.com/caddyserver/caddy v1.0.3
-## explicit
 # github.com/caddyserver/caddy => github.com/caddyserver/caddy v1.0.3
 github.com/caddyserver/caddy/caddyfile
 # github.com/cenkalti/backoff => github.com/cenkalti/backoff v2.1.1+incompatible
Vendor Verify failed.
If you're seeing this locally, run the below command to fix your directories:

looks like ./hack/update-vendor.sh
might be required.

@neolit123
Copy link
Member

/cc @SataQiu @pacoxu
PTAL if you have the time for more 👀 on the change.

@@ -141,9 +141,6 @@ type DNSAddOnType string
const (
// CoreDNS add-on type
CoreDNS DNSAddOnType = "CoreDNS"

// KubeDNS add-on type
KubeDNS DNSAddOnType = "kube-dns"
Copy link
Member

@neolit123 neolit123 Mar 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to double check.

what happens if the user passes kube-dns as the type after this change:
https://github.com/kubernetes/kubernetes/blob/700047303c5414b5fe9b0d1240602cb680274ca1/cmd/kubeadm/app/apis/kubeadm/v1beta2/types.go#L93

is it a NOOP now?

if so, i think we must reflect that in the release note with an additional sentence.

The "kube-dns" type is no longer recognized if passed to ClusterConfiguration.dns.type.

should we error out if anything other than "CoreDNS" is used?

Copy link
Member

@neolit123 neolit123 Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajansandeep

should we error out if anything other than "CoreDNS" is used?

i cannot find validation for ClusterConfiguration.DNS.Type.
in EnsureDNSAddon() we just check if Type is coredns first and if not it uses kube-dns.
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/dns/dns.go#L103-L114

this validation should have been originally in:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/validation/validation.go#L59

so...we can instead do this in the release note:

kubeadm: the deprecated kube-dns is no longer supported as an option. Given kubeadm now only supports CoreDNS as the DNS addon, the value of the field "ClusterConfiguration.dns.type" is ignored.

instead of adding the validation in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I updated the release notes

Copy link
Member

@liggitt liggitt Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the value of the field "ClusterConfiguration.dns.type" is ignored

That's surprising to me... was that field validated in 1.20? If so, we should error rather than silently accept a config that explicitly requests kube-dns and previously worked and not honor it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apparently the field was never validated after the API was converted to internal type...
and during runtime the check was if type == "CoreDNS" { // install coredns } else { // install kube-dns }.

having the validation seems better to me overall, but i didn't want to introduce it now.


if we are going it introduce it, it would look like this:

add this:

// ValidateDNS validates the DNS object and collects all encountered errors
func ValidateDNS(dns *kubeadm.DNS, fldPath *field.Path) field.ErrorList {
	allErrs := field.ErrorList{}
	if dns.Type != kubeadm.CoreDNS {
		allErrs = append(allErrs, field.Invalid(fldPath, dns.Type, fmt.Sprintf("only DNS type %q is supported", kubeadm.CoreDNS)))
	}
	return allErrs
}

here https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/validation/validation.go#L493

it should be called like so:

allErrs = append(allErrs, ValidateDNS(&c.DNS, field.NewPath("dns"))...)

here:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/validation/validation.go#L60

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this means that we need to keep the kube-dns type constant and related issues open with a TODO.

Seems like you could inline it as "kube-dns" with a comment that this value is no longer supported. Not sure why you'd need to remove or adjust it in the future.

Copy link
Member

@neolit123 neolit123 Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, it can be inlined.

since i logged this ticket to remove the "type" field in v1beta3.
kubernetes/kubeadm#2398
i guess we will have to remove this validation too...EDIT: actually it should be removed with v1beta2 i guess.

@rajansandeep please let me know if you have any questions for this update.

// ValidateDNS validates the DNS object and collects all encountered errors
func ValidateDNS(dns *kubeadm.DNS, fldPath *field.Path) field.ErrorList {
	allErrs := field.ErrorList{}
	const kubeDNSType = "kube-dns"
	if dns.Type == kubeDNSType {
		allErrs = append(allErrs, field.Invalid(fldPath, dns.Type, fmt.Sprintf("DNS type %q is no longer supported", kubeDNSType)))
	}
	return allErrs
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajansandeep
we also need to change the release note to:

kubeadm: the deprecated kube-dns is no longer supported as an option. If "ClusterConfiguration.dns.type" is set to "kube-dns" kubeadm will now throw an error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the validation and changed the release notes.
Please let me know if I should squash the commits or leave it as is...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks.

@neolit123
Copy link
Member

@rajansandeep we also need to cleanup any mentions for kube-dns in the kubeadm docs at k/website 's dev-1.21 branch.

pages with instances that i found:
https://github.com/kubernetes/website/blob/6a85a072904eb73bbdbf1577a91a918f6dba14b9/content/en/docs/tasks/administer-cluster/coredns.md
https://github.com/kubernetes/website/blob/b1f370dd13a6b9e12ca12394393497c5528c7c8d/content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md
https://github.com/kubernetes/website/blob/ad35d64797ca8897e69e6aed48e4cb3315ce0508/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md#kubeadm-init-phase-addon-cmd-phase-addon
https://github.com/kubernetes/website/blob/0661d65a18315007af7493905abb533753e62298/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md
https://github.com/kubernetes/website/blob/6a85a072904eb73bbdbf1577a91a918f6dba14b9/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md#kubeadm-upgrade-guidance
https://github.com/kubernetes/website/blob/d2e7f4acab52d85cc59c396b664be3c007a4d5df/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md#dns
https://github.com/kubernetes/website/blob/42baccaf974dbfd790a3d4e14e766323ea57a47f/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md#pods-in-runcontainererror-crashloopbackoff-or-error-state
https://github.com/kubernetes/website/blob/42baccaf974dbfd790a3d4e14e766323ea57a47f/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md#coredns-or-kube-dns-is-stuck-in-the-pending-state
https://github.com/kubernetes/website/blob/391eef6bda33f1b5721d4b80140016b5f025575b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md#init-workflow-init-workflow

i think we should remove all instances that talk about kubeadm with kube-dns, including migration to coredns.

would you be able to help with a PR for that too?
if not, i can try to send the PR this week.

@neolit123
Copy link
Member

/lgtm
/approve
/hold
for the release note update mentioned here
#99646 (comment)

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 4, 2021
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2021
@neolit123
Copy link
Member

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2021
@neolit123
Copy link
Member

/lgtm
/approve
for kubeadm. thanks @rajansandeep

@neolit123
Copy link
Member

/hold cancel

@k8s-ci-robot k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 4, 2021
@neolit123
Copy link
Member

looks like ./hack/update-gofmt.sh is required

@rajansandeep
Copy link
Contributor Author

looks like ./hack/update-gofmt.sh is required

gofmt seems okay, it seems to be a flaky test fail

@rajansandeep
Copy link
Contributor Author

/test pull-kubernetes-integration

@k8s-ci-robot
Copy link
Contributor

@neolit123: GitHub didn't allow me to assign the following users: liggit.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @LiGgit
PTAL for approval of this go.mod change.
thanks.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@liggitt
Copy link
Member

liggitt commented Mar 4, 2021

go.mod change lgtm, one comment about ignoring a config that explicitly requests kube-dns

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2021
@neolit123
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2021
@liggitt
Copy link
Member

liggitt commented Mar 4, 2021

/approve
for go.mod change

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, neolit123, rajansandeep

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 4, 2021
@desponda
Copy link

desponda commented Mar 4, 2021

👋 Hello! I am from the Bug Triage team! Friendly reminder that code freeze is starting March 9th, 2021 (about 1 week from now)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/dependency Issues or PRs related to dependency changes area/kubeadm cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

deprecate and remove kube-dns support in kubeadm
6 participants