In two previous posts I’ve described how it’s possible to install a couple of applications with Applications
and ApplicationSets
, and this one is the last in a row. And here I’m going to install the same applications (VPA and Goldilocks) with helmfile, and I will tell why I think that it’s better than ArgoCD
So let’s start. Here you can find the initial config . Let’s see what we got here:
The main file is /helmfile.yaml
---
{{ readFile "releases.yaml" }}
bases:
- environments.yaml
- repositories.yaml
releases:
helmfiles:
- path: {{.Environment.Name }}/helmfile.yaml
You can see several imports here, let’s check them one by one:
# releases.yaml <- here we will define all the charts that are going to be used by helmfile. It's a templating layer, so we don't have to copy-paste a lot of yaml stuff
---
templates:
# It's supposed to be empty when nothing is installed, but I've decided to show that helmfile also can be used to manage CRDs. With this hooks, you'll be able to install CRDs with helmfile, see diffs when updating and update them while updating releases. But I need to say that I haven't checked how it's wotking on big systems, so consider this thing experimental
crd-management-hook: &crd-management-hook
hooks:
- events: ["preapply"]
showlogs: true
command: "sh"
args:
- -c
- "helm show crds {{ .Release.Chart }} --version {{ .Release.Version }}| kubectl apply -f -"
- events: ["prepare"]
showlogs: true
command: "sh"
args:
- -c
- "helm show crds {{ .Release.Chart }} --version {{ .Release.Version }} | kubectl diff -f - || true"
- events: ["postuninstall"]
showlogs: true
command: "sh"
args:
- -c
- "helm show crds {{ .Release.Chart }} --version {{ .Release.Version }} | kubectl delete -f -"
# environments.yaml
---
# Here you can define all you clusters so you can sync helmfiles using the `-e` flag
# helmfile -e cluster_test sync
environments:
cluster_test:
kubeContext: kind-kind
cluster1:
kubeContext: kind-kind
cluster2:
kubeContext: kind-kind
# repositories.yaml
---
# it's empty now, because we have nothing installed yet. But here we will add all the helm repos to avoid copy-paste again.
repositories:
Also there is a more complicated and less obvious import:
helmfiles:
- path: {{.Environment.Name }}/helmfile.yaml
It’s going to import helmfiles that are not used across all clusters. So if you want to install something to cluster-2 across, you will add it to the /cluster2/helmfile.yaml and sync the main helmfile. I will show an example later.
So we’re all set and ready to begin installing new stuff.
- Add a new repo to
repositories.yaml
repositories:
- name: fairwinds-stable
url: https://charts.fairwinds.com/stable
- Add new charts to
releases.yaml
releases:
vpa: &vpa
name: vpa
chart: fairwinds-stable/vpa
version: 1.6.0
values:
- common/{{ .Release.Name }}/values.yaml
inherit:
template: crd-management-hook # <- Here we are using the crd-management-hook, so VPA CRDs are managed outside of helm itself
goldilocks: &goldilocks
name: goldilocks
chart: fairwinds-stable/goldilocks
version: 6.5.0
And now we only need to say to which clusters we want to install those releases. So to the main helmfile.yaml I’m adding this:
releases:
- <<: *vpa
installed: true
namespace: vpa-system
createNamespace: true
And to cluster2/helmfile.yaml and cluster_test/helmfile.yaml I’m adding:
releases:
- <<: *goldilocks
installed: true
namespace: vpa-system
Let’s see how the CI could be implemented with this setup.
- If branch is not
mainhelmfile -e cluster_test applyhelmfile -e cluster1 diffhelmfile -e cluster2 diff
- If branch is
mainhelmfile -e cluster_test applyhelmfile -e cluster1 applyhelmfile -e cluster2 apply
Diffs will look like that:
Comparing release=goldilocks, chart=fairwinds-stable/goldilocks
********************
Release was not present in Helm. Diff will show entire contents as new.
********************
vpa-system, goldilocks-controller, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: goldilocks/templates/controller-clusterrole.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: goldilocks-controller
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: controller
+ rules:
+ - apiGroups:
+ - 'apps'
+ resources:
+ - '*'
+ verbs:
+ - 'get'
+ - 'list'
+ - 'watch'
+ - apiGroups:
+ - ''
+ resources:
+ - 'namespaces'
+ - 'pods'
+ verbs:
+ - 'get'
+ - 'list'
+ - 'watch'
+ - apiGroups:
+ - 'autoscaling.k8s.io'
+ resources:
+ - 'verticalpodautoscalers'
+ verbs:
+ - 'get'
+ - 'list'
+ - 'create'
+ - 'delete'
+ - 'update'
+ - apiGroups:
+ - 'argoproj.io'
+ resources:
+ - rollouts
+ verbs:
+ - 'get'
+ - 'list'
+ - 'watch'
vpa-system, goldilocks-controller, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: goldilocks/templates/controller-clusterrolebinding.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: goldilocks-controller
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: controller
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: goldilocks-controller
+ subjects:
+ - kind: ServiceAccount
+ name: goldilocks-controller
+ namespace: vpa-system
vpa-system, goldilocks-controller, Deployment (apps) has been added:
-
+ # Source: goldilocks/templates/controller-deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: goldilocks-controller
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: controller
+ spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: goldilocks
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/component: controller
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/name: goldilocks
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/component: controller
+ spec:
+ serviceAccountName: goldilocks-controller
+ securityContext:
+ {}
+ containers:
+ - name: goldilocks
+ image: "us-docker.pkg.dev/fairwinds-ops/oss/goldilocks:v4.6.2"
+ imagePullPolicy: Always
+ command:
+ - /goldilocks
+ - controller
+ - -v2
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ runAsNonRoot: true
+ runAsUser: 10324
+ resources:
+ limits:
+ cpu: 25m
+ memory: 32Mi
+ requests:
+ cpu: 25m
+ memory: 32Mi
vpa-system, goldilocks-controller, ServiceAccount (v1) has been added:
-
+ # Source: goldilocks/templates/controller-serviceaccount.yaml
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: goldilocks-controller
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: controller
vpa-system, goldilocks-dashboard, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: goldilocks/templates/dashboard-clusterrole.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: goldilocks-dashboard
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: dashboard
+ rules:
+ - apiGroups:
+ - 'autoscaling.k8s.io'
+ resources:
+ - 'verticalpodautoscalers'
+ verbs:
+ - 'get'
+ - 'list'
+ - apiGroups:
+ - 'apps'
+ resources:
+ - '*'
+ verbs:
+ - 'get'
+ - 'list'
+ - apiGroups:
+ - ''
+ resources:
+ - 'namespaces'
+ - 'pods'
+ verbs:
+ - 'get'
+ - 'list'
+ - apiGroups:
+ - 'argoproj.io'
+ resources:
+ - rollouts
+ verbs:
+ - 'get'
+ - 'list'
vpa-system, goldilocks-dashboard, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: goldilocks/templates/dashboard-clusterrolebinding.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: goldilocks-dashboard
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: dashboard
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: goldilocks-dashboard
+ subjects:
+ - kind: ServiceAccount
+ name: goldilocks-dashboard
+ namespace: vpa-system
vpa-system, goldilocks-dashboard, Deployment (apps) has been added:
-
+ # Source: goldilocks/templates/dashboard-deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: goldilocks-dashboard
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: dashboard
+ spec:
+ replicas: 2
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: goldilocks
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/component: dashboard
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/name: goldilocks
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/component: dashboard
+ spec:
+ serviceAccountName: goldilocks-dashboard
+ securityContext:
+ {}
+ containers:
+ - name: goldilocks
+ image: "us-docker.pkg.dev/fairwinds-ops/oss/goldilocks:v4.6.2"
+ imagePullPolicy: Always
+ command:
+ - /goldilocks
+ - dashboard
+ - --exclude-containers=linkerd-proxy,istio-proxy
+ - -v2
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop:
+ - ALL
+ readOnlyRootFilesystem: true
+ runAsNonRoot: true
+ runAsUser: 10324
+ ports:
+ - name: http
+ containerPort: 8080
+ protocol: TCP
+ livenessProbe:
+ httpGet:
+ path: /health
+ port: http
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: http
+ resources:
+ limits:
+ cpu: 25m
+ memory: 32Mi
+ requests:
+ cpu: 25m
+ memory: 32Mi
vpa-system, goldilocks-dashboard, Service (v1) has been added:
-
+ # Source: goldilocks/templates/dashboard-service.yaml
+ apiVersion: v1
+ kind: Service
+ metadata:
+ name: goldilocks-dashboard
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: dashboard
+ spec:
+ type: ClusterIP
+ ports:
+ - port: 80
+ targetPort: http
+ protocol: TCP
+ name: http
+ selector:
+ app.kubernetes.io/name: goldilocks
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/component: dashboard
vpa-system, goldilocks-dashboard, ServiceAccount (v1) has been added:
-
+ # Source: goldilocks/templates/dashboard-serviceaccount.yaml
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+ name: goldilocks-dashboard
+ labels:
+ app.kubernetes.io/name: goldilocks
+ helm.sh/chart: goldilocks-6.5.0
+ app.kubernetes.io/instance: goldilocks
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: dashboard
hook[prepare] logs | diff -u -N /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/LIVE-4051758900/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalercheckpoints.autoscaling.k8s.io /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/MERGED-3664876659/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalercheckpoints.autoscaling.k8s.io
hook[prepare] logs | --- /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/LIVE-4051758900/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalercheckpoints.autoscaling.k8s.io 2023-02-17 13:15:29
hook[prepare] logs | +++ /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/MERGED-3664876659/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalercheckpoints.autoscaling.k8s.io 2023-02-17 13:15:29
hook[prepare] logs | @@ -0,0 +1,216 @@
hook[prepare] logs | +apiVersion: apiextensions.k8s.io/v1
hook[prepare] logs | +kind: CustomResourceDefinition
hook[prepare] logs | +metadata:
hook[prepare] logs | + annotations:
hook[prepare] logs | + api-approved.kubernetes.io: https://github.com/kubernetes/kubernetes/pull/63797
hook[prepare] logs | + controller-gen.kubebuilder.io/version: v0.4.0
hook[prepare] logs | + creationTimestamp: "2023-02-17T12:15:29Z"
hook[prepare] logs | + generation: 1
hook[prepare] logs | + name: verticalpodautoscalercheckpoints.autoscaling.k8s.io
hook[prepare] logs | + uid: bd3d2d46-b0c1-48c4-bc72-968cebfd7640
hook[prepare] logs | +spec:
hook[prepare] logs | + conversion:
hook[prepare] logs | + strategy: None
hook[prepare] logs | + group: autoscaling.k8s.io
hook[prepare] logs | + names:
hook[prepare] logs | + kind: VerticalPodAutoscalerCheckpoint
hook[prepare] logs | + listKind: VerticalPodAutoscalerCheckpointList
hook[prepare] logs | + plural: verticalpodautoscalercheckpoints
hook[prepare] logs | + shortNames:
hook[prepare] logs | + - vpacheckpoint
hook[prepare] logs | + singular: verticalpodautoscalercheckpoint
hook[prepare] logs | + scope: Namespaced
hook[prepare] logs | + versions:
hook[prepare] logs | + - name: v1
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscalerCheckpoint is the checkpoint of the internal
hook[prepare] logs | + state of VPA that is used for recovery after recommender's restart.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the checkpoint. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the checkpointed container.
hook[prepare] logs | + type: string
hook[prepare] logs | + vpaObjectName:
hook[prepare] logs | + description: Name of the VPA object that stored VerticalPodAutoscalerCheckpoint
hook[prepare] logs | + object.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Data of the checkpoint.
hook[prepare] logs | + properties:
hook[prepare] logs | + cpuHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of CPU.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + type: object
hook[prepare] logs | + x-kubernetes-preserve-unknown-fields: true
hook[prepare] logs | + referenceTimestamp:
hook[prepare] logs | + description: Reference timestamp for samples collected within
hook[prepare] logs | + this histogram.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + totalWeight:
hook[prepare] logs | + description: Sum of samples to be used as denominator for weights
hook[prepare] logs | + from BucketWeights.
hook[prepare] logs | + type: number
hook[prepare] logs | + type: object
hook[prepare] logs | + firstSampleStart:
hook[prepare] logs | + description: Timestamp of the fist sample from the histograms.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + lastSampleStart:
hook[prepare] logs | + description: Timestamp of the last sample from the histograms.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + lastUpdateTime:
hook[prepare] logs | + description: The time when the status was last refreshed.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + memoryHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of memory.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + type: object
hook[prepare] logs | + x-kubernetes-preserve-unknown-fields: true
hook[prepare] logs | + referenceTimestamp:
hook[prepare] logs | + description: Reference timestamp for samples collected within
hook[prepare] logs | + this histogram.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + totalWeight:
hook[prepare] logs | + description: Sum of samples to be used as denominator for weights
hook[prepare] logs | + from BucketWeights.
hook[prepare] logs | + type: number
hook[prepare] logs | + type: object
hook[prepare] logs | + totalSamplesCount:
hook[prepare] logs | + description: Total number of samples in the histograms.
hook[prepare] logs | + type: integer
hook[prepare] logs | + version:
hook[prepare] logs | + description: Version of the format of the stored data.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: true
hook[prepare] logs | + - name: v1beta2
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscalerCheckpoint is the checkpoint of the internal
hook[prepare] logs | + state of VPA that is used for recovery after recommender's restart.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the checkpoint. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the checkpointed container.
hook[prepare] logs | + type: string
hook[prepare] logs | + vpaObjectName:
hook[prepare] logs | + description: Name of the VPA object that stored VerticalPodAutoscalerCheckpoint
hook[prepare] logs | + object.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Data of the checkpoint.
hook[prepare] logs | + properties:
hook[prepare] logs | + cpuHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of CPU.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + type: object
hook[prepare] logs | + x-kubernetes-preserve-unknown-fields: true
hook[prepare] logs | + referenceTimestamp:
hook[prepare] logs | + description: Reference timestamp for samples collected within
hook[prepare] logs | + this histogram.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + totalWeight:
hook[prepare] logs | + description: Sum of samples to be used as denominator for weights
hook[prepare] logs | + from BucketWeights.
hook[prepare] logs | + type: number
hook[prepare] logs | + type: object
hook[prepare] logs | + firstSampleStart:
hook[prepare] logs | + description: Timestamp of the fist sample from the histograms.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + lastSampleStart:
hook[prepare] logs | + description: Timestamp of the last sample from the histograms.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + lastUpdateTime:
hook[prepare] logs | + description: The time when the status was last refreshed.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + memoryHistogram:
hook[prepare] logs | + description: Checkpoint of histogram for consumption of memory.
hook[prepare] logs | + properties:
hook[prepare] logs | + bucketWeights:
hook[prepare] logs | + description: Map from bucket index to bucket weight.
hook[prepare] logs | + type: object
hook[prepare] logs | + x-kubernetes-preserve-unknown-fields: true
hook[prepare] logs | + referenceTimestamp:
hook[prepare] logs | + description: Reference timestamp for samples collected within
hook[prepare] logs | + this histogram.
hook[prepare] logs | + format: date-time
hook[prepare] logs | + nullable: true
hook[prepare] logs | + type: string
hook[prepare] logs | + totalWeight:
hook[prepare] logs | + description: Sum of samples to be used as denominator for weights
hook[prepare] logs | + from BucketWeights.
hook[prepare] logs | + type: number
hook[prepare] logs | + type: object
hook[prepare] logs | + totalSamplesCount:
hook[prepare] logs | + description: Total number of samples in the histograms.
hook[prepare] logs | + type: integer
hook[prepare] logs | + version:
hook[prepare] logs | + description: Version of the format of the stored data.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: false
hook[prepare] logs | +status:
hook[prepare] logs | + acceptedNames:
hook[prepare] logs | + kind: ""
hook[prepare] logs | + plural: ""
hook[prepare] logs | + conditions: null
hook[prepare] logs | + storedVersions:
hook[prepare] logs | + - v1
hook[prepare] logs | diff -u -N /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/LIVE-4051758900/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalers.autoscaling.k8s.io /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/MERGED-3664876659/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalers.autoscaling.k8s.io
hook[prepare] logs | --- /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/LIVE-4051758900/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalers.autoscaling.k8s.io 2023-02-17 13:15:29
hook[prepare] logs | +++ /var/folders/w1/27ptcr29547f0g8732kmffwm0000gn/T/MERGED-3664876659/apiextensions.k8s.io.v1.CustomResourceDefinition..verticalpodautoscalers.autoscaling.k8s.io 2023-02-17 13:15:29
hook[prepare] logs | @@ -0,0 +1,550 @@
hook[prepare] logs | +apiVersion: apiextensions.k8s.io/v1
hook[prepare] logs | +kind: CustomResourceDefinition
hook[prepare] logs | +metadata:
hook[prepare] logs | + annotations:
hook[prepare] logs | + api-approved.kubernetes.io: https://github.com/kubernetes/kubernetes/pull/63797
hook[prepare] logs | + controller-gen.kubebuilder.io/version: v0.4.0
hook[prepare] logs | + creationTimestamp: "2023-02-17T12:15:29Z"
hook[prepare] logs | + generation: 1
hook[prepare] logs | + name: verticalpodautoscalers.autoscaling.k8s.io
hook[prepare] logs | + uid: 91c297f7-3d26-43f3-bce8-14885ca00c10
hook[prepare] logs | +spec:
hook[prepare] logs | + conversion:
hook[prepare] logs | + strategy: None
hook[prepare] logs | + group: autoscaling.k8s.io
hook[prepare] logs | + names:
hook[prepare] logs | + kind: VerticalPodAutoscaler
hook[prepare] logs | + listKind: VerticalPodAutoscalerList
hook[prepare] logs | + plural: verticalpodautoscalers
hook[prepare] logs | + shortNames:
hook[prepare] logs | + - vpa
hook[prepare] logs | + singular: verticalpodautoscaler
hook[prepare] logs | + scope: Namespaced
hook[prepare] logs | + versions:
hook[prepare] logs | + - additionalPrinterColumns:
hook[prepare] logs | + - jsonPath: .spec.updatePolicy.updateMode
hook[prepare] logs | + name: Mode
hook[prepare] logs | + type: string
hook[prepare] logs | + - jsonPath: .status.recommendation.containerRecommendations[0].target.cpu
hook[prepare] logs | + name: CPU
hook[prepare] logs | + type: string
hook[prepare] logs | + - jsonPath: .status.recommendation.containerRecommendations[0].target.memory
hook[prepare] logs | + name: Mem
hook[prepare] logs | + type: string
hook[prepare] logs | + - jsonPath: .status.conditions[?(@.type=='RecommendationProvided')].status
hook[prepare] logs | + name: Provided
hook[prepare] logs | + type: string
hook[prepare] logs | + - jsonPath: .metadata.creationTimestamp
hook[prepare] logs | + name: Age
hook[prepare] logs | + type: date
hook[prepare] logs | + name: v1
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscaler is the configuration for a vertical pod
hook[prepare] logs | + autoscaler, which automatically manages pod resources based on historical
hook[prepare] logs | + and real time resource utilization.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the behavior of the autoscaler. More info:
hook[prepare] logs | + https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + recommenders:
hook[prepare] logs | + description: Recommender responsible for generating recommendation
hook[prepare] logs | + for this object. List should be empty (then the default recommender
hook[prepare] logs | + will generate the recommendation) or contain exactly one recommender.
hook[prepare] logs | + items:
hook[prepare] logs | + description: VerticalPodAutoscalerRecommenderSelector points to
hook[prepare] logs | + a specific Vertical Pod Autoscaler recommender. In the future
hook[prepare] logs | + it might pass parameters to the recommender.
hook[prepare] logs | + properties:
hook[prepare] logs | + name:
hook[prepare] logs | + description: Name of the recommender responsible for generating
hook[prepare] logs | + recommendation for this object.
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - name
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + resourcePolicy:
hook[prepare] logs | + description: Controls how the autoscaler computes recommended resources.
hook[prepare] logs | + The resource policy may be used to set constraints on the recommendations
hook[prepare] logs | + for individual containers. If not specified, the autoscaler computes
hook[prepare] logs | + recommended resources for all containers in the pod, without additional
hook[prepare] logs | + constraints.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerPolicies:
hook[prepare] logs | + description: Per-container resource policies.
hook[prepare] logs | + items:
hook[prepare] logs | + description: ContainerResourcePolicy controls how autoscaler
hook[prepare] logs | + computes the recommended resources for a specific container.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container or DefaultContainerResourcePolicy,
hook[prepare] logs | + in which case the policy is used by the containers that
hook[prepare] logs | + don't have their own policy specified.
hook[prepare] logs | + type: string
hook[prepare] logs | + controlledResources:
hook[prepare] logs | + description: Specifies the type of recommendations that
hook[prepare] logs | + will be computed (and possibly applied) by VPA. If not
hook[prepare] logs | + specified, the default of [ResourceCPU, ResourceMemory]
hook[prepare] logs | + will be used.
hook[prepare] logs | + items:
hook[prepare] logs | + description: ResourceName is the name identifying various
hook[prepare] logs | + resources in a ResourceList.
hook[prepare] logs | + type: string
hook[prepare] logs | + type: array
hook[prepare] logs | + controlledValues:
hook[prepare] logs | + description: Specifies which resource values should be controlled.
hook[prepare] logs | + The default is "RequestsAndLimits".
hook[prepare] logs | + enum:
hook[prepare] logs | + - RequestsAndLimits
hook[prepare] logs | + - RequestsOnly
hook[prepare] logs | + type: string
hook[prepare] logs | + maxAllowed:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Specifies the maximum amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no maximum.
hook[prepare] logs | + type: object
hook[prepare] logs | + minAllowed:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Specifies the minimal amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no minimum.
hook[prepare] logs | + type: object
hook[prepare] logs | + mode:
hook[prepare] logs | + description: Whether autoscaler is enabled for the container.
hook[prepare] logs | + The default is "Auto".
hook[prepare] logs | + enum:
hook[prepare] logs | + - Auto
hook[prepare] logs | + - "Off"
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + targetRef:
hook[prepare] logs | + description: TargetRef points to the controller managing the set of
hook[prepare] logs | + pods for the autoscaler to control - e.g. Deployment, StatefulSet.
hook[prepare] logs | + VerticalPodAutoscaler can be targeted at controller implementing
hook[prepare] logs | + scale subresource (the pod set is retrieved from the controller's
hook[prepare] logs | + ScaleStatus) or some well known controllers (e.g. for DaemonSet
hook[prepare] logs | + the pod set is read from the controller's spec). If VerticalPodAutoscaler
hook[prepare] logs | + cannot use specified target it will report ConfigUnsupported condition.
hook[prepare] logs | + Note that VerticalPodAutoscaler does not require full implementation
hook[prepare] logs | + of scale subresource - it will not use it to modify the replica
hook[prepare] logs | + count. The only thing retrieved is a label selector matching pods
hook[prepare] logs | + grouped by the target resource.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: API version of the referent
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"'
hook[prepare] logs | + type: string
hook[prepare] logs | + name:
hook[prepare] logs | + description: 'Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names'
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - kind
hook[prepare] logs | + - name
hook[prepare] logs | + type: object
hook[prepare] logs | + updatePolicy:
hook[prepare] logs | + description: Describes the rules on how changes are applied to the
hook[prepare] logs | + pods. If not specified, all fields in the `PodUpdatePolicy` are
hook[prepare] logs | + set to their default values.
hook[prepare] logs | + properties:
hook[prepare] logs | + minReplicas:
hook[prepare] logs | + description: Minimal number of replicas which need to be alive
hook[prepare] logs | + for Updater to attempt pod eviction (pending other checks like
hook[prepare] logs | + PDB). Only positive values are allowed. Overrides global '--min-replicas'
hook[prepare] logs | + flag.
hook[prepare] logs | + format: int32
hook[prepare] logs | + type: integer
hook[prepare] logs | + updateMode:
hook[prepare] logs | + description: Controls when autoscaler applies changes to the pod
hook[prepare] logs | + resources. The default is 'Auto'.
hook[prepare] logs | + enum:
hook[prepare] logs | + - "Off"
hook[prepare] logs | + - Initial
hook[prepare] logs | + - Recreate
hook[prepare] logs | + - Auto
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - targetRef
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Current information about the autoscaler.
hook[prepare] logs | + properties:
hook[prepare] logs | + conditions:
hook[prepare] logs | + description: Conditions is the set of conditions required for this
hook[prepare] logs | + autoscaler to scale its target, and indicates whether or not those
hook[prepare] logs | + conditions are met.
hook[prepare] logs | + items:
hook[prepare] logs | + description: VerticalPodAutoscalerCondition describes the state
hook[prepare] logs | + of a VerticalPodAutoscaler at a certain point.
hook[prepare] logs | + properties:
hook[prepare] logs | + lastTransitionTime:
hook[prepare] logs | + description: lastTransitionTime is the last time the condition
hook[prepare] logs | + transitioned from one status to another
hook[prepare] logs | + format: date-time
hook[prepare] logs | + type: string
hook[prepare] logs | + message:
hook[prepare] logs | + description: message is a human-readable explanation containing
hook[prepare] logs | + details about the transition
hook[prepare] logs | + type: string
hook[prepare] logs | + reason:
hook[prepare] logs | + description: reason is the reason for the condition's last transition.
hook[prepare] logs | + type: string
hook[prepare] logs | + status:
hook[prepare] logs | + description: status is the status of the condition (True, False,
hook[prepare] logs | + Unknown)
hook[prepare] logs | + type: string
hook[prepare] logs | + type:
hook[prepare] logs | + description: type describes the current condition
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - status
hook[prepare] logs | + - type
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + recommendation:
hook[prepare] logs | + description: The most recently computed amount of resources recommended
hook[prepare] logs | + by the autoscaler for the controlled pods.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerRecommendations:
hook[prepare] logs | + description: Resources recommended by the autoscaler for each
hook[prepare] logs | + container.
hook[prepare] logs | + items:
hook[prepare] logs | + description: RecommendedContainerResources is the recommendation
hook[prepare] logs | + of resources computed by autoscaler for a specific container.
hook[prepare] logs | + Respects the container resource policy if present in the spec.
hook[prepare] logs | + In particular the recommendation is not produced for containers
hook[prepare] logs | + with `ContainerScalingMode` set to 'Off'.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container.
hook[prepare] logs | + type: string
hook[prepare] logs | + lowerBound:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Minimum recommended amount of resources. Observes
hook[prepare] logs | + ContainerResourcePolicy. This amount is not guaranteed
hook[prepare] logs | + to be sufficient for the application to operate in a stable
hook[prepare] logs | + way, however running with less resources is likely to
hook[prepare] logs | + have significant impact on performance/availability.
hook[prepare] logs | + type: object
hook[prepare] logs | + target:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Recommended amount of resources. Observes ContainerResourcePolicy.
hook[prepare] logs | + type: object
hook[prepare] logs | + uncappedTarget:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: The most recent recommended resources target
hook[prepare] logs | + computed by the autoscaler for the controlled pods, based
hook[prepare] logs | + only on actual resource usage, not taking into account
hook[prepare] logs | + the ContainerResourcePolicy. May differ from the Recommendation
hook[prepare] logs | + if the actual resource usage causes the target to violate
hook[prepare] logs | + the ContainerResourcePolicy (lower than MinAllowed or
hook[prepare] logs | + higher that MaxAllowed). Used only as status indication,
hook[prepare] logs | + will not affect actual resource assignment.
hook[prepare] logs | + type: object
hook[prepare] logs | + upperBound:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Maximum recommended amount of resources. Observes
hook[prepare] logs | + ContainerResourcePolicy. Any resources allocated beyond
hook[prepare] logs | + this value are likely wasted. This value may be larger
hook[prepare] logs | + than the maximum amount of application is actually capable
hook[prepare] logs | + of consuming.
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - target
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - spec
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: true
hook[prepare] logs | + subresources: {}
hook[prepare] logs | + - name: v1beta2
hook[prepare] logs | + schema:
hook[prepare] logs | + openAPIV3Schema:
hook[prepare] logs | + description: VerticalPodAutoscaler is the configuration for a vertical pod
hook[prepare] logs | + autoscaler, which automatically manages pod resources based on historical
hook[prepare] logs | + and real time resource utilization.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: 'APIVersion defines the versioned schema of this representation
hook[prepare] logs | + of an object. Servers should convert recognized schemas to the latest
hook[prepare] logs | + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind is a string value representing the REST resource this
hook[prepare] logs | + object represents. Servers may infer this from the endpoint the client
hook[prepare] logs | + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
hook[prepare] logs | + type: string
hook[prepare] logs | + metadata:
hook[prepare] logs | + type: object
hook[prepare] logs | + spec:
hook[prepare] logs | + description: 'Specification of the behavior of the autoscaler. More info:
hook[prepare] logs | + https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.'
hook[prepare] logs | + properties:
hook[prepare] logs | + resourcePolicy:
hook[prepare] logs | + description: Controls how the autoscaler computes recommended resources.
hook[prepare] logs | + The resource policy may be used to set constraints on the recommendations
hook[prepare] logs | + for individual containers. If not specified, the autoscaler computes
hook[prepare] logs | + recommended resources for all containers in the pod, without additional
hook[prepare] logs | + constraints.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerPolicies:
hook[prepare] logs | + description: Per-container resource policies.
hook[prepare] logs | + items:
hook[prepare] logs | + description: ContainerResourcePolicy controls how autoscaler
hook[prepare] logs | + computes the recommended resources for a specific container.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container or DefaultContainerResourcePolicy,
hook[prepare] logs | + in which case the policy is used by the containers that
hook[prepare] logs | + don't have their own policy specified.
hook[prepare] logs | + type: string
hook[prepare] logs | + maxAllowed:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Specifies the maximum amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no maximum.
hook[prepare] logs | + type: object
hook[prepare] logs | + minAllowed:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Specifies the minimal amount of resources that
hook[prepare] logs | + will be recommended for the container. The default is
hook[prepare] logs | + no minimum.
hook[prepare] logs | + type: object
hook[prepare] logs | + mode:
hook[prepare] logs | + description: Whether autoscaler is enabled for the container.
hook[prepare] logs | + The default is "Auto".
hook[prepare] logs | + enum:
hook[prepare] logs | + - Auto
hook[prepare] logs | + - "Off"
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + targetRef:
hook[prepare] logs | + description: TargetRef points to the controller managing the set of
hook[prepare] logs | + pods for the autoscaler to control - e.g. Deployment, StatefulSet.
hook[prepare] logs | + VerticalPodAutoscaler can be targeted at controller implementing
hook[prepare] logs | + scale subresource (the pod set is retrieved from the controller's
hook[prepare] logs | + ScaleStatus) or some well known controllers (e.g. for DaemonSet
hook[prepare] logs | + the pod set is read from the controller's spec). If VerticalPodAutoscaler
hook[prepare] logs | + cannot use specified target it will report ConfigUnsupported condition.
hook[prepare] logs | + Note that VerticalPodAutoscaler does not require full implementation
hook[prepare] logs | + of scale subresource - it will not use it to modify the replica
hook[prepare] logs | + count. The only thing retrieved is a label selector matching pods
hook[prepare] logs | + grouped by the target resource.
hook[prepare] logs | + properties:
hook[prepare] logs | + apiVersion:
hook[prepare] logs | + description: API version of the referent
hook[prepare] logs | + type: string
hook[prepare] logs | + kind:
hook[prepare] logs | + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"'
hook[prepare] logs | + type: string
hook[prepare] logs | + name:
hook[prepare] logs | + description: 'Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names'
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - kind
hook[prepare] logs | + - name
hook[prepare] logs | + type: object
hook[prepare] logs | + updatePolicy:
hook[prepare] logs | + description: Describes the rules on how changes are applied to the
hook[prepare] logs | + pods. If not specified, all fields in the `PodUpdatePolicy` are
hook[prepare] logs | + set to their default values.
hook[prepare] logs | + properties:
hook[prepare] logs | + updateMode:
hook[prepare] logs | + description: Controls when autoscaler applies changes to the pod
hook[prepare] logs | + resources. The default is 'Auto'.
hook[prepare] logs | + enum:
hook[prepare] logs | + - "Off"
hook[prepare] logs | + - Initial
hook[prepare] logs | + - Recreate
hook[prepare] logs | + - Auto
hook[prepare] logs | + type: string
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - targetRef
hook[prepare] logs | + type: object
hook[prepare] logs | + status:
hook[prepare] logs | + description: Current information about the autoscaler.
hook[prepare] logs | + properties:
hook[prepare] logs | + conditions:
hook[prepare] logs | + description: Conditions is the set of conditions required for this
hook[prepare] logs | + autoscaler to scale its target, and indicates whether or not those
hook[prepare] logs | + conditions are met.
hook[prepare] logs | + items:
hook[prepare] logs | + description: VerticalPodAutoscalerCondition describes the state
hook[prepare] logs | + of a VerticalPodAutoscaler at a certain point.
hook[prepare] logs | + properties:
hook[prepare] logs | + lastTransitionTime:
hook[prepare] logs | + description: lastTransitionTime is the last time the condition
hook[prepare] logs | + transitioned from one status to another
hook[prepare] logs | + format: date-time
hook[prepare] logs | + type: string
hook[prepare] logs | + message:
hook[prepare] logs | + description: message is a human-readable explanation containing
hook[prepare] logs | + details about the transition
hook[prepare] logs | + type: string
hook[prepare] logs | + reason:
hook[prepare] logs | + description: reason is the reason for the condition's last transition.
hook[prepare] logs | + type: string
hook[prepare] logs | + status:
hook[prepare] logs | + description: status is the status of the condition (True, False,
hook[prepare] logs | + Unknown)
hook[prepare] logs | + type: string
hook[prepare] logs | + type:
hook[prepare] logs | + description: type describes the current condition
hook[prepare] logs | + type: string
hook[prepare] logs | + required:
hook[prepare] logs | + - status
hook[prepare] logs | + - type
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + recommendation:
hook[prepare] logs | + description: The most recently computed amount of resources recommended
hook[prepare] logs | + by the autoscaler for the controlled pods.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerRecommendations:
hook[prepare] logs | + description: Resources recommended by the autoscaler for each
hook[prepare] logs | + container.
hook[prepare] logs | + items:
hook[prepare] logs | + description: RecommendedContainerResources is the recommendation
hook[prepare] logs | + of resources computed by autoscaler for a specific container.
hook[prepare] logs | + Respects the container resource policy if present in the spec.
hook[prepare] logs | + In particular the recommendation is not produced for containers
hook[prepare] logs | + with `ContainerScalingMode` set to 'Off'.
hook[prepare] logs | + properties:
hook[prepare] logs | + containerName:
hook[prepare] logs | + description: Name of the container.
hook[prepare] logs | + type: string
hook[prepare] logs | + lowerBound:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Minimum recommended amount of resources. Observes
hook[prepare] logs | + ContainerResourcePolicy. This amount is not guaranteed
hook[prepare] logs | + to be sufficient for the application to operate in a stable
hook[prepare] logs | + way, however running with less resources is likely to
hook[prepare] logs | + have significant impact on performance/availability.
hook[prepare] logs | + type: object
hook[prepare] logs | + target:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Recommended amount of resources. Observes ContainerResourcePolicy.
hook[prepare] logs | + type: object
hook[prepare] logs | + uncappedTarget:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: The most recent recommended resources target
hook[prepare] logs | + computed by the autoscaler for the controlled pods, based
hook[prepare] logs | + only on actual resource usage, not taking into account
hook[prepare] logs | + the ContainerResourcePolicy. May differ from the Recommendation
hook[prepare] logs | + if the actual resource usage causes the target to violate
hook[prepare] logs | + the ContainerResourcePolicy (lower than MinAllowed or
hook[prepare] logs | + higher that MaxAllowed). Used only as status indication,
hook[prepare] logs | + will not affect actual resource assignment.
hook[prepare] logs | + type: object
hook[prepare] logs | + upperBound:
hook[prepare] logs | + additionalProperties:
hook[prepare] logs | + anyOf:
hook[prepare] logs | + - type: integer
hook[prepare] logs | + - type: string
hook[prepare] logs | + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
hook[prepare] logs | + x-kubernetes-int-or-string: true
hook[prepare] logs | + description: Maximum recommended amount of resources. Observes
hook[prepare] logs | + ContainerResourcePolicy. Any resources allocated beyond
hook[prepare] logs | + this value are likely wasted. This value may be larger
hook[prepare] logs | + than the maximum amount of application is actually capable
hook[prepare] logs | + of consuming.
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - target
hook[prepare] logs | + type: object
hook[prepare] logs | + type: array
hook[prepare] logs | + type: object
hook[prepare] logs | + type: object
hook[prepare] logs | + required:
hook[prepare] logs | + - spec
hook[prepare] logs | + type: object
hook[prepare] logs | + served: true
hook[prepare] logs | + storage: false
hook[prepare] logs | +status:
hook[prepare] logs | + acceptedNames:
hook[prepare] logs | + kind: ""
hook[prepare] logs | + plural: ""
hook[prepare] logs | + conditions: null
hook[prepare] logs | + storedVersions:
hook[prepare] logs | + - v1
hook[prepare] logs |
Comparing release=vpa, chart=fairwinds-stable/vpa
********************
Release was not present in Helm. Diff will show entire contents as new.
********************
vpa-system, vpa-actor, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-actor
+ rules:
+ - apiGroups:
+ - ""
+ resources:
+ - pods
+ - nodes
+ - limitranges
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - apiGroups:
+ - "poc.autoscaling.k8s.io"
+ resources:
+ - verticalpodautoscalers
+ verbs:
+ - get
+ - list
+ - watch
+ - patch
+ - apiGroups:
+ - "autoscaling.k8s.io"
+ resources:
+ - verticalpodautoscalers
+ verbs:
+ - get
+ - list
+ - watch
+ - patch
vpa-system, vpa-actor, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterrolebindings.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: vpa-actor
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: vpa-actor
+ subjects:
+ - kind: ServiceAccount
+ name: vpa-recommender
+ namespace: vpa-system
vpa-system, vpa-checkpoint-actor, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-checkpoint-actor
+ rules:
+ - apiGroups:
+ - "poc.autoscaling.k8s.io"
+ resources:
+ - verticalpodautoscalercheckpoints
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - patch
+ - delete
+ - apiGroups:
+ - "autoscaling.k8s.io"
+ resources:
+ - verticalpodautoscalercheckpoints
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - patch
+ - delete
+ - apiGroups:
+ - ""
+ resources:
+ - namespaces
+ verbs:
+ - get
+ - list
vpa-system, vpa-checkpoint-actor, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterrolebindings.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: vpa-checkpoint-actor
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: vpa-checkpoint-actor
+ subjects:
+ - kind: ServiceAccount
+ name: vpa-recommender
+ namespace: vpa-system
vpa-system, vpa-evictioner, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-evictioner
+ rules:
+ - apiGroups:
+ - "apps"
+ - "extensions"
+ resources:
+ - replicasets
+ verbs:
+ - get
+ - apiGroups:
+ - ""
+ resources:
+ - pods/eviction
+ verbs:
+ - create
vpa-system, vpa-metrics-reader, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-metrics-reader
+ rules:
+ - apiGroups:
+ - "metrics.k8s.io"
+ resources:
+ - pods
+ verbs:
+ - get
+ - list
vpa-system, vpa-metrics-reader, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterrolebindings.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: vpa-metrics-reader
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: vpa-metrics-reader
+ subjects:
+ - kind: ServiceAccount
+ name: vpa-recommender
+ namespace: vpa-system
vpa-system, vpa-recommender, Deployment (apps) has been added:
-
+ # Source: vpa/templates/recommender-deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: vpa-recommender
+ labels:
+ app.kubernetes.io/component: recommender
+ helm.sh/chart: vpa-1.6.0
+ app.kubernetes.io/name: vpa
+ app.kubernetes.io/instance: vpa
+ app.kubernetes.io/version: "0.11.0"
+ app.kubernetes.io/managed-by: Helm
+ spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/component: recommender
+ app.kubernetes.io/name: vpa
+ app.kubernetes.io/instance: vpa
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/component: recommender
+ app.kubernetes.io/name: vpa
+ app.kubernetes.io/instance: vpa
+ spec:
+ serviceAccountName: vpa-recommender
+ securityContext:
+ runAsNonRoot: true
+ runAsUser: 65534
+ containers:
+ - name: vpa
+ securityContext:
+ {}
+ image: "k8s.gcr.io/autoscaling/vpa-recommender:0.11.0"
+ imagePullPolicy: Always
+ args:
+ - --pod-recommendation-min-cpu-millicores=15
+ - --pod-recommendation-min-memory-mb=100
+ - --v=4
+ livenessProbe:
+ failureThreshold: 6
+ httpGet:
+ path: /health-check
+ port: metrics
+ scheme: HTTP
+ periodSeconds: 5
+ successThreshold: 1
+ timeoutSeconds: 3
+ readinessProbe:
+ failureThreshold: 120
+ httpGet:
+ path: /health-check
+ port: metrics
+ scheme: HTTP
+ periodSeconds: 5
+ successThreshold: 1
+ timeoutSeconds: 3
+ ports:
+ - name: metrics
+ containerPort: 8942
+ protocol: TCP
+ resources:
+ limits:
+ cpu: 200m
+ memory: 1000Mi
+ requests:
+ cpu: 50m
+ memory: 500Mi
vpa-system, vpa-recommender, ServiceAccount (v1) has been added:
-
+ # Source: vpa/templates/recommender-service-account.yaml
+ apiVersion: v1
+ kind: ServiceAccount
+ automountServiceAccountToken: true
+ metadata:
+ name: vpa-recommender
+ labels:
+ helm.sh/chart: vpa-1.6.0
+ app.kubernetes.io/name: vpa
+ app.kubernetes.io/instance: vpa
+ app.kubernetes.io/version: "0.11.0"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: recommender
vpa-system, vpa-status-reader, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-status-reader
+ rules:
+ - apiGroups:
+ - "coordination.k8s.io"
+ resources:
+ - leases
+ verbs:
+ - get
+ - list
+ - watch
vpa-system, vpa-target-reader, ClusterRole (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterroles.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRole
+ metadata:
+ name: vpa-target-reader
+ rules:
+ - apiGroups:
+ - '*'
+ resources:
+ - '*/scale'
+ verbs:
+ - get
+ - watch
+ - apiGroups:
+ - ""
+ resources:
+ - replicationcontrollers
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - apps
+ resources:
+ - daemonsets
+ - deployments
+ - replicasets
+ - statefulsets
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - batch
+ resources:
+ - jobs
+ - cronjobs
+ verbs:
+ - get
+ - list
+ - watch
vpa-system, vpa-target-reader-binding, ClusterRoleBinding (rbac.authorization.k8s.io) has been added:
-
+ # Source: vpa/templates/clusterrolebindings.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+ name: vpa-target-reader-binding
+ roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: vpa-target-reader
+ subjects:
+ - kind: ServiceAccount
+ name: vpa-recommender
+ namespace: vpa-system
vpa-system, vpa-updater, ServiceAccount (v1) has been added:
-
+ # Source: vpa/templates/updater-service-account.yaml
+ apiVersion: v1
+ kind: ServiceAccount
+ automountServiceAccountToken: true
+ metadata:
+ name: vpa-updater
+ labels:
+ helm.sh/chart: vpa-1.6.0
+ app.kubernetes.io/name: vpa
+ app.kubernetes.io/instance: vpa
+ app.kubernetes.io/version: "0.11.0"
+ app.kubernetes.io/managed-by: Helm
+ app.kubernetes.io/component: updater
Yeah, it’s huge, but you can see everything that’s going to happen. So I’d say it’s good.
Conclusion
It’s a short article, because I think the whole setup is super easy, CI is easy too. You still have a full GitOps (or almost full) but you also have control. I love this setup and would like to use it for my infrastructure.
Why do I think it’s better that ArgoCD?
With ArgoCD I either have a lot of yaml to install things, or I have complicated setups with ApplicationSets that are most probably very special and won’t be reused in other companies. I need to care about how ArgoCD will handle a lot of applications that are added there only for diffing. I need additional applications installed in my clusters not only as a part of infrastructure itself, but also as a service that I’m providing other teams with. Because I want to manage applications that are being developed by other teams with Argo, so I’m mixing a lot of different kinds of applications here.
Helmfile lets me separate infra from applications. ArgoCD can be only provided as a service, and other teams can use, because it’s making k8s easier for those who don’t need to understand it so deeply. Also, helmfile lets me use helm-secrets to encrypt values. I can do it with Argo too, but then I need to either have a custom ArgoCD image, or support a CMP plugin, that will handle SOPS.
You can find an example of PR here: https://git.badhouseplants.net/allanger/helmfile-vs-argo/pulls/1/files
When helmfile is not GitOps? To uninstall a helm release, you need to add
isntalled: falseto it. If you just remove a release from helmfile.yaml, it isn’t going to be removed. So in such cases it’s not GitOps. You can write a hook, that is comparing a previous state of your helmfile to the current one and doing a cleanup, then it’s again fully GitOps. But I prefer removing things manually, so to me, it’s not a problem. Removing stuff is something that I think should be mostly done by a human being, if it’s not a part of your daily work.
Thanks,
Oi!