Automatically copying resources to downstream clusters

This is an experimental feature.

From Fleet v0.14.0 onwards, Fleet supports propagating external resources to downstream clusters.

This simplifies dealing with dependencies of charts, such as values coming from external resources. See also valuesFrom.

How it works

HelmOps support a new downstreamResources field, which can be used to reference resources by kind and name. Those resources must:

  • Be either secrets or config maps. No other `kind`s are currently supported.

  • Exist before being referenced from the HelmOp, and live in the same namespace as the HelmOp referencing them.

Example:

apiVersion: fleet.cattle.io/v1alpha1
kind: HelmOp
[...] # metadata
spec:
  helm:
    [...] # Helm options
  downstreamResources:
    - kind: Secret
      name: my-secret
    - kind: ConfigMap
      name: my-config

This instructs the Fleet controller to copy those resources to each targeted downstream cluster, before deploying the workload (in this case specified through a Helm chart) to said downstream cluster.

When a cluster is not targeted anymore, the Fleet agent will delete those resources from the cluster as well. They will remain on the upstream cluster, though.

If resources referenced through downstreamResources should stay on downstream clusters even after they are no longer targeted, keepResources should be set to true on the HelmOp.

Permissions

Since Fleet v0.15.7, when a ServiceAccount is resolved for the deployment on the downstream cluster, the agent writes the copies as that ServiceAccount. This is the account set in serviceAccount, or the fleet-default account in cattle-fleet-system if none is set and it exists. The ServiceAccount needs get, create and update on the copied kinds in the deployment namespace.

The deployment namespace must exist before the copy runs. If it does not, Fleet tries to create it as the ServiceAccount, which requires create on the cluster-scoped namespaces resource. Do not grant that verb to a tenant, because it cannot be limited to particular namespaces. Create the namespace as an operator instead.

Removing the copies when a cluster is no longer targeted is done by the agent itself and needs no permission on the ServiceAccount. The agent only removes objects that carry the fleet.cattle.io/bundledeployment label Fleet sets on the copies.

A missing permission is not a failed deployment

If the ServiceAccount may not create the deployment namespace or write a copied object, Fleet stops before installing anything. The bundle deployment reports not ready, with a message naming the namespace or object and the forbidden error, and Fleet retries on its own. Once the permission is granted or the namespace is created, the deployment proceeds without further action.

Monitoring

From version v0.15.0 onwards, Fleet monitors changes made to secrets and configmaps that are listed in DownstreamResources. It also checks whether a secret or configmap was created after the Bundle was created.

When a referenced secret or configmap is created or updated on the upstream cluster, Fleet will propagate those changes to downstream clusters, and re-deploy the workload.

This enables deployments to make use of the latest versions of those referenced resources, which may represent configuration, Helm values, etc.