Register Downstream Clusters

Overview

There are two specific styles to registering clusters. These styles will be referred to as agent-initiated and manager-initiated registration. Typically one would go with the agent-initiated registration but there are specific use cases in which manager-initiated is a better workflow.

Agent-Initiated Registration

Agent-initiated refers to a pattern in which the downstream cluster installs an agent with a cluster registration token and optionally a client ID. The cluster agent will then make a API request to the Fleet manager and initiate the registration process. Using this process the Manager will never make an outbound API request to the downstream clusters and will thus never need to have direct network access.

It is not commonly used in Rancher. Rancher does not need to make an outbound API request to the downstream cluster, as it uses a tunnel to provide that connectivity.

Manager-Initiated Registration

Manager-initiated registration is a process in which you register an existing Kubernetes cluster with the Fleet manager and the Fleet manager will make an API call to the downstream cluster to deploy the agent. This style can place additional network access requirements because the Fleet manager must be able to communicate with the downstream cluster API server for the registration process. After the cluster is registered there is no further need for the manager to contact the downstream cluster API. This style is more compatible if you wish to manage the creation of all your Kubernetes clusters through GitOps using something like cluster-api or Rancher.

Feature Agent-Initiated Registration Manager-Initiated Registration (Rancher Mode)

Feature	Agent-Initiated Registration	Manager-Initiated Registration (Rancher Mode)
Registration Initiator	The downstream cluster initiates the registration process by deploying the agent.	The Fleet Manager (upstream cluster) initiates the registration process.
Prerequisite: Upstream Admin Action	Requires manual administrative action to create a `ClusterRegistrationToken` resource upstream. This token is required for agent authorization.	Requires creating a `Cluster` resource in the Fleet Manager that references a Kubernetes Secret containing a valid kubeconfig for the downstream cluster.
Primary Mechanism	Installing the Fleet agent via Helm on the downstream cluster using the manually generated cluster registration token.	The Fleet Controller uses the provided kubeconfig to make an API call to the downstream cluster’s API server and deploy the agent.
Registration Network Direction	Communication flows from the managed cluster to the Fleet Controller (upstream). This is a PULL mechanism for registration credentials.	The Manager initiates a connection (PUSH) to the downstream cluster API server to deploy the agent.
Network Requirement (Manager)	The Fleet Manager does not need direct inbound network access to the downstream cluster API. Clusters can run in private networks/behind NATs.	The Fleet Manager must be able to communicate directly with the downstream cluster API server during the registration phase.
Network Requirement (Agent)	The downstream cluster must be able to make outbound HTTPS calls to the Fleet Manager.	The downstream cluster must be able to make outbound HTTPS calls to the Fleet Manager (post-deployment, for communication and data synchronization).
Data Synchronization (Pull)	The agent pulls configuration bundles and agent updates from the Fleet Controller (part of the two-stage pull model).	The agent pulls configuration bundles and agent updates from the Fleet Controller.
Agent Management (Push/Redeploy)	The upstream controller typically lacks the direct network access/credentials required for proactive management actions (PUSH) on the agent deployment.	The initial deployment mechanism grants the Manager explicit access. This infrastructure can be used for administrative management actions (e.g., triggering deployment using the kubeconfig).
Agent Redeployment Control Field	The `redeployAgentGeneration` field (on `ClusterSpec`) is not used by the upstream controller to trigger redeployment, as the manager lacks the API access necessary to directly manipulate the downstream agent deployment.	The `redeployAgentGeneration` field on the `ClusterSpec` can be incremented to force redeployment of the agent by the Fleet Controller.
Agent Adoption Post-Registration	After initial registration using the token, the agent is adopted into the standard management lifecycle via bundles created by the `manageagent` controller.
Rancher Context	This method is not commonly used when registering clusters through the Rancher dashboard.	This method is used when adding a cluster via the Rancher dashboard (often via a Rancher agent proxying the downstream kubeconfig upstream).
Post-Registration Token Status	The agent forgets the registration token after success; a new token must be generated for re-registration.	N/A; registration token/kubeconfig management is handled internally by the Fleet Manager.

Registration Initiator

The downstream cluster initiates the registration process by deploying the agent.

The Fleet Manager (upstream cluster) initiates the registration process.

Prerequisite: Upstream Admin Action

Requires manual administrative action to create a ClusterRegistrationToken resource upstream. This token is required for agent authorization.

Requires creating a Cluster resource in the Fleet Manager that references a Kubernetes Secret containing a valid kubeconfig for the downstream cluster.

Primary Mechanism

Installing the Fleet agent via Helm on the downstream cluster using the manually generated cluster registration token.

The Fleet Controller uses the provided kubeconfig to make an API call to the downstream cluster’s API server and deploy the agent.

Registration Network Direction

Communication flows from the managed cluster to the Fleet Controller (upstream). This is a PULL mechanism for registration credentials.

The Manager initiates a connection (PUSH) to the downstream cluster API server to deploy the agent.

Network Requirement (Manager)

The Fleet Manager does not need direct inbound network access to the downstream cluster API. Clusters can run in private networks/behind NATs.

The Fleet Manager must be able to communicate directly with the downstream cluster API server during the registration phase.

Network Requirement (Agent)

The downstream cluster must be able to make outbound HTTPS calls to the Fleet Manager.

The downstream cluster must be able to make outbound HTTPS calls to the Fleet Manager (post-deployment, for communication and data synchronization).

Data Synchronization (Pull)

The agent pulls configuration bundles and agent updates from the Fleet Controller (part of the two-stage pull model).

The agent pulls configuration bundles and agent updates from the Fleet Controller.

Agent Management (Push/Redeploy)

The upstream controller typically lacks the direct network access/credentials required for proactive management actions (PUSH) on the agent deployment.

The initial deployment mechanism grants the Manager explicit access. This infrastructure can be used for administrative management actions (e.g., triggering deployment using the kubeconfig).

Agent Redeployment Control Field

The redeployAgentGeneration field (on ClusterSpec) is not used by the upstream controller to trigger redeployment, as the manager lacks the API access necessary to directly manipulate the downstream agent deployment.

The redeployAgentGeneration field on the ClusterSpec can be incremented to force redeployment of the agent by the Fleet Controller.

Agent Adoption Post-Registration

After initial registration using the token, the agent is adopted into the standard management lifecycle via bundles created by the manageagent controller.

Rancher Context

This method is not commonly used when registering clusters through the Rancher dashboard.

This method is used when adding a cluster via the Rancher dashboard (often via a Rancher agent proxying the downstream kubeconfig upstream).

Post-Registration Token Status

The agent forgets the registration token after success; a new token must be generated for re-registration.

N/A; registration token/kubeconfig management is handled internally by the Fleet Manager.

Agent Initiated

A downstream cluster is registered by installing an agent via helm and using the cluster registration token and optionally a client ID or cluster labels.

The cluster resource on upstream does not have a kubecConfigSecret field, as the Fleet manager does not need to communicate with the downstream cluster API server. However, a bundle for the agent is created and the agent will update itself from the bundle. The bundle uses a different naming scheme, so the downstream cluster will end up with two helm charts.

It’s not necessary to configure the Fleet manager for multi cluster, as the downstream agent we install via Helm will connect to the Kubernetes API of the upstream cluster directly.

Agent-initiated registration is normally not used with Rancher.

Cluster Registration Token and Client ID

The cluster registration token is a credential that will authorize the downstream cluster agent to be able to initiate the registration process. This is required. The cluster registration token is manifested as a values.yaml file that will be passed to the helm install process. Alternatively one can pass the token directly to the helm install command via --set token="$token".

There are two styles of registering an agent. You can have the cluster for this agent dynamically created, in which case you will probably want to specify cluster labels upon registration. Or you can have the agent register to a predefined cluster in the Fleet manager, in which case you will need a client ID. The former approach is typically the easiest.

Install Agent For a New Cluster

The Fleet agent is installed as a Helm chart. Following are explanations how to determine and set its parameters.

First, follow the cluster registration token instructions to obtain the values.yaml which contains the registration token to authenticate against the Fleet cluster.

Second, optionally you can define labels that will assigned to the newly created cluster upon registration. After registration is completed an agent cannot change the labels of the cluster. To add cluster labels add --set-string labels.KEY=VALUE to the below Helm command. To add the labels foo=bar and bar=baz then you would add --set-string labels.foo=bar --set-string labels.bar=baz to the command line.

# Leave blank if you do not want any labels
CLUSTER_LABELS="--set-string labels.example=true --set-string labels.env=dev"

Third, set variables with the Fleet cluster’s API Server URL and CA, for the downstream cluster to use for connecting.

API_SERVER_URL=https://<API_URL>:6443
API_SERVER_CA_DATA=...

If the API server is not listening on the https port (443), the API_SERVER_URL should include the port, e.g. https://<API_URL>:6443. The URL can be found in the .kube/config file.

Value in API_SERVER_CA_DATA can be obtained from a .kube/config file with valid data to connect to the upstream cluster (under the certificate-authority-data key). Alternatively it can be obtained from within the upstream cluster itself, by looking up the default ServiceAccount secret name (typically prefixed with default-token-, in the default namespace), under the ca.crt key.

Use proper namespace and release name: For the agent chart the namespace must be cattle-fleet-system and the release name fleet-agent

Kubectl Context

Ensure you are installing to the right cluster: Helm will use the default context in ${HOME}/.kube/config to deploy the agent. Use --kubeconfig and --kube-context to change which cluster Helm is installing to.

Fleet in Rancher

Rancher has separate helm charts for Fleet and uses a different repository.

Add Fleet’s Helm repo.

helm repo add fleet https://rancher.github.io/fleet-helm-charts/

Finally, install the agent using Helm.

Install
Validate

helm -n cattle-fleet-system install --create-namespace --wait \
    --set clientID="$CLUSTER_CLIENT_ID" \
    --values values.yaml \
    fleet-agent fleet/fleet-agent

You can check that status of the fleet pods by running the below commands:

# Ensure kubectl is pointing to the right cluster kubectl -n cattle-fleet-system logs -l app=fleet-agent kubectl -n cattle-fleet-system get pods -l app=fleet-agent

The agent should now be deployed.

Additionally you should see a new cluster registered in the Fleet manager. Below is an example of checking that a new cluster was registered in the clusters namespace. Please ensure your ${HOME}/.kube/config is pointed to the Fleet manager to run this command.

kubectl -n clusters get clusters.fleet.cattle.io

NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE             LAST-SEEN              STATUS
cluster-ab13e54400f1   1/1             1/1           k3d-cluster2-server-0   2020-08-31T19:23:10Z

Install Agent For a Predefined Cluster

Client IDs are for the purpose of predefining clusters in the Fleet manager with existing labels and repos targeted to them. A client ID is not required and is just one approach to managing clusters. The client ID is a unique string that will identify the cluster.

This string is user generated and opaque to the Fleet manager and agent. It is assumed to be sufficiently unique. For security reasons one should not be able to easily guess this value as then one cluster could impersonate another. The client ID is optional and if not specified the UID field of the kube-system namespace resource will be used as the client ID. Upon registration if the client ID is found on a Cluster resource in the Fleet manager it will associate the agent with that Cluster. If no Cluster resource is found with that client ID a new Cluster resource will be created with the specific client ID.

After registration, the Cluster is bound to the registration token it registered with, see Registration Token Binding.

The Fleet agent is installed as a Helm chart. The only parameters to the helm chart installation should be the cluster registration token, which is represented by the values.yaml file and the client ID. The client ID is optional.

First, create a Cluster in the Fleet Manager with the random client ID you have chosen.

kind: Cluster
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: my-cluster
  namespace: clusters
spec:
  clientID: "really-random"

Second, follow the [cluster registration token instructions]((#create-cluster-registration-tokens) to obtain the values.yaml file to be used.

Third, setup your environment to use the client ID.

CLUSTER_CLIENT_ID="really-random"

Use proper namespace and release name: For the agent chart the namespace must be cattle-fleet-system and the release name fleet-agent

Add Fleet’s Helm repo.

helm repo add fleet https://rancher.github.io/fleet-helm-charts/

Finally, install the agent using Helm.

Install
Validate

helm -n cattle-fleet-system install --create-namespace --wait \
    --set clientID="$CLUSTER_CLIENT_ID" \
    --values values.yaml \
    fleet-agent fleet/fleet-agent

You can check that status of the fleet pods by running the below commands:

# Ensure kubectl is pointing to the right cluster kubectl -n cattle-fleet-system logs -l app=fleet-agent kubectl -n cattle-fleet-system get pods -l app=fleet-agent

The agent should now be deployed.

kubectl -n clusters get clusters.fleet.cattle.io

NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE             LAST-SEEN              STATUS
my-cluster             1/1             1/1           k3d-cluster2-server-0   2020-08-31T19:23:10Z

Create Cluster Registration Tokens

Not needed for Manager-initiated registration: For manager-initiated registrations the token is managed by the Fleet manager and does not need to be manually created and obtained.

For an agent-initiated registration the downstream cluster must have a cluster registration token. Cluster registration tokens are used to establish a new identity for a cluster. Internally cluster registration tokens are managed by creating Kubernetes service accounts that have the permissions to create ClusterRegistrationRequests within a specific namespace. Once the cluster is registered a new ServiceAccount is created for that cluster that is used as the unique identity of the cluster. The agent is designed to forget the cluster registration token after registration. While the agent will not maintain a reference to the cluster registration token after a successful registration please note that usually other system bootstrap scripts do.

Since the cluster registration token is forgotten, if you need to re-register a cluster you must give the cluster a new registration token. If that token has a different name than the token the cluster registered with, update the binding first, see Registration Token Binding.

Token TTL

Cluster registration tokens can be reused by any cluster in a namespace. The tokens can be given a TTL such that it will expire after a specific time.

Clusters registered with the same token are not distinguished from each other. Do not share a cluster registration token across tenants or trust boundaries. Instead, use a separate namespace with its own tokens for each tenant.

Create a new Token

The ClusterRegistationToken is a namespaced type and should be created in the same namespace in which you will create GitRepo and ClusterGroup resources. For in depth details on how namespaces are used in Fleet refer to the documentation on namespaces. Create a new token with the below YAML.

kind: ClusterRegistrationToken
apiVersion: "fleet.cattle.io/v1alpha1"
metadata:
    name: new-token
    namespace: clusters
spec:
    # A duration string for how long this token is valid for. A value <= 0 or null means infinite time.
    ttl: 240h

After the ClusterRegistrationToken is created, Fleet will create a corresponding Secret with the same name. As the Secret creation is performed asynchronously, you will need to wait until it’s available before using it.

One way to do so is via the following one-liner:

while ! kubectl --namespace=clusters  get secret new-token; do sleep 5; done

Obtaining Token Value (Agent values.yaml)

The token value contains YAML content for a values.yaml file that is expected to be passed to helm install to install the Fleet agent on a downstream cluster.

Such value is contained in the values field of the Secret mentioned above. To obtain the YAML content for the above example one can run the following one-liner:

kubectl --namespace clusters get secret new-token -o 'jsonpath={.data.values}' | base64 --decode > values.yaml

Once the values.yaml is ready it can be used repeatedly by clusters to register until the TTL expires.

Registration Token Binding

Registration token binding is available since Fleet v0.16.3.

Fleet binds each cluster registered through agent-initiated registration to the cluster registration token it registered with. The name of that token is recorded in the fleet.cattle.io/registered-token annotation of the Cluster resource. New clusters receive the annotation when they are created. Existing clusters without the annotation receive it the next time a registration is granted for them after an upgrade of Fleet.

A registration that uses the client ID of an existing cluster together with a different registration token is not granted. The registration stays pending, and Fleet emits a RegistrationTokenMismatch warning event on the Cluster. The event names the annotation value that allows the new token. The commands below assume the Cluster is in the clusters namespace; replace clusters with its actual namespace:

kubectl -n clusters get events --field-selector reason=RegistrationTokenMismatch

Re-registration with a token of the same name is not affected, and neither are clusters whose agent is deployed by Fleet through manager-initiated registration. If the registration token named by a registration does not exist, the registration stays pending until the token exists.

To re-register a cluster with a different registration token on purpose, set the annotation to the name of the new token. Fleet then processes the pending registrations of that cluster again:

kubectl -n clusters annotate --overwrite clusters.fleet.cattle.io my-cluster fleet.cattle.io/registered-token=new-token

To accept registrations with a different registration token without updating the annotation, as in earlier releases, install Fleet with --set allowClusterRegistrationTokenChange=true. This disables the protection described in this section.

Manager Initiated

The manager-initiated registration flow is accomplished by creating a Cluster resource in the Fleet Manager that refers to a Kubernetes Secret containing a valid kubeconfig file in the data field called value.

If you are using Fleet standalone without Rancher, it must be installed as described in installation details.

The manager-initiated registration is used when you add a cluster from the Rancher dashboard.

Create Kubeconfig Secret

The format of this secret is intended to match the format of the kubeconfig secret used in cluster-api. This means you can use cluster-api to create a cluster that is dynamically registered withFleet.

title="Kubeconfig Secret Example"
kind: Secret
apiVersion: v1
metadata:
  name: my-cluster-kubeconfig
  namespace: clusters
data:
  value: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIHNlcnZlcjogaHR0cHM6Ly9leGFtcGxlLmNvbTo2NDQzCiAgbmFtZTogY2x1c3Rlcgpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3RlcjogY2x1c3RlcgogICAgdXNlcjogdXNlcgogIG5hbWU6IGRlZmF1bHQKY3VycmVudC1jb250ZXh0OiBkZWZhdWx0CmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogdXNlcgogIHVzZXI6CiAgICB0b2tlbjogc29tZXRoaW5nCg==

Create Cluster Resource

The cluster resource needs to reference the kubeconfig secret.

title="Cluster Resource Example"
apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  name: my-cluster
  namespace: clusters
  labels:
    demo: "true"
    env: dev
spec:
  kubeConfigSecret: my-cluster-kubeconfig

Reference a Kubeconfig Secret in Another Namespace

The restrictions described in this section apply since Fleet v0.16.3.

By default, Fleet reads the kubeconfig secret from the namespace of the Cluster resource. The kubeConfigSecretNamespace field selects a different namespace:

apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  name: my-cluster
  namespace: clusters
spec:
  kubeConfigSecret: my-cluster-kubeconfig
  kubeConfigSecretNamespace: kubeconfigs

Fleet only uses a kubeconfig secret in another namespace if the secret carries the label fleet.cattle.io/kubeconfig-referenceable-from with the namespace of the Cluster as its value. Only users who can modify secrets in the namespace of the secret can set this label, so they decide which namespace can use the credentials stored there. For example, to allow Cluster resources in the clusters namespace to use the secret above:

kubectl -n kubeconfigs label secret my-cluster-kubeconfig fleet.cattle.io/kubeconfig-referenceable-from=clusters

If the label is missing or names a different namespace, Fleet does not import the cluster, and the Imported condition of the Cluster names the required label. An agent that is already deployed stays in place. Fleet detects a label that is added later without further changes.

Rancher keeps the kubeconfig secret of a cluster that is assigned to a workspace other than fleet-default in the fleet-default namespace. Rancher versions that bundle this Fleet release add the label to these secrets automatically.

Earlier releases accepted unlabeled kubeconfig secrets from any namespace if the secret was in one of the well-known Fleet namespaces: fleet-local, fleet-default, and the Fleet system namespaces, such as cattle-fleet-system and cattle-fleet-clusters-system. To keep this behavior until the affected secrets are labeled, install Fleet with --set allowUnlabeledKubeConfigSecretNamespaces=true. This weakens the protection described in this section, so use it only temporarily. A secret with a nonempty label can still only be used from the namespace that the label names.

Independently of the label and this setting, Fleet only deploys an agent onto the management cluster for Cluster resources in the bootstrap namespace, fleet-local by default, or in the Fleet system namespaces.