|
This is unreleased documentation for Fleet Next. |
Tenant Setup
This guide applies only to setups where multiple independent teams share a Fleet manager. If all users are administrators with full trust, none of this is required.
This guide walks through setting up a tenant — a team or project that shares a Fleet manager (and often downstream clusters) with other tenants — with proper isolation. For the trust model behind the steps below, see Multi-Tenancy.
A tenant is isolated by three things working together:
-
Upstream RBAC that gives the tenant the Fleet verbs they need and explicitly withholds the ones that would let them widen their own restrictions.
-
Downstream RBAC on a
ServiceAccountthat exists on every downstream cluster the tenant deploys to. This RBAC is the source of truth for what the tenant may do on each cluster. -
A
Policyin the tenant’s upstream namespace that pins the tenant to the allowedServiceAccount(s) and to allowed source repositories and credential secrets.
The remainder of this page is the recipe.
|
Tenant isolation in Fleet relies on each of the three layers being configured correctly. Skipping any one of them — or granting the tenant write access to |
Step 1 — Upstream Tenant RBAC
Each tenant gets one or more upstream namespaces (in stand-alone Fleet) or one or more FleetWorkspace objects (in a Rancher-integrated setup). Inside those namespaces, the tenant may create and manage their own GitRepo, HelmOp, and Bundle resources. They must not create, update, patch, or delete Policy objects.
The reason is structural. Multiple Policy objects in the same namespace aggregate with union semantics — see Aggregation — so a tenant who can write a Policy can extend the allow-list and reference service accounts the operator did not intend to grant.
Granting tenants read access to Policy is optional. A Policy violation surfaces as a status condition on the rejected resource with a clear message, so tenants can diagnose issues without reading the Policy directly. Read access is a convenience: it lets tenants see their constraints up front rather than discovering them through trial-and-error.
Whether to grant access to Cluster, ClusterGroup, and ClusterRegistrationToken resources depends on the cluster ownership model:
-
Operator manages clusters — the operator registers downstream clusters in a privileged namespace and maps them into tenant namespaces via
BundleNamespaceMapping. Tenants need no access to cluster resources. Do not grant read access toClusterobjects either: their status aggregates deployment state across all tenants targeting that cluster, which is a cross-tenant information leak. Tenants can check their own deployment status on theirGitRepoandBundleobjects. -
Tenant registers their own clusters — the tenant has full control over
Cluster,ClusterGroup, andClusterRegistrationTokenresources in their namespace and manages their own cluster registration. NoBundleNamespaceMappingis needed.
Both examples below include the cluster resources. Remove them if using the operator-managed model.
Stand-alone Fleet
This creates a tenant whose service account can manage their own resources in namespace project1. The tenant1-policy-read role is optional — omit it if you prefer tenants not to see the Policy directly.
kubectl create namespace project1
# The ServiceAccount represents the tenant's upstream identity on the management
# cluster. Co-locating it with the tenant's namespace is cleaner than using 'default'.
kubectl create serviceaccount tenant1 -n project1
# Tenant gets write access to their own GitRepo / HelmOp / Bundle resources,
# and to cluster resources if they register their own downstream clusters.
# Remove the three cluster resources if using the operator-managed model.
kubectl create clusterrole tenant1-write \
--verb=get,list,watch,create,update,patch,delete \
--resource=gitrepos.fleet.cattle.io,helmops.fleet.cattle.io,bundles.fleet.cattle.io,clusters.fleet.cattle.io,clustergroups.fleet.cattle.io,clusterregistrationtokens.fleet.cattle.io
# Optional: read-only access to Policy lets tenants see their constraints
# without trial-and-error. Omit if you prefer they not read Policy objects.
# Tenants must never be able to create or modify Policy objects.
kubectl create clusterrole tenant1-policy-read \
--verb=get,list,watch \
--resource=policies.fleet.cattle.io
kubectl create rolebinding tenant1-write -n project1 \
--serviceaccount=project1:tenant1 --clusterrole=tenant1-write
kubectl create rolebinding tenant1-policy-read -n project1 \
--serviceaccount=project1:tenant1 --clusterrole=tenant1-policy-read
For a second tenant in their own namespace, repeat with a different name and namespace; the two ClusterRole objects can be reused.
Rancher-integrated Fleet
In a Rancher-integrated setup, Rancher’s own auth provider (local users, LDAP, OIDC) manages tenant identity — no ServiceAccount needs to be created. Each tenant gets a FleetWorkspace and the tenant’s users get a GlobalRole that grants the necessary verbs on the workspace’s backing namespace.
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
name: project1
---
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
name: project2
Then the GlobalRole. The policies block is optional — see above. The cluster resources follow the same model described above: remove them if the operator manages cluster registration.
apiVersion: management.cattle.io/v3
kind: GlobalRole
metadata:
name: fleet-projects1and2
namespacedRules:
project1:
- apiGroups:
- fleet.cattle.io
resources:
- gitrepos
- helmops
- bundles
# Remove the three resources below if using the operator-managed model.
- clusterregistrationtokens
- clusters
- clustergroups
verbs:
- '*'
- apiGroups:
- fleet.cattle.io
resources:
- policies
verbs:
- get
- list
- watch
project2:
- apiGroups:
- fleet.cattle.io
resources:
- gitrepos
- helmops
- bundles
# Remove the three resources below if using the operator-managed model.
- clusterregistrationtokens
- clusters
- clustergroups
verbs:
- '*'
- apiGroups:
- fleet.cattle.io
resources:
- policies
verbs:
- get
- list
- watch
rules:
- apiGroups:
- management.cattle.io
resourceNames:
- project1
- project2
resources:
- fleetworkspaces
verbs:
- '*'
Assign the GlobalRole to users or groups; see the Rancher docs for details.
Allow Tenants to Target Downstream Clusters
In the operator-managed model, the operator places a BundleNamespaceMapping in the tenant’s namespace to connect their bundles to the clusters registered in the operator-owned namespace. In the tenant-managed model, no BundleNamespaceMapping is needed — the tenant’s resources and clusters share the same namespace.
kind: BundleNamespaceMapping
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: mapping
namespace: project1
# Bundles to match by label. The labels are defined in fleet.yaml's labels
# field or in the GitRepo metadata.labels field.
bundleSelector:
matchLabels:
tenant: project1
# Namespaces containing clusters, matched by label.
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: fleet-default
See Cross Namespace Deployments for the full mechanism.
Step 2 — Downstream ServiceAccount and RBAC
For each downstream cluster a tenant will deploy to, the operator creates a ServiceAccount plus a Role (or ClusterRole) and RoleBinding describing what that tenant may do on the cluster. This RBAC is the source of truth for the tenant’s downstream authority — see The Pre-Existing ServiceAccount Requirement.
The ServiceAccount lives in the Fleet agent’s namespace on the downstream cluster (cattle-fleet-system). The Role/RoleBinding describe what verbs that account holds on the actual workload namespaces.
The example below sets up two tenants, each scoped to one workload namespace on the downstream cluster.
apiVersion: v1
kind: ServiceAccount
metadata:
name: tenant-1-deployer
namespace: cattle-fleet-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: tenant-2-deployer
namespace: cattle-fleet-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tenant-1-deploy
namespace: project1-workloads
rules:
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets"]
verbs: ["*"]
- apiGroups: [""]
resources: ["services", "configmaps", "secrets", "serviceaccounts"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tenant-1-deploy
namespace: project1-workloads
subjects:
- kind: ServiceAccount
name: tenant-1-deployer
namespace: cattle-fleet-system
roleRef:
kind: Role
name: tenant-1-deploy
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: tenant-2-deploy
namespace: project2-workloads
rules:
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets"]
verbs: ["*"]
- apiGroups: [""]
resources: ["services", "configmaps", "secrets", "serviceaccounts"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tenant-2-deploy
namespace: project2-workloads
subjects:
- kind: ServiceAccount
name: tenant-2-deployer
namespace: cattle-fleet-system
roleRef:
kind: Role
name: tenant-2-deploy
apiGroup: rbac.authorization.k8s.io
This Role grants the tenant write access to a small set of resource kinds in a specific namespace. Using Role rather than ClusterRole is deliberate: a Role confines the tenant to the named namespace and prevents creation of cluster-scoped resources such as Namespace, ClusterRole, or PersistentVolume. If a tenant genuinely needs to manage cluster-scoped resources, use a ClusterRole and be explicit about which resource kinds and verbs are granted — a ClusterRole with verbs: ["*"] on all resources is equivalent to cluster-admin. Adapt the resource list, verbs, and namespace set to what each tenant actually needs.
Bootstrap Pattern: Roll Out Tenant ServiceAccounts to All Downstream Clusters
In a realistic deployment with many downstream clusters, the operator does not apply the YAML above to each cluster by hand. Instead, the operator uses Fleet itself — from a privileged upstream namespace that the operator alone controls — to fan the manifests out to every relevant downstream cluster.
The pattern: place the ServiceAccount / Role / RoleBinding manifests in a git repository, then declare a GitRepo in an operator-owned namespace (typically fleet-default) that targets all downstream clusters that should host tenants.
# In the operator-owned upstream namespace, e.g. fleet-default.
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: tenant-rbac-bootstrap
namespace: fleet-default
spec:
repo: https://github.com/example-org/fleet-operator-config
branch: main
paths:
- tenant-rbac
# This bootstrap GitRepo runs as the operator. It needs cluster-admin
# equivalent permissions on the downstream cluster to create Roles and
# RoleBindings. Tenants do not get this serviceAccount in their Policy.
serviceAccount: operator-bootstrap
# Target every downstream cluster that should host tenant workloads.
# Adjust the selector or use a ClusterGroup to match your inventory.
targets:
- clusterSelector: {}
Inside the tenant-rbac directory in the git repository, place the ServiceAccount / Role / RoleBinding YAML for all tenants the operator has provisioned. Every downstream cluster matched by the selector will receive the manifests for every tenant. Adding or removing a tenant becomes a git commit on the operator’s repository.
A few things to note about this bootstrap:
-
The bootstrap
GitReporuns with an operator-levelServiceAccount(operator-bootstrapabove), which must itself pre-exist on every downstream cluster and have permission to create cluster-scoped RBAC. The operator’sPolicyshould keep thisServiceAccountoff the tenant allow-list. -
If you use a
ClusterGroupfor the downstream cluster set, target theClusterGroupinstead of usingclusterSelector: {}. -
For tenants whose downstream surface differs per cluster, structure the git repository so each cluster’s RBAC lives in a per-cluster directory and use the GitRepo’s
targetCustomizationsto select the right directory per cluster.
Step 3 — The Policy
With the upstream RBAC and downstream ServiceAccounts in place, the Policy in the tenant’s upstream namespace ties the two together. It declares that the tenant must use one of the allowed ServiceAccounts and may pull from a restricted set of repositories and credential secrets.
kind: Policy
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: tenant-1-policy
namespace: project1
requireServiceAccount: true
allowedServiceAccounts:
- tenant-1-deployer
gitRepo:
defaultServiceAccount: tenant-1-deployer
defaultClientSecretName: tenant-1-git-credentials
allowedClientSecretNames:
- tenant-1-git-credentials
allowedRepoPatterns:
- https://github\.com/tenant-1/.*
helmOp:
defaultServiceAccount: tenant-1-deployer
defaultHelmSecretName: tenant-1-helm-credentials
allowedHelmSecretNames:
- tenant-1-helm-credentials
allowedHelmRepoPatterns:
- https://charts\.tenant-1\.example\.com/.*
allowedChartPatterns:
- tenant-1/.*
For the field-by-field reference, see Policy Resource. Repeat for each tenant in their own namespace, adjusting the allowed ServiceAccount, allowed repositories, and allowed secrets.
Step 4 — Tenant’s First GitRepo
A tenant working under the configuration above creates a GitRepo like this:
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: simpleapp
namespace: project1
labels:
tenant: project1
spec:
repo: https://github.com/tenant-1/simpleapp
paths:
- manifests
# The serviceAccount line may be omitted: the Policy's
# gitRepo.defaultServiceAccount fills it in before validation runs.
serviceAccount: tenant-1-deployer
targets:
- clusterSelector:
matchLabels:
env: dev
The BundleNamespaceMapping from Step 1 lets this GitRepo find downstream Cluster objects in the fleet-default namespace; the Policy from Step 3 pins it to the tenant-1-deployer service account; and the downstream RBAC from Step 2 determines what tenant-1-deployer may actually do on the matched clusters.
If the tenant submits a GitRepo that violates the Policy — for example, a serviceAccount that is not in the allow-list, or a repo URL that does not match the allowed patterns — the `GitRepo’s status reports the violation and no deployment proceeds.
A tenant’s HelmOp works the same way, validated against the helmOp.* fields of the same Policy.
Verifying Isolation
After applying the configuration, verify each layer:
-
As the tenant, attempt to create a
Policyin your own namespace. The request should be denied by the upstream RBAC. -
As the tenant, attempt to create a
GitRepothat references aserviceAccountoutside the allow-list. The request should be accepted, but theGitRepo’s status condition should report the `Policyviolation and noBundleshould be produced. -
As the tenant, attempt to create a
GitRepowhoserepoURL does not matchallowedRepoPatterns. Same outcome. -
On a downstream cluster, attempt to use the tenant’s
ServiceAccountto write to a namespace not granted by theRole. The request should be denied by Kubernetes RBAC.
Common Mistakes
-
Granting tenants write access to
Policy. Allow-lists union across allPolicyobjects in a namespace, so a tenant who can writePolicycan widen their own restrictions. Tenants must have at mostget/list/watchonpolicies.fleet.cattle.io. -
Forgetting
requireServiceAccount: true. Without this field, a tenant who omitsserviceAccountfrom theirGitRepoorHelmOpleaves it empty, which the downstream agent may resolve to a privileged identity. Set it explicitly. -
Setting a
defaultServiceAccountthat is not inallowedServiceAccounts. Defaults are applied before validation, so a default outside the allow-list is rejected, not silently accepted. EveryGitRepoorHelmOpthat omits the field and falls back to the default then fails validation. Keep defaults inside the allow-list. -
Forgetting to bootstrap the downstream
ServiceAccounton every cluster the tenant targets. Deployments to clusters without theServiceAccountfail at the agent. Use the bootstrap pattern in Step 2 to keep coverage in sync with the cluster inventory. -
Configuring
PolicyforGitRepoonly, ignoringHelmOp. ThehelmOp.*block must be set if tenants have RBAC to createHelmOpresources. Without it, only the top-levelrequireServiceAccount/allowedServiceAccountsconstrain HelmOps; source repositories, charts, and Helm credential secrets remain unrestricted.
Migration from GitRepoRestriction
GitRepoRestriction is deprecated and will be removed in a future release. Migration is strongly recommended for security reasons: its repo patterns are matched unanchored (a pattern like github.com/myorg also matches https://evil.com/?ref=github.com/myorg), and it provides no coverage of HelmOp or Bundle resources. It remains supported by the controller in the meantime. To migrate:
-
Author a
Policyin the same namespace as the existingGitRepoRestriction, translating each field per the table below. -
Add the downstream
ServiceAccount+ RBAC layer from Step 2 if it is not already present. The downstream RBAC replacesallowedTargetNamespacesandallowedTargetNamespaceSelector, which have no equivalent inPolicy— see What Policy Deliberately Does Not Cover. -
Set
requireServiceAccount: trueon the newPolicyand confirm the allow-list is exhaustive. -
Tighten tenant RBAC to remove any write access to
policies(and, while it exists,gitreporestrictions). -
Once the new configuration is verified, delete the
GitRepoRestriction.
|
While a Default values follow the same merging: |
Field Mapping
| GitRepoRestriction field | Policy equivalent | Notes |
|---|---|---|
|
|
Policy also offers |
|
|
Now applies to GitRepo, HelmOp, and Bundle — not just GitRepo. |
|
|
|
|
|
|
|
|
Policy patterns are anchored ( |
|
no equivalent |
Express as a |
|
no equivalent |
Same: encode namespace selection in the downstream `ServiceAccount’s RBAC. |
(no equivalent in GitRepoRestriction) |
|
New: declares explicitly that |
(no equivalent in GitRepoRestriction) |
|
New: HelmOp source and credential restrictions, which |