This is unreleased documentation for Fleet Next.

Observability

Status Fields

Fleet reports most information via status fields on its custom resources. These fields are also used by the Rancher UI to display information about the state of the resources.

See status fields reference for more information on status fields and conditions.

K8s Events

Fleet will generate K8s events a user can subscribe to. This is the list of events:

  • Created- a new git cloning job is created

  • GotNewCommit- a git repository has a new commit

  • JobDeleted- a successful git cloning job is removed

  • FailedValidatingSecret- a git cloning job cannot be created, because a required secret is missing

  • FailedToApplyRestrictions- the GitRepo resource violates the rules of a Policy (or, where still in use, a GitRepoRestriction) in its namespace

  • FailedToCheckCommit- cannot get latest commit from the git server

  • FailedToGetGitJob- cannot retrieve information from the git cloning job

  • Failed- polling is disabled, triggered via webhook, but cannot get latest commit from the git server

Source Labels on Deployed Resources

Fleet labels every resource it deploys with the source it came from. Use these labels to find which GitRepo, HelmOp or Bundle manages a resource in a downstream cluster.

Label Value

fleet.cattle.io/managed-by-kind

The kind of source: gitrepo, helmop or bundle. bundle is used for bundles created directly, not through a GitRepo or HelmOp.

fleet.cattle.io/managed-by-name

The name of the source. A name longer than 63 characters is shortened, so that it fits in a label value.

fleet.cattle.io/managed-by-namespace

The namespace of the source in the Fleet management cluster, for example fleet-default.

To see which source manages each resource, show the labels as columns. Run this in a downstream cluster:

kubectl get deployments -A \
  -L fleet.cattle.io/managed-by-kind,fleet.cattle.io/managed-by-name,fleet.cattle.io/managed-by-namespace
NAMESPACE   NAME       READY   UP-TO-DATE   AVAILABLE   AGE   MANAGED-BY-KIND   MANAGED-BY-NAME   MANAGED-BY-NAMESPACE
my-app      frontend   1/1     1            1           2m    gitrepo           my-app            fleet-default
kafka       kafka      1/1     1            1           5m    helmop            kafka             fleet-default
legacy      api        1/1     1            1           30d

To list only the resources deployed from the GitRepo named my-app in fleet-default:

kubectl get deployments,services,configmaps -A \
  -l fleet.cattle.io/managed-by-kind=gitrepo,fleet.cattle.io/managed-by-name=my-app,fleet.cattle.io/managed-by-namespace=fleet-default

Fleet does not add these labels to its own resources, such as the fleet-agent deployment.

The labels are informational. Users who can create or edit a source can influence their values, so do not use them for authorization decisions.

Resources Deployed Before an Upgrade

Fleet adds the source labels when it deploys a resource. After upgrading Fleet, resources that were already deployed stay unlabelled until their source is deployed again. A GitRepo or HelmOp that does not change after the upgrade keeps its resources unlabelled, like legacy/api in the example above.

To check whether a source’s resources are labelled yet, show the label columns in the namespace it deploys to. Empty columns mean the resource has not been labelled:

kubectl get deployments,services,configmaps -n legacy \
  -L fleet.cattle.io/managed-by-kind,fleet.cattle.io/managed-by-name,fleet.cattle.io/managed-by-namespace

To label the resources of an unchanged source, deploy it again by incrementing its spec.forceSyncGeneration. Fleet redeploys only when the value changes, so read the current value and set it one higher. If the field is not set yet, the new value is 1.

For a GitRepo:

GEN=$(kubectl get gitrepo my-app -n fleet-default -o jsonpath='{.spec.forceSyncGeneration}')
kubectl patch gitrepo my-app -n fleet-default --type=merge \
  -p "{\"spec\":{\"forceSyncGeneration\":$((GEN + 1))}}"

For a HelmOp:

GEN=$(kubectl get helmop kafka -n fleet-default -o jsonpath='{.spec.forceSyncGeneration}')
kubectl patch helmop kafka -n fleet-default --type=merge \
  -p "{\"spec\":{\"forceSyncGeneration\":$((GEN + 1))}}"

Metrics

Fleet publishes Prometheus metrics. They can be retrieved from these services:

  • monitoring-fleet-controller.cattle-fleet-system.svc.cluster.local:8080/metrics

  • monitoring-gitjob.cattle-fleet-system.svc.cluster.local:8081/metrics

The collection of exported metrics includes all the information from controller-runtime, like the number of reconciled resources, the number of errors, and the time it took to reconcile.

Grafana

When using Grafana and Prometheus (for example, from https://github.com/prometheus-community/helm-charts), some setup is needed to access Fleet metrics.

  1. Create a ServiceMonitor resource to scrape Fleet metrics. Here is an example:

    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  # Create this in the same namespace as your application
  namespace: cattle-fleet-system
  name: fleet-controller-monitor
  labels:
    # This label makes the ServiceMonitor discoverable by the Prometheus Operator
    release: monitoring  # <-- ADD THIS LABEL!
spec:
  selector:
    matchLabels:
      # This label must exist on the service you want to scrape
      app: fleet-controller # Assumed label, verify this
  namespaceSelector:
    matchNames:
      # We are only looking for the service in its own namespace
      - cattle-fleet-system
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  # Create this in the same namespace as your application
  namespace: cattle-fleet-system
  name: fleet-gitjob-monitor
  labels:
    # This label makes the ServiceMonitor discoverable by the Prometheus Operator
    release: monitoring  # <-- ADD THIS LABEL!
spec:
  selector:
    matchLabels:
      # This label must exist on the service you want to scrape
      app: gitjob
  namespaceSelector:
    matchNames:
      # We are only looking for the service in its own namespace
      - cattle-fleet-system
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

    And apply the manifest in Fleet’s namespace, for example in cattle-fleet-system:

    kubectl create -f servicemonitor.yaml -n cattle-fleet-system

  2. Build the Grafana dashboards and import them into Grafana. You can find the dashboards in the fleet-dashboards repository. Follow the README instructions to build them.