Customizing Kf

• 1: Customizing Overview
• 2: Cluster Configuration
• 3: Customizing Kf Features

1 - Customizing Overview

Kf supports customizing some cluster wide configurations by manipulating the kfsystem custom rsource.

2 - Cluster Configuration

Kf uses a Kubernetes configmap named config-defaults in the kf namespace to store cluster wide configuration settings. This document explains its structure and fields.

Structure of the config-defaults configmap

The configmap contains three types of key/value pairs in the .data field:

• Comment keys prefixed by _ contain examples, notes, and warnings.
• String keys contain plain text values.
• Object keys contain a JSON or YAML value that has been encoded as a string.

Example:

_note: "This is some note"
stringKey: "This is a string key that's not encoded as JSON or YAML."
objectKey: |
  - "These keys contain nested YAML or JSON."
  - true
  - 123.45  

Example section

The example section under the _example key contains explanations for other fields and examples. Changes to this section have no effect.

Space container registry

The spaceContainerRegistry property is a plain text value that specifies the default container registry each space uses to store built images.

Example:

spaceContainerRegistry: gcr.io/my-project

Space cluster domains

The spaceClusterDomains property is a string encoded YAML array of domain objects.

Each space in the cluster adds all items in the array to its list of domains that developers can bind their apps to.

Example:

spaceClusterDomains: |
  # Support canonical and vanity domains
  - domain: $(SPACE_NAME).prod.example.com
  - domain: $(SPACE_NAME).kf.us-east1.prod.example.com

  # Using a dynamic DNS resolver
  - domain: $(SPACE_NAME).$(CLUSTER_INGRESS_IP).nip.io

  # Creating an internal domain only visible within the cluster
  - domain: $(SPACE_NAME)-apps.internal
    gatewayName: kf/internal-gateway  

Buildpacks V2 lifecycle builder

The buildpacksV2LifecycleBuilder property contains the version of the Cloud Foundry builder binary used execute buildpack v2 builds.

The value is a Git reference. To use a specific version, append an @ symbol followed by a Git SHA to the end.

Example:

buildpacksV2LifecycleBuilder: "code.cloudfoundry.org/buildpackapplifecycle/builder@GIT_SHA"

Buildpacks V2 lifecycle launcher

The buildpacksV2LifecycleLauncher property contains the version of the Cloud Foundry launcher binary built into every buildpack V2 application.

The value is a Git reference. To use a specific version, append an @ symbol followed by a Git SHA to the end.

Example:

buildpacksV2LifecycleLauncher: "code.cloudfoundry.org/buildpackapplifecycle/launcher@GIT_SHA"

Buildpacks V2 list

The spaceBuildpacksV2 property is a string encoded YAML array that holds an ordered list of default buildpacks that are used to build applications compatible with the V2 buildpacks process.

Stacks V2 list

The spaceBuildpacksV2 property is a string encoded YAML array that holds an ordered list of stacks that can be used with Cloud Foundry compatible builds.

Stacks V3 list

The spaceStacksV3 property is a string encoded YAML array that holds an ordered list of stacks that can be used with Cloud Native Buildpack builds.

Example:

spaceStacksV3: |
  - name: heroku-18
    description: The official Heroku stack based on Ubuntu 18.04
    buildImage: heroku/pack:18-build
    runImage: heroku/pack:18
    nodeSelector:
       kubernetes.io/os: windows  

Default to V3 Stack

The spaceDefaultToV3Stack property contains a quoted value true or false indicating whether spaces should use V3 stacks if a user doesn’t specify one.

Feature flags

The featureFlags property contains a string encoded YAML map of feature flags that can enable and disable features of Kf.

Flag names that aren’t supported by Kf will be ignored.

Example:

featureFlags: |
  disable_custom_builds: false
  enable_dockerfile_builds: true
  enable_some_feature: true  

ProgressDeadlineSeconds

ProgressDeadlineSeconds contains a configurable quoted integer indicating the maximum allowed time between state transition and reaching a stable state before provisioning or deprovisioning when pushing an application. The default value is 600 seconds.

TerminationGracePeriodSeconds

The TerminationGracePeriodSeconds contains a configurable quoted integer indicating the time between when the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. The default value is 30 seconds.

3 - Customizing Kf Features

Build Retention

You can control how many Kf Builds are kept before being garbage collected.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildRetentionCount', 'value': 1}]"

Task Retention

You can control how many Kf Tasks are kept before being garbage collected.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/taskRetentionCount', 'value': 1}]"

Enable or Disable the Istio Sidecar

If you do not require the Istio sidecar for the Build pods, then they can be disabled by setting the value to true. Enable by setting the value to false.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildDisableIstioSidecar', 'value': true}]"

Enable or Disable Routing Retries

Allows enabling/disbaling retries in the VirtualServices that route traffic to Apps. Kf leaves this value unset by default and it’s inherited from Istio.

Istio’s default retry mechanism attempts to make up for instability inherent in service meshes, however allowing retries requires the contents of the payload to be buffered within Envoy. This may fail for large payloads and the buffering will need to be disabled at the expense of some stability.

Values for routeDisableRetries:

• false Inherit Istio’s retry settings. (Default)
• true Set retries to 0.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/routeDisableRetries','value':true}]"

Enable or Disable Routing Hosts Ignoring Any Port Numbers

Allows enabling/disabling routing hosts ignoring any specified port number. By default hosts are matched using the exact value specified in the route Host (e.g a request with a Host header value of example.com:443 does not match with preconfigured route Host example.com). By enabling, ports are ignored and only hosts are used (e.g example.com:443 matches example.com)

Note: Feature will only work in clusters with istio 1.15+. In older versions it will function as though it where disabled.

Values for routeHostIgnoringPort:

• false Will match the Host header in request to the exact configured route Host. (Default)
• true Will use regexp to match to the configured route Host ignoring any port specified in the Host header of the request.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/routeHostIgnoringPort','value':true}]"

Build Pod Resource Limits

The default pod resource size can be increased from the default to accommodate very large builds. The units for the value are in Mi or Gi.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildPodResources', 'value': {'limits': {'memory': '234Mi'}}}]"

Read Kubernetes container resource docs for more information about container resource management.

Robust Build Snapshots

Kf uses Kaniko to build the final application containers in the v2 buildpack lifecycle. To produce image layers, Kaniko needs to take “snapsohts” of the image which requires iterating over all files in the image and checking if they’ve changed.

The fast mode checks for file attribute changes (like timestamps and size) and the slow mode checks full file hashes to determine if a file changed.

Kf apps aren’t expected to overwrite operating system files in their build, so fast mode should be used to reduce disk pressure.

Values for buildKanikoRobustSnapshot:

• false Will use a fast snapshot mode for v2 builds. (Default)
• true Will use a robust snapshot mode for v2 builds to catch uncommon cases.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildKanikoRobustSnapshot', 'value': true}]"

Self Signed Certificates for Service Brokers

If you want to use self signed certificates for TLS (https instead of http) for the service broker URL, the Kf controller requires the CA certificate. To configure Kf for this scenario, create an immutable Kubernetes secret in the kf namespace and update the kfsystem.spec.kf.config.secrets.controllerCACerts.name object to point to it.

1. Create a secret to store the self-signed certificate.

    kubectl create secret generic cacerts -nkf --from-file /path/to/cert/certs.pem
    

2. Make the secret immutable.

    kubectl patch -nkf secret cacerts \
        --type='json' \
        -p="[{'op':'add','path':'/immutable','value':true}]"
    

3. Update kfsystem to point to the secret.

   kubectl patch \
     kfsystem kfsystem \
     --type='json' \
     -p="[{'op':'add','path':'/spec/kf/config/secrets','value':{'controllerCACerts':{'name':'<var>cacerts</var>'}}}]"
   

Set CPU minimums and ratios

Application default CPU ratios and minimums can be set in the operator.

Values are set in CPU units. Units are typically expressed in millicpus (m), or thousandths of a CPU.

The spec.kf.config.appCPUMin property specifies a minimum amount of CPU per application, even if the developer has specified less.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appCPUMin','value':'<var>200m</var>'}]"

The spec.kf.config.appCPUPerGBOfRAM property specifies a default amount of CPU to give each app per GB or RAM requested.

You can choose different approaches based on the desired outcome:

• Choose the ratio of CPU to RAM for the cluster’s nodes if you want to maximize utilization.
• Choose a ratio of 1 CPU to 4 GB of RAM which typically works well for I/0 or memory bound web applications.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appCPUPerGBOfRAM','value':'<var>250m</var>'}]"

Set buildpacks using git tags

Buildpacks can support pinning by using git tags instead of automatically sourcing the latest buildpack from a git repository.

Add a new buildpack as follows and use a git tag to specify which version of the buildpack the app should use. Otherwise the buildpack will default to the latest version.

For example, to pin Golang buildpack version 1.9.49 do:

kubectl patch \
kfsystem kfsystem \
--type='json' \
 -p='[{"op":"add","path":"data/spec/kf/config/spaceBuildpacksV2","value":[{"name":"go_buildpack_v1.9.49","url":"https://github.com/cloudfoundry/go-buildpack.git#v1.9.49"}]}]'

This command will add the following to the config-defaults configmaps resource:

data:
  SpaceBuildpacksV2: |
    - name: go_buildpack_v1.9.49
      url: https://github.com/cloudfoundry/go-buildpack.git#v1.9.49

The kubectl patch command will replace all the existing buildpacks in the config-defaults configmaps. If the user would like the existing buildpacks to remain, these too need to be included in the command.

To get the list of existing buildpacks in the configmaps run the following command:

kubectl describe configmaps config-defaults -n kf

Set ProgressDeadlineSeconds

ProgressDeadlineSeconds can be set in the kfsystem operator.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/progressDeadlineSeconds','value':100}]"

Set TerminationGracePeriodSeconds

TerminationGracePeriodSeconds can be set in the kfsystem operator.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/terminationGracePeriodSeconds','value':200}]"

Enable/Disable App Start Command Lookup

Allows enabling/disbaling start command lookup in the App reconciler. This behavior requires the reconciler to fetch container configuration for every app from the container registry and enables displaying the start command on kf push and in kf app.

Enabling this behavior on a large cluster may make the reconcilation times for Apps slow.

Values for appDisableStartCommandLookup:

• false Enable start command lookup. (Default)
• true Disable start command lookup.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appDisableStartCommandLookup','value':true}]"

Enable/Disable Kubernetes service Account (KSA) overrides

Allows enabling/disbaling the ability to override the Kubernetes Service Account for Apps via annotation.

Values for appEnableServiceAccountOverride:

• false Don’t allow overriding service account. (Default)
• true Allow developers to override KSAs for their Apps.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appEnableServiceAccountOverride','value':true}]"

Set default Kf Task timeout

Kf uses Tekton TaskRuns as its mechanism to run Kf Tasks. Tekton may impose a default timeout on TaskRuns that differs depending on the version of Tekton you have installed.

You can override this setting either on the Tekton configmap – which will set the value for both Kf Tasks and Builds or on the Kf operator to apply the value only to Tasks.

The following values are supported:

• null - Inherit the value from Tekton (Default).
• Value <= 0 - Tasks get an infinite timeout.
• Value >= 0 - Tasks get the given timeout.

Consider setting a long, but non-infinite timeout to prevent improperly programmed tasks from consuming resources.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/taskDefaultTimeoutMinutes','value':-1}]"

Enable/Disbale NFS in Kf Tasks

You can enable/disable the ability for Tasks run from Apps to mount NFS volumes.

Mounting NFS volumes requires FUSE which grants Task Pods with NFS additional system privileges on the node and may be a security concern.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/taskDisableVolumeMounts','value':true}]"