Docs

• 1: Documentation
• 1.1: Getting Started
• 1.1.1: Getting Started Overview
• 1.1.2: Quickstart
• 1.1.3: Compare Cloud Foundry and Kf services
• 1.2: Develop Applications on Kf
• 1.2.1: Build and deploy applications
• 1.2.1.1: Deploy an application
• 1.2.1.2: Get started with buildpacks
• 1.2.1.3: Reduce deployment risk with blue-green deployments
• 1.2.1.4: Compare V2 and V3 Buildpacks
• 1.2.1.5: App Manifest
• 1.2.1.6: App runtime
• 1.2.1.7: Build runtime
• 1.2.2: Backing services
• 1.2.2.1: Backing Services Overview
• 1.2.2.2: Use managed services
• 1.2.2.3: Configure user-provided services
• 1.2.2.4: User Provided Service Templates
• 1.2.2.5: Configure NFS volumes
• 1.2.3: Configure and Use Service Accounts
• 1.2.4: Scaling
• 1.2.4.1: Scaling overview
• 1.2.4.2: Manage Autoscaling
• 1.2.4.3: Managing Resources for Apps
• 1.2.5: Service discovery
• 1.2.6: Debugging workloads
• 1.2.7: Configure routes and domains
• 1.2.8: Tasks
• 1.2.8.1: Tasks Overview
• 1.2.8.2: Run Tasks
• 1.2.8.3: Schedule Tasks
• 1.3: Operate and Maintain Kf
• 1.3.1: Service brokers
• 1.3.1.1: Service Brokers Overview
• 1.3.1.2: About Kf Cloud Service Broker
• 1.3.1.3: Deploy Kf Cloud Service Broker
• 1.3.1.4: Set up NFS platform
• 1.3.2: Customizing Kf
• 1.3.2.1: Customizing Overview
• 1.3.2.2: Cluster Configuration
• 1.3.2.3: Customizing Kf Features
• 1.3.3: Testing Kf Components
• 1.3.4: Kf dependencies and architecture
• 1.3.5: Kf reference architecture diagrams
• 1.3.6: Logging and Monitoring
• 1.3.6.1: Create and user monitoring dashboards.
• 1.3.6.2: Logging and monitoring
• 1.3.6.3: Logging and monitoring overview
• 1.3.6.4: View logs
• 1.3.7: Networking
• 1.3.7.1: Set up a custom domain
• 1.3.7.2: Set up HTTPS ingress
• 1.3.7.3: Set up network policies
• 1.3.8: Security
• 1.3.8.1: Security Overview
• 1.3.8.2: Role-based access control
• 1.3.8.3: Configure role-based access control
• 1.3.8.4: Enable compute isolation
• 1.3.8.5: Kubernetes Roles
• 1.3.9: Stacks and Buildpacks
• 1.3.9.1: Customize stacks
• 1.3.9.2: Customize stacks and buildpacks
• 1.4: Kf CLI Reference
• 1.4.1: kf about
• 1.4.2: kf app
• 1.4.3: kf apps
• 1.4.4: kf bind-route-service
• 1.4.5: kf bind-service
• 1.4.6: kf bindings
• 1.4.7: kf build
• 1.4.8: kf build-logs
• 1.4.9: kf buildpacks
• 1.4.10: kf builds
• 1.4.11: kf configure-cluster
• 1.4.12: kf configure-cluster get-feature-flags
• 1.4.13: kf configure-cluster set-feature-flag
• 1.4.14: kf configure-cluster unset-feature-flag
• 1.4.15: kf configure-space
• 1.4.16: kf configure-space append-domain
• 1.4.17: kf configure-space get-build-service-account
• 1.4.18: kf configure-space get-buildpack-env
• 1.4.19: kf configure-space get-container-registry
• 1.4.20: kf configure-space get-domains
• 1.4.21: kf configure-space get-execution-env
• 1.4.22: kf configure-space get-nodeselector
• 1.4.23: kf configure-space remove-domain
• 1.4.24: kf configure-space set-app-egress-policy
• 1.4.25: kf configure-space set-app-ingress-policy
• 1.4.26: kf configure-space set-build-egress-policy
• 1.4.27: kf configure-space set-build-ingress-policy
• 1.4.28: kf configure-space set-build-service-account
• 1.4.29: kf configure-space set-buildpack-env
• 1.4.30: kf configure-space set-container-registry
• 1.4.31: kf configure-space set-default-domain
• 1.4.32: kf configure-space set-env
• 1.4.33: kf configure-space set-nodeselector
• 1.4.34: kf configure-space unset-buildpack-env
• 1.4.35: kf configure-space unset-env
• 1.4.36: kf configure-space unset-nodeselector
• 1.4.37: kf create-autoscaling-rule
• 1.4.38: kf create-job
• 1.4.39: kf create-route
• 1.4.40: kf create-service
• 1.4.41: kf create-service-broker
• 1.4.42: kf create-space
• 1.4.43: kf create-user-provided-service
• 1.4.44: kf debug
• 1.4.45: kf delete
• 1.4.46: kf delete-autoscaling-rules
• 1.4.47: kf delete-job
• 1.4.48: kf delete-job-schedule
• 1.4.49: kf delete-network-policy
• 1.4.50: kf delete-orphaned-routes
• 1.4.51: kf delete-route
• 1.4.52: kf delete-service
• 1.4.53: kf delete-service-broker
• 1.4.54: kf delete-space
• 1.4.55: kf disable-autoscaling
• 1.4.56: kf doctor
• 1.4.57: kf domains
• 1.4.58: kf enable-autoscaling
• 1.4.59: kf env
• 1.4.60: kf fix-orphaned-bindings
• 1.4.61: kf job-history
• 1.4.62: kf job-schedules
• 1.4.63: kf jobs
• 1.4.64: kf logs
• 1.4.65: kf map-route
• 1.4.66: kf marketplace
• 1.4.67: kf network-policies
• 1.4.68: kf network-policy
• 1.4.69: kf proxy
• 1.4.70: kf proxy-route
• 1.4.71: kf push
• 1.4.72: kf restage
• 1.4.73: kf restart
• 1.4.74: kf routes
• 1.4.75: kf run-job
• 1.4.76: kf run-task
• 1.4.77: kf scale
• 1.4.78: kf schedule-job
• 1.4.79: kf service
• 1.4.80: kf services
• 1.4.81: kf set-env
• 1.4.82: kf set-space-role
• 1.4.83: kf space
• 1.4.84: kf space-users
• 1.4.85: kf spaces
• 1.4.86: kf ssh
• 1.4.87: kf stacks
• 1.4.88: kf start
• 1.4.89: kf stop
• 1.4.90: kf target
• 1.4.91: kf tasks
• 1.4.92: kf terminate-task
• 1.4.93: kf unbind-route-service
• 1.4.94: kf unbind-service
• 1.4.95: kf unmap-route
• 1.4.96: kf unset-env
• 1.4.97: kf unset-space-role
• 1.4.98: kf update-autoscaling-limits
• 1.4.99: kf update-user-provided-service
• 1.4.100: kf vcap-services
• 1.4.101: kf version
• 1.4.102: kf wrap-v2-buildpack
• 1.4.103: kf xargs-apps
• 1.5: Migrate to Kubernetes
• 1.5.1: Build Container Images
• 1.6: Troubleshooting
• 1.6.1: Temporarily stop reconciliation
• 1.6.2: Troubleshoot Apps
• 1.6.3: Troubleshoot Builds
• 1.6.4: Troubleshoot ClusterServiceBrokers
• 1.6.5: Troubleshoot Routes
• 1.6.6: Troubleshoot ServiceBrokers
• 1.6.7: Troubleshoot ServiceInstanceBindings
• 1.6.8: Troubleshoot ServiceInstances
• 1.6.9: Troubleshoot SourcePackages
• 1.6.10: Troubleshoot Spaces
• 1.6.11: Troubleshoot Tasks
• 1.7: Examples
• 1.7.1: Add X-VCAP-REQUEST-ID HTTP Headers.
• 1.7.2: Deploy Docker apps with NFS UID/GID mapping.
• 1.7.3: Deploy Spring Cloud Config
• 1.7.4: Deploy Spring Music
• 1.7.5: Deploy With Offline Java Buildpack

1 - Documentation

Kf offers developers the Cloud Foundry experience while empowering operators to adopt declarative Kubernetes practices. It makes migrating Cloud Foundry workloads to Kubernetes straightforward, and most importantly, avoids major changes to developer workflows.

1.1 - Getting Started

1.1.1 - Getting Started Overview

With Kf, you can migrate your Cloud Foundry apps to Kubernetes without changing your developer workflows.

Use the following sections to get started deploying Cloud Foundry apps to Kf.

Get started with a quickstart

Use the quickstart to install Kf and deploy a simple test Cloud Foundry app. This introduces you to the basic steps you’d perform for most app deployments.

Get an introduction to key concepts

For an introduction to the value of Kf, as well as high-level overviews of the technology, see the following documents:

• To learn more about how Kf works, see Kf dependencies and architecture.
• For a side-by-side comparison of Cloud Foundry and Kf components, see Compare Cloud Foundry and Kf services.
• To learn how to use service brokers with Kf, see Service brokers.
• For more information about Kf security design, see the Security overview.

1.1.2 - Quickstart

In this quickstart, you will deploy a sample Cloud Foundry app on an existing Kf cluster.

Push an application

Prerequisites

The following are required to complete this section:

1. The kf CLI installed and in your path.

2. You have connected to the Kf Kubernetes cluster:

   gcloud container clusters get-credentials CLUSTER_NAME \
       --project=CLUSTER_PROJECT_ID \
       --zone=CLUSTER_LOCATION
   

3. The git CLI installed and in your path.

Prepare space

1. Create new space:

   kf create-space test-space
   

2. Target the space:

   kf target -s test-space
   

Push the Cloud Foundry test app

1. Clone the test-app repo.

   git clone https://github.com/cloudfoundry-samples/test-app go-test-app
   
   cd go-test-app
   

2. Push the app.

   kf push test-app
   

3. Get the application’s URL.

   kf apps
   

4. Open the URL in your browser where you should see the app running.

   

Clean up

These steps should return the cluster to the starting state.

1. Delete the application.

   kf delete test-app
   

2. Delete the Space.

   kf delete-space test-space
   

1.1.3 - Compare Cloud Foundry and Kf services

This document provides a side-by-side comparison of the various services available on Cloud Foundry (CF) and those that Kf integrates with on Google Cloud.

1.2 - Develop Applications on Kf

1.2.1 - Build and deploy applications

1.2.1.1 - Deploy an application

When pushing an app (via kf push) to Kf, there are three lifecycles that Kf uses to take your source code and allow it to handle traffic:

1. Source code upload
2. Build
3. Run

Source code upload

The first thing that happens when you kf push is the Kf CLI (kf) packages up your directory (either current or --path/-p) into a container and publishes it to the container registry configured for the Space. This is called the source container. The Kf CLI then creates an App type in Kubernetes that contains both the source image and configuration from the App manifest and push flags.

Ignore files during push

In many cases, you will not want to upload certain files during kf push (i.e., “ignore” them). This is where a .kfignore (or .cfignore) file can be used. Similar to a .gitignore file, this file instructs the Kf CLI which files to not include in the source code container.

To create a .kfignore file, create a text file named .kfignore in the base directory of your app (similar to where you would store the manifest file). Then populate it with a newline delimited list of files and directories you don’t want published. For example:

bin
.idea

This will tell the Kf CLI to not include anything in the bin or .idea directories.

Kf supports gitignore style syntax.

Build

The Build lifecycle is handled by a Tekton TaskRun. Depending on the flags that you provide while pushing, it will choose a specific Tekton Task. Kf currently has the following Tekton Tasks:

• buildpackv2
• buildpackv3
• kaniko

Kf tracks each TaskRun as a Build. If a Build succeeds, the resulting container image is then deployed via the Run lifecycle (described below).

More information can be found at Build runtime.

Run

The Run lifecycle is responsible for taking a container image and creating a Kubernetes Deployment.

It also creates:

• Istio Virtual Services
• Kubernetes Secrets

More information can be found at Build runtime.

Push timeouts

Kf supports setting an environment variable to instruct the CLI to time out while pushing apps. If set, the variables KF_STARTUP_TIMEOUT or CF_STARTUP_TIMEOUT are parsed as a golang style duration (for example 15m, 1h). If a value is not set, the push timeout defaults to 15 minutes.

1.2.1.2 - Get started with buildpacks

Kf supports a variety of buildpacks. This document covers some starter examples for using them.

Before you begin

• You should have Kf running on a cluster.
• You should have run kf target -s <space-name> to target your space.

Java (v2) buildpack

Use spring initializr to create a Java 8 maven project with a spring web dependency and JAR packaging. Download it, extract it, and once extracted you can generate a JAR.

./mvnw package

Push the JAR to Kf with the Java v2 buildpack.

kf push java-v2 --path target/helloworld-0.0.1-SNAPSHOT.jar

Java (v3) buildpack

Use spring initializr to create a Java 8 maven project with a spring web dependency and JAR packaging. Download it, extract it, and once extracted, push to Kf with the cloud native buildpack.

kf push java-v3 --stack org.cloudfoundry.stacks.cflinuxfs3

Python (v2) buildpack

Create a new directory with files as shown in the following structure.

tree
.
├── Procfile
├── requirements.txt
└── server.py

cat Procfile
web: python server.py

cat requirements.txt
Flask

cat server.py
from flask import Flask
import os

app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
  port = int(os.getenv("PORT", 8080))
  app.run(host='0.0.0.0', port=port)

Push the Python flask app using v2 buildpacks.

kf push python --buildpack python\_buildpack

Python (v3) buildpack

(same as above)

Push the Python flask app using cloud native buildpacks.

kf push pythonv3 --stack org.cloudfoundry.stacks.cflinuxfs3

Staticfile (v2) buildpack

Create a new directory that holds your source code.

Add an index.html file with this content.

<!DOCTYPE html>

<html lang="en">

<head><title>Hello, world!</title></head>

<body><h1>Hello, world!</h1></body>

</html>

Push the static content with the staticfile buildpack.

kf push staticsite --buildpack staticfile\_buildpack

1.2.1.3 - Reduce deployment risk with blue-green deployments

This page shows you how to deploy a new version of your application and migrate traffic over from an old to a new version.

Push the initial App

Use the Kf CLI to push the initial version of your App with any routes:

$ kf push app-v1 --route my-app.my-space.example.com

Push the updated App

Use the Kf CLI to push a new version of your App without any routes:

$ kf push app-v2 --no-route

Add routes to the updated App

Use the Kf CLI to bind all existing routes to the updated App with a weight of 0 to ensure that they don’t get any requests:

$ kf map-route app-v2 my-space.example.com --hostname my-app --weight 0

Shift traffic

Start shifting traffic from the old App to the updated App by updating the weights on the routes:

$ kf map-route app-v1 my-space.example.com --hostname my-app --weight 80
$ kf map-route app-v2 my-space.example.com --hostname my-app --weight 20

If the deployment is going well, you can shift more traffic by updating the weights again:

$ kf map-route app-v1 my-space.example.com --hostname my-app --weight 50
$ kf map-route app-v2 my-space.example.com --hostname my-app --weight 50

Complete traffic shifting

After you’re satisfied that the new service hasn’t introduced regressions, complete the rollout by shifting all traffic to the new instance:

$ kf map-route app-v1 my-space.example.com --hostname my-app --weight 0
$ kf map-route app-v2 my-space.example.com --hostname my-app --weight 100

Turn down the original App

After you’re satisfied that quick rollbacks aren’t needed, remove the original route and stop the App:

$ kf unmap-route app-v1 myspace.example.com --hostname my-app
$ kf stop app-v1

Or delete the App and all associated route mappings:

$ kf delete app-v1

1.2.1.4 - Compare V2 and V3 Buildpacks

A buildpack converts source code into an executable, and is used to deliver a simple, reliable, and repeatable way to create containers. Kf supports both V2 and V3 buildpacks, and it is important to understand the differences between them.

V2 buildpacks

Most Cloud Foundry applications already use V2 buildpacks. When using V2 buildpacks with Kf, the lifecycle binaries and the buildpacks are downloaded and configured from their git URLs. Kf then uses the lifecycle CLI to execute each buildpack against the source code.

Pros

• Ready out of the box without pipeline or code changes.

Cons

• Legacy buildpack supersceded by V3.
• Weaker performance and reliability. The Kf build pipeline requires more IO for V2 buildpacks.
• Fewer community resources.
• Kf only supports OSS git repos.

V3 buildpacks

V3 buildpacks are a Cloud Native Computing Foundation (CNCF) project with a well defined spec, a CLI (pack) and a growing community that is innovating around different languages and frameworks. Google Cloud also has its own set of OSS buildpacks.

V3 buildpacks have two overarching OCI containers:

• Builder image
• Run image

Builder image

The builder image is used while your source code is being built into a runnable container. The image has the necessary detect scripts and other utilities to compile source code.

Run image

The run image is the base image that a container is built on. This means that it is the base image that will run when the App executes.

Layers

V3 buildpacks use layers to compose the final container. Each buildpack included in a build is given the opportunity to manipulate the file system and environment variables of the App. This layering approach allows for buildpacks to be thinner and more generic.

V3 buildpacks are built on OCI containers. This requires that the V3 builder image be stored in a container registry that the Kf build pipeline has access to. The build pipeline uses the builder image to apply the underlying scripts to build the source code into a runnable container.

Pros

• Google supported builder and run image.
• Works with various CI/CD runtimes like Cloud Build.
• Growing community and buildpack registry.

Cons

• May require code/process updates. For example, the Java buildpack requires source code while the V2 buildpack requires a jar file.
• V3 buildpacks are newer and might require additional validation (is using community developed buildpacks).

Kf Stacks

View Stacks

When pushing an App, the build pipeline determines the buildpack based on the selected Stack (specified via the --stack flag or the manifest).

To see which Stacks are available in a Space first ensure a Space is targeted:

kf target -s myspace

The kf stacks subcommand can then be used to list the Stacks:

kf stacks

The output shows both V2 and V3 Stacks:

Getting stacks in Space: myspace
Version  Name                                Build Image                                                                                          Run Image
V2       cflinuxfs3                          cloudfoundry/cflinuxfs3@sha256:5219e9e30000e43e5da17906581127b38fa6417f297f522e332a801e737928f5      cloudfoundry/cflinuxfs3@sha256:5219e9e30000e43e5da17906581127b38fa6417f297f522e332a801e737928f5
V3       kf-v2-to-v3-shim                    gcr.io/kf-releases/v2-to-v3:v2.7.0                                                                   gcr.io/buildpacks/gcp/run:v1                                                                       This is a stack added by the integration tests to assert that v2->v3 shim works
V3       google                              gcr.io/buildpacks/builder:v1                                                                         gcr.io/buildpacks/gcp/run:v1                                                                       Google buildpacks (https://github.com/GoogleCloudPlatform/buildpacks)
V3       org.cloudfoundry.stacks.cflinuxfs3  cloudfoundry/cnb:cflinuxfs3@sha256:f96b6e3528185368dd6af1d9657527437cefdaa5fa135338462f68f9c9db3022  cloudfoundry/run:full-cnb@sha256:dbe17be507b1cc6ffae1e9edf02806fe0e28ffbbb89a6c7ef41f37b69156c3c2  A large Cloud Foundry stack based on Ubuntu 18.04

V2 to V3 Buildpack Migration

Kf provides a V3 stack to build applications that were built with standard V2 buildpacks, using a stack named kf-v2-to-v3-shim. The kf-v2-to-v3-shim stack is created following the standard V3 buildpacks API. A Google maintained builder image is created with each Kf release, following the standard buildpack process. The builder image aggregates a list of V3 buildpacks created by the same process used with the kf wrap-v2-buildpack command. The V3 buildpack images are created using the standard V2 buildpack images. It’s important to note that the V3 buildpacks do not contain the binary of the referenced V2 buildpacks. Instead, the V2 buildpack images are referenced, and the bits are downloaded at App build time (by running kf push).

At App build time, the V2 buildpack is downloaded from the corresponding git repository. When V3 detection runs, it delegates to the downloaded V2 detection script. For the first buildpack group that passes detection, it proceeds to the build step, which delegates the build execution to the downloaded V2 builder script.

The following V2 buildpacks are supported in the kf-v2-to-v3-shim stack:

Option 1: Migrate Apps built with standard V2 buildpacks

To build Apps with the kf-v2-to-v3-shim stack, use the following command:

kf push myapp --stack kf-v2-to-v3-shim

The kf-v2-to-v3-shim stack will automatically detect the runtime with the wrapped V2 buildpacks. The resulting App image is created using the V3 standard and build pipeline, but the builder of the equivalent V2 buildpack.

Option 2: Migrate Apps built with custom V2 buildpacks

Kf has a buildpack migration tool that can take a V2 buildpack and wrap it with a V3 buildpack. The wrapped buildpack can then be used anywhere V3 buildpacks are available.

kf wrap-v2-buildpack gcr.io/your-project/v2-go-buildpack https://github.com/cloudfoundry/go-buildpack --publish

This will create a buildpack image named gcr.io/your-project/v2-go-buildpack. It can then be used to create a builder by following the create a builder docs.

This subcommand uses the following CLIs transparently:

• go
• git
• pack
• unzip

1.2.1.5 - App Manifest

App manifests provide a way for developers to record their App’s execution environment in a declarative way. They allow Apps to be deployed consistently and reproducibly.

Format

Manifests are YAML files in the root directory of the App. They must be named manifest.yml or manifest.yaml.

Kf App manifests are allowed to have a single top-level element: applications. The applications element can contain one or more application entries.

Application fields

The following fields are valid for objects under applications:

† Unique to Kf

Docker fields

The following fields are valid for application.docker objects:

Route fields

The following fields are valid for application.routes objects:

Port fields

The following fields are valid for application.ports objects:

Metadata fields

The following fields are valid for application.metadata objects:

Probe fields

Probes allow a subset of functionality from Kubernetes probes.

A probe must contain one action and other settings.

TCPSocketAction fields

Describes an action based on TCP requests.

HTTPGetAction fields

Describes an action based on HTTP get requests.

Examples

Minimal App

This is a bare-bones manifest that will build an App by auto-detecting the buildpack based on the uploaded source, and deploy one instance of it.

---
applications:
- name: my-minimal-application

Simple App

This is a full manifest for a more traditional Java App.

---
applications:
- name: account-manager
  # only upload src/ on push
  path: src
  # use the Java buildpack
  buildpacks:
  - java
  env:
    # manually configure the buildpack's Java version
    BP_JAVA_VERSION: 8
    ENVIRONMENT: PRODUCTION
  # use less disk and memory than default
  disk_quota: 512M
  memory: 512M
  # Give the app a minimum of 2/10ths of a CPU
  cpu: 200m
  instances: 3
  # make the app listen on three routes
  routes:
  - route: accounts.mycompany.com
  - route: accounts.datacenter.mycompany.internal
  - route: mycompany.com/accounts
  # set up a longer timeout and custom endpoint to validate
  # when the app comes up
  timeout: 300
  health-check-type: http
  health-check-http-endpoint: /healthz
  # attach two services by name
  services:
  - customer-database
  - web-cache

Docker App

Kf can deploy Docker containers as well as manifest deployed App. These Docker Apps MUST listen on the PORT environment variable.

---
applications:
- name: white-label-app
  # use a pre-built docker image (must listen on $PORT)
  docker:
    image: gcr.io/my-company/white-label-app:123
  env:
    # add additional environment variables
    ENVIRONMENT: PRODUCTION
  disk_quota: 1G
  memory: 1G
  # Give the app a minimum of 2 full CPUs
  cpu: 2000m
  instances: 1
  routes:
  - route: white-label-app.mycompany.com

App with multiple ports

This App has multiple ports to expose an admin console, website, and SMTP server.

---
applications:
- name: b2b-server
  ports:
  - port: 8080
    protocol: http
  - port: 9090
    protocol: http
  - port: 2525
    protocol: tcp
  routes:
  - route: b2b-admin.mycompany.com
    appPort: 9090
  - route: b2b.mycompany.com
    # gets the default (first) port

Health check types

Kf supports three different health check types:

1. port (default)
2. http
3. process (or none)

port and http set a Kubernetes readiness and liveness probe that ensures the application is ready before sending traffic to it.

The port health check will ensure the port found at $PORT is being listened to. Under the hood Kf uses a TCP probe.

The http health check will use the configured value in health-check-http-endpoint to check the application’s health. Under the hood Kf uses an HTTP probe.

A process health check only checks to see if the process running on the container is alive. It does NOT set a Kubernetes readiness or liveness probe.

Known differences

The following are known differences between Kf manifests and CF manifests:

• Kf does not support deprecated CF manifest fields. This includes all fields at the root-level of the manifest (other than applications) and routing fields.
• Kf is missing support for the following v2 manifest fields:
  • docker.username
• Kf does not support auto-detecting ports for Docker containers.

1.2.1.6 - App runtime

The app runtime is the environment apps are executed in.

Environment variables

Environment variables are injected into the app at runtime by Kubernetes. Variables are added based on the following order, where later values override earlier ones with the same name:

1. Space (set by administrators)
2. App (set by developers)
3. System (set by Kf)

Kf provides the following system environment variables:

Service credentials from bound services get injected into Apps via the VCAP_SERVICES environment variable. The variable is a valid JSON object with the following structure.

VCAPServices

A JSON object where the keys are Service labels and the values are an array of VCAPService. The array represents every bound service with that label. User provided services are placed under the label user-provided.

Example

{
  "mysql": [...],
  "postgresql": [...],
  "user-provided": [...]
}

VCAPService

This type represents a single bound service instance.

Example

{
  "binding_name": string,
  "instance_name": string,
  "name": string,
  "label": string,
  "tags": string[],
  "plan": string,
  "credentials": object
}

Fields

VCAP_APPLICATION

TheVCAP_APPLICATION environment variable is a JSON object containing metadata about the App.

Example

{
  "application_id": "12345",
  "application_name": "my-app",
  "application_uris": ["my-app.example.com"],
  "limits": {
    "disk": 1024,
    "mem": 256
  },
  "name": "my-app",
  "process_id": "12345",
  "process_type": "web",
  "space_name": "my-ns",
  "uris": ["my-app.example.com"]
}

Fields

Missing Fields

Some fields in VCAP_APPLICATION that are in Cloud Foundry are currently not supported in Kf.

Besides CF-specific and deprecated fields (cf_api, host, users) the fields that are not supported in Kf are:

• application_version (identical to version)
• organization_id
• organization_name
• space_id
• start (identical to started_at)
• started_at_timestamp (identical to state_timestamp)

1.2.1.7 - Build runtime

The Build runtime is the environment Apps are built in.

Environment variables

Environment variables are injected into the Build at runtime. Variables are added based on the following order, where later values override earlier ones with the same name:

1. Space (set by administrators)
2. App (set by developers)
3. System (set by Kf)

Kf provides the following system environment variables to Builds:

1.2.2 - Backing services

1.2.2.1 - Backing Services Overview

Backing services are any processes that the App contacts over the network during its operation. In traditional operating systems, these services could have been accessed over the network, a UNIX socket, or could even be a sub-process. Examples include the following:

• Databases — for example: MySQL, PostgreSQL
• File storage — for example: NFS, FTP
• Logging services — for example: syslog endpoints
• Traditional HTTP APIs — for example: Google Maps, WikiData, Parcel Tracking APIs

Connecting to backing services over the network rather than installing them all into the same machine allows developers to focus on their App, independent security upgrades for different components, and flexibility to swap implementations.

Backing services in Kf

Kf supports two major types of backing services:

• Managed services: These services are created by a service broker and are tied to the Kf cluster.

• User-provided services: These services are created outside of Kf, but get can be bound to apps in the same way as managed services.

1.2.2.2 - Use managed services

Find a service

Use the kf marketplace command to find a service you want to use in your App. Running the command without arguments will show all the service classes available. A service class represents a specific type of service e.g. a MySQL database or a Postfix SMTP relay.

$ kf marketplace
5 services can be used in Space "test", use the --service flag to list the plans for a service

Broker      Name        Space      Status  Description
minibroker  mariadb                Active  Helm Chart for mariadb
minibroker  mongodb                Active  Helm Chart for mongodb
minibroker  mysql                  Active  Helm Chart for mysql
minibroker  postgresql             Active  Helm Chart for postgresql
minibroker  redis                  Active  Helm Chart for redis

Service classes can have multiple plans available. A service plan generally corresponds to a version or pricing tier of the software. You can view the plans for a specific service by supplying the service name with the marketplace command:

$ kf marketplace --service mysql
Name    Free  Status  Description
5-7-14  true  Active  Fast, reliable, scalable, and easy to use open-source relational database system.
5-7-27  true  Active  Fast, reliable, scalable, and easy to use open-source relational database system.
5-7-28  true  Active  Fast, reliable, scalable, and easy to use open-source relational database system.

Provision a service

Once you have identified a service class and plan to provision, you can create an instance of the service using kf create-service:

$ kf create-service mysql 5-7-28 my-db
Creating service instance "my-db" in Space "test"
Waiting for service instance to become ready...
Success

Services are provisioned into a single Space. You can see the services in the current Space by running kf services:

$ kf services
Listing services in Space: "test"
Name   ClassName  PlanName  Age   Ready  Reason
my-db  mysql      5-7-28    111s  True   <nil>

You can delete a service using kf delete-service:

$ kf delete-service my-db

Bind a service

Once a service has been created, you can bind it to an App, which will inject credentials into the App so the service can be used. You can create the binding using kf bind-service:

$ kf bind-service my-app my-db
Creating service instance binding "binding-my-app-my-db" in Space "test"
Waiting for service instance binding to become ready...
Success

You can list all bindings in a Space using the kf bindings command:

$ kf bindings
Listing bindings in Space: "test"
Name                  App     Service  Age  Ready
binding-my-app-my-db  my-app  my-db    82s  True

Once a service is bound, restart the App using kf restart and the credentials will be in the VCAP_SERVICES environment variable.

You can delete a service binding with the kf unbind-service command:

$ kf unbind-service my-app my-db

1.2.2.3 - Configure user-provided services

Users can leverage services that aren’t available in the marketplace by creating user-provided service instances. Once created, user-provided service instances behave like managed service instances created through kf create-service. Creating, listing, updating, deleting, binding, and unbinding user-provided services are all supported in Kf.

Create a user-provided service instance

The name given to a user-provided service must be unique across all service instances in a Space, including services created through a service broker.

Deliver service credentials to an app

A user-provided service instance can be used to deliver credentials to an App. For example, a database admin can have a set of credentials for an existing database managed outside of Kf, and these credentials include the URL, port, username, and password used to connect to the database.

The admin can create a user-provided service with the credentials and the developer can bind the service instance to the App. This allows the credentials to be shared without ever leaving the platform. Binding a service instance to an App has the same effect regardless of whether the service is a user-provided service or a marketplace service.

The App is configured with the credentials provided by the user, and the App runtime environment variable VCAP_SERVICES is populated with information about all of the bound services to that App.

A user-provided service can be created with credentials and/or a list of tags.

kf cups my-db -p '{"username":"admin", "password":"test123", "some-int-val": 1, "some-bool": true}' -t "comma, separated, tags"

This will create the user-provided service instance my-db with the provided credentials and tags. The credentials passed in to the -p flag must be valid JSON (either inline or loaded from a file path).

To deliver the credentials to one or more Apps, the user can run kf bind-service.

Suppose we have an App with one bound service, the user-provided service my-db defined above. The VCAP_SERVICES environment variable for that App will have the following contents:

{
  "user-provided": [
    {
      "name": "my-db",
      "instance_name": "my-db",
      "label": "user-provided",
      "tags": [
        "comma",
        "separated",
        "tags"
      ],
      "credentials": {
        "username": "admin",
        "password": "test123",
        "some-int-val": 1,
        "some-bool": true
      }
    }
  ]
}

Update a user-provided service instance

A user-provided service can be updated with the uups command. New credentials and/or tags passed in completely overwrite existing ones. For example, if the user created the user-provided service my-db above, called kf bind-service to bind the service to an App, then ran the command.

kf uups my-db -p '{"username":"admin", "password":"new-pw", "new-key": "new-val"}'

The updated credentials will only be reflected on the App after the user unbinds and rebinds the service to the App. No restart or restage of the App is required. The updated VCAP_SERVICES environment variable will have the following contents:

{
  "user-provided": [
    {
      "name": "my-db",
      "instance_name": "my-db",
      "label": "user-provided",
      "tags": [
        "comma",
        "separated",
        "tags"
      ],
      "credentials": {
        "username": "admin",
        "password": "new-pw",
        "new-key": "new-val"
      }
    }
  ]
}

The new credentials overwrite the old credentials, and the tags are unchanged because they were not specified in the update command.

1.2.2.4 - User Provided Service Templates

Before you begin

• Ensure your service is running and accessible on the same network running your Kf cluster.
• Ensure you have targeted the Space where you want to create the service.

Create the user-provided instance

The following examples use the most common parameters used by applications to autoconfigure services. Most libraries use tags to find the right bound service and a URI to connect.

MySQL

MySQL libraries usually expect the tag mysql and the following parameters:

uri

Example mysql://username:password@host:port/dbname. The MySQL documentation can help with creating a URI string. The port is usually 3306.

username

The connection username, required by some libraries even if included in uri.

password

The connection password, required by some libraries even if included in uri.

kf cups service-instance-name \
  -p '{"username":"my-username", "password":"my-password", "uri":"mysql://my-username:my-password@mysql-host:3306/my-db"}' \
  -t "mysql"

RabbitMQ

RabbitMQ libraries usually expect the tag rabbitmq and the following parameters:

uri

Example amqp://username:password@host:port/vhost?query. The RabbitMQ documentation can help with creating a URI string. The port is usually 5672.

Example:

kf cups service-instance-name \
  -p '{"uri":"amqp://username:password@host:5672"}' \
  -t "rabbitmq"

Redis

Redis libraries usually expect the tag redis and the following parameters:

uri

Example redis://:password@host:port/uery. The IANA Redis URI documentation can help with creating a URI string. The port is usually 6379.

Example for Redis with no AUTH configured:

kf cups service-instance-name \
  -p '{"uri":"redis://redis-host:6379"}' \
  -t "redis"

Example for Redis with AUTH configured:

kf cups service-instance-name \
  -p '{"uri":"redis://:password@redis-host:6379"}' \
  -t "redis"

Bind your App

Once the user-provided service has been created, you can bind your App to the user provided service by name, just like a managed service:

kf bind-service application-name service-instance-name

What’s next

• Learn about how the credentials are injected into your app.

1.2.2.5 - Configure NFS volumes

Kf supports mounting NFS volumes using the kf marketplace.

Prerequisites

• Your administrator must have completed the NFS platform setup guide.

Create an NFS service instance

Run kf marketplace to see available services. The built-in NFS service appears on the list if NFS is enabled on the platform.

Broker           Name  Namespace  Description
nfsvolumebroker  nfs              mount nfs shares

Mount an external filesystem

Create a service instance

To mount to an existing NFS service:

kf create-service nfs existing SERVICE-INSTANCE-NAME -c '{"share":"SERVER/SHARE", "capacity":"CAPACITY"}'

Replace variables with your values.

• SERVICE-INSTANCE-NAME is the name you want for this NFS volume service instance.
• SERVER/SHARE is the NFS address of your server and share.
• CAPACITY uses the Kubernetes quantity format.

Confirm that the NFS volume service appears in your list of services. You can expect output similar to this example:

$ kf services
...
Listing services in Space: demo-space
Name                Type      ClassName         PlanName  Age    Ready  Reason
filestore-nfs       volume    nfs               existing  6s     True   <nil>
...

Bind your service instance to an App

To bind an NFS service instance to an App, run:

kf bind-service YOUR-APP-NAME SERVICE-NAME -c '{"uid":"2000","gid":"2000","mount":"MOUNT-PATH","readonly":true}'

Replace variables with your values.

• YOUR-APP-NAME is the name of the App for which you want to use the volume service.

• SERVICE-NAME is the name of the volume service instance you created in the previous step.

• uid:UID and gid:GID specify the directory permissions of the mounting share.

• MOUNT-PATH is the path the volume should be mounted to within your App.

• (Optional) "readonly":true is an optional JSON string that creates a read-only mount. By default, Volume Services mounts a read-write file system.

  Note: Your App automatically restarts when the NFS binding changes.

You can list all bindings in a Space using the kf bindings command. You will see output similar to this example:

$ kf bindings
...
Listing bindings in Space: demo-space
Name                                     App           Service             Age  Ready
binding-spring-music-filestore-nfs       spring-music  filestore-nfs       71s  True
...

Access the volume service from your App

To access the volume service from your App, you must know which file path to use in your code. You can view the file path in the details of the service binding, which are visible in the environment variables for your App.

View environment variables for your App:

kf vcap-services YOUR-APP-NAME

Replace YOUR-APP-NAME with the name of your App.

The following is example output of the kf vcap-services command:

$ kf vcap-services YOUR-APP-NAME
{
  "nfs": [
    {
      "instance_name": "nfs-instance",
      "name": "nfs-instance",
      "label": "nfs",
      "tags": [],
      "plan": "existing",
      "credentials": {
        "capacity": "1Gi",
        "gid": 2000,
        "mount": "/test/mount",
        "share": "10.91.208.210/test",
        "uid": 2000
      },
      "volume_mounts": [
        {
          "container_dir": "/test/mount",
          "device_type": "shared",
          "mode": "rw"
        }
      ]
    }
  ]
}

Use the properties under volume_mounts for any information required by your App.

1.2.3 - Configure and Use Service Accounts

By default, all applications in Kf are assigned a unique Kubernetes Service Account (KSA) named sa-<APP_NAME>. Kf uses this KSA as the “user” it runs application instances and tasks under.

Each App KSA receives a copy of the container registry credentials used by the Space’s build KSA so Kf apps can pull container images that were created during kf push.

Using the service account

Kuberenetes Pods (the building blocks of Apps and Tasks) automatically receive a JWT for the KSA mounted in the container:

$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt
namespace
token

• ca.crt The Kubernetes control plane’s certificate.
• namespace The Kubernetes namespace of the workload.
• token A Base64 encoded JWT for the Kf App’s Service Account.

Below is an example of what the JWT looks like, note that:

• It expires and needs to be periodically refreshed from disk.
• It’s audience is only valid within the Kubernetes cluster.

{
    "aud": [
        "<KUBERNETES_CLUSTER_URI>"
    ],
    "exp": 3600,
    "iat": 0,
    "iss": "<KUBERNETES_CLUSTER_URI>",
    "kubernetes.io": {
        "namespace": "<SPACE_NAME>",
        "pod": {
            "name": "<APP_NAME>-<RANDOM_SUFFIX>",
            "uid": "<APP_GUID>"
        },
        "serviceaccount": {
            "name": "sa-<APP_NAME>",
            "uid": "<SERVICE_ACCOUNT_GUID>"
        },
        "warnafter": 3500
    },
    "nbf": 0,
    "sub": "system:serviceaccount:<SPACE_NAME>:sa-<APP_NAME>"
}

You can use this credential to connect to the Kubernetes control plane listed in the issuer (iss) field.

Customizing the service account

You want to use a different service account than the default one Kf provides, for example to:

• Allow blue/green apps to have the same identity.
• Use Kf with a federated identity system.
• Provide custom image pull credentials for a specific app.

You can enable this by adding the apps.kf.dev/service-account-name annotation to your app manifest. The value should be the name of the KSA you want the application and tasks to use.

Example:

applications:
- name: my-app
  metadata:
    annotations:
      "apps.kf.dev/service-account-name": "override-sa-name"

Limitations:

• Only KSAs within the same Kubernetes namespace–corresponding to a Kf Space–are allowed.
• The KSA must exist and be readable by Kf, otherwise the app will not deploy.
• The KSA or the cluster must have permission to pull the application’s container images.

Additional resources

• If you use GKE, learn how to inject apps with a Google Service Account credential.
• Learn how to use federated identity in GCP to allow authenticating KSAs to GCP infrastructure.

1.2.4 - Scaling

1.2.4.1 - Scaling overview

Kf leverages the Kubernetes Horizontal Pod Autoscaler (HPA) to automatically scale the number of Pods in a App. When autoscaling is enabled for an App, an HPA object is created and bound to the App object. It then dynamically calculates the target scale and sets it for the App.

Kf Apps are also compatible with HPA policies created outside of Kf.

How Kf scaling works

The number of Pods that are deployed for a Kf App is controlled by its underlying Deployment object’s replicas field. The target number of Deployment replicas is set through the App’s replicas field.

Scaling can be done manually with the kf scale command. This command is disabled when autoscaling is enabled to avoid conflicting targets.

How the Kubernetes Horizontal Pod Autoscaler works

The Horizontal Pod Autoscaler (HPA) is implemented as a Kubernetes API resource (the HPA object) and a control loop (the HPA controller) which periodically calculates the number of desired replicas based on current resource utilization. The HPA controller then passes the number to the target object that implements the Scale subresource. The actual scaling is delegated to the underlying object and its controller. You can find more information in the Kubernetes documentation.

How the Autoscaler determines when to scale

Periodically, the HPA controller queries the resource utilization against the metrics specified in each HorizontalPodAutoscaler definition. The controller obtains the metrics from the resource metrics API for each Pod. Then the controller calculates the utilization value as a percentage of the equivalent resource request. The desired number of replicas is then calculated based on the ratio of current percentage and desired percentage. You can read more about the autoscaling algorithm in the Kubernetes documentation.

Metrics

Kf uses HPA v1 which only supports CPU as the target metric.

How the Kubernetes Horizontal Autoscaler works with Kf

When autoscaling is enabled for a Kf App, the Kf controller will create an HPA object based on the scaling limits and rules specified on the App. Then the HPA controller fetches the specs from the HPA object and scales the App accordingly.

The HPA object will be deleted if Autoscaling is disabled or if the corresponding App is deleted.

1.2.4.2 - Manage Autoscaling

Kf supports two primary autoscaling modes:

• Built-in autosacling similar to Cloud Foundry.
• Advanced autoscaling through the Kubernetes Horizontal Pod Autoscaler (HPA).

Built-in autoscaling

Kf Apps can be automatically scaled based on CPU usage. You can configure autoscaling limits for your Apps and the target CPU usage for each App instance. Kf automatically scales your Apps up and down in response to demand.

By default, autoscaling is disabled. Follow the steps below to enable autoscaling.

View Apps

You can view the autoscaling status for an App using the kf apps command. If autoscaling is enabled for an App, Instances includes the autoscaling status.

$ kf apps

Name   Instances              Memory  Disk  CPU
app1   4 (autoscaled 4 to 5)  256Mi   1Gi   100m
app2   1                      256Mi   1Gi   100m

Autoscaling is enabled for app1 with min-instances set to 4 and max-instances set to 5. Autoscaling is disabled for app2.

Update autoscaling limits

You can update the instance limits using the kf update-autoscaling-limits command.

kf update-autoscaling-limits app-name min-instances max-instances

Create autoscaling rule

You can create autoscaling rules using the kf create-autoscaling-rule command.

kf create-autoscaling-rule app-name CPU min-threshold max-threshold

Delete autoscaling rules

You can delete all autoscaling rules with the kf delete-autoscaling-rule command. Kf only supports one autoscaling rule.

kf delete-autoscaling-rules app-name

Enable and disable autoscaling

Autoscaling can be enabled by using enable-autoscaling and disabled by using disable-autoscaling. When it is disabled, the configurations, including limits and rules, are preserved.

kf enable-autoscaling app-name

kf disable-autoscaling app-name

Advanced autoscaling

Kf Apps support the Kubernetes Horizontal Pod Autoscaler interface and will therefore work with HPAs created using kubectl.

Kubernetes HPA policies are less restrictive than Kf’s built-in support for autoscaling.

They include support for:

• Scaling on memory, CPU, or disk usage.
• Scaling based on custom metrics, such as traffic load or queue length.
• Scaling on multiple metrics.
• The ability to tune reactivity to smooth out rapid scaling.

Using custom HPAs with apps

You can follow the Kubernetes HPA walkthrough to learn how to set up autoscalers.

When you create the HPA, make sure to set the scaleTargetRef to be your application:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-scaler
  namespace: SPACE_NAME
spec:
  scaleTargetRef:
    apiVersion: kf.dev/v1alpha1
    kind: App
    name: APP_NAME
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 60

Caveats

• You shouldn’t use Kf autoscaling with an HPA.
• When you use an HPA, kf apps will show the current number of instances, it won’t show that the App is being autoscaled.

1.2.4.3 - Managing Resources for Apps

When you create an app, you can optionally specify how much of each resource an instance of the application will receive when it runs.

Kf simplifies the Kubernetes model of resources and provides defaults that should work for most I/O bound applications out of the box.

Resource types

Kf supports three types of resources, memory, CPU, and ephemeral disk.

• Memory specifies the amount of RAM an application receives when running. If it exceeds this amount then the container is restarted.
• Ephemeral disk specifies how much an application can write to a local disk. If an application exceeds this amount then it may not be able to write more.
• CPU specifies the number of CPUs an application receives when running.

Manifest

Resources are specified using four values in the manifest:

• memory sets the guaranteed minimum an app will receive and the maximum it’s permitted to use.
• disk_quota sets the guaranteed minimum an app will receive and the maximum it’s permitted to use.
• cpu sets the guaranteed minimum an app will receive.
• cpu-limit sets the maximum CPU an app can use.

Example:

applications:
- name: "example"
  disk_quota: 512M
  memory: 512M
  cpu: 200m
  cpu-limit: 2000m

Defaults

Memory and ephemeral storage are both set to 1Gi if not specified.

CPU defaults to one of the following

• 1/10th of a CPU if the platform operator hasn’t overridden it.
• A CPU value proportionally scaled by the amount of memory requested.
• A minimum CPU value set by the platform operator.

Resource units

Memory and disk

Cloud Foundry used the units T, G, M, and K to represent powers of two. Kubernetes uses the units Ei, Pi, Gi, Mi, and Ki for the same.

Kf allows you to specify memory and disk in either units.

CPU

Kf and Kubernetes use the unit m for CPU, representing milli-CPU cores (thousandths of a core).

Sidecar overhead

When Kf schedules your app’s container as a Kubernetes Pod, it may bundle additional containers to your app to provide additional functionality. It’s likely your application will also have an Istio sidecar which is responsible for networking.

These containers will supply their own resource requests and limits and are overhead associated with running your application.

Best practices

• All applications should set memory and disk quotas.
• CPU intensive applications should set a CPU request and limit to guarantee they’ll have the resources they need without starving other apps.
• I/O bound applications shouldn’t set a CPU limit so they can burst during startup.

Additional reading

• Learn how Kubernetes defines resources and schedules apps based on resource use

1.2.5 - Service discovery

This document is an overview of Kubernetes DNS-based service discovery and how it can be used with Kf.

When to use Kubernetes service discovery with Kf

Kubernetes service discovery can be used by applications that need to locate backing services in a consistent way regardless of where the application is deployed. For example, a team might want to use a common URI in their configuration that always points at the local SMTP gateway to decouple code from the environment it ran in.

Service discovery helps application teams by:

• Reducing the amount of per-environment configuration.
• Decoupling client and server applications.
• Allowing applications to be portable to new environments.

You can use Kubernetes service discovery when:

• Applications use their container’s DNS configurations to resolve hosts.
• Applications are deployed with their backing services in the same Kubernetes cluster or namespace.
• Backing services have an associated Kubernetes service. Kf creates these for each app.
• Kubernetes NetworkPolicies allow traffic between an application and the Kubernetes service it needs to communicate with. Kf creates these policies in each Kf space.

You should not use Kubernetes service discovery if:

• Applications need to failover between multiple clusters.
• You override the DNS resolver used by your application.
• Applications need specific types of load balancing.

How Kubernetes service discovery works

Kubernetes service discovery works by modifying the DNS configuration of containers running on a Kubernetes node. When an application looks up an unqualified domain name, the local DNS resolver will first attempt to resolve the name in the local cluster.

Domains without multiple parts will be resolved against the names of Kubernetes services in the container’s namespace. Each Kf app creates a Kubernetes service with the same name. If two Kf apps ping and pong were deployed in the same Kf space, then ping could use the URL http://pong to send traffic to the other service.

Domains with a single dot will be resolved against the Kubernetes services in the Kubernetes namespace with the same name as the label after the dot. For example, if there was a PostgreSQL database with a customers service in the database namespace, an application in another namespace could resolve it using postgres://customers.database.

How to use service discovery with Kf

Kubernetes DNS based service discovery can be used in any Kf app. Each Kf app creates a Kubernetes service of the same name, and each Kf space creates a Kubernetes namespace with the same name.

• Refer to a Kf app in the current space using protocol://app-name.
• Refer to a Kf app in a different space using protocol://app-name.space-name.
• Refer to a Kf app in the current space listening on a custom port using protocol://app-name:port.
• Refer to a Kf app in a different space listening a custom port using protocol://app-name.space-name:port.

Best practices

Applications that are going to be the target of DNS based service discovery should have frequent health checks to ensure they are rapidly added and removed from the pool of hosts that accept connections.

Applications using DNS based service discovery should not cache the IP addresses of the resolved services because they are not guaranteed to be stable.

If environment specific services exist outside of the cluster, they can be resolved using Kubernetes DNS if you set up ExternalName Kubernetes services. These Kubernetes services provide the same resolution capabilities, but return a CNAME record to redirect requests to an external authority.

Comparison to Eureka

Eureka is an open source client-side load-balancer created by Netflix. It is commonly used as part of the Spring Cloud Services service broker. Eureka was built to be a regional load balancer and service discovery mechanism for services running in an environment that caused frequent disruptions to workloads leading to unstable IP addresses.

Eureka is designed as a client/server model. Clients register themselves with the server indicating which names they want to be associated with and periodically send the server heartbeats. The server allows all connected clients to resolve names.

In general, you should use Kubernetes DNS rather than Eureka in Kubernetes for the following reasons:

• DNS works with all programming languages and applications without the need for libraries.
• Your application’s existing health check will be reused reducing combinations of errors.
• Kubernetes manages the DNS server, allowing you to rely on fewer dependencies.
• Kubernetes DNS respects the same policy and RBAC constraints as the rest of Kubernetes.

There are a few times when deploying a Eureka server would be advantageous:

• You need service discovery across Kubernetes and VM based applications.
• You need client based load-balancing.
• You need independent health checks.

What’s next

• Read more about service discovery in GKE.
• Learn about Service Directory, a managed offering similar to Eureka.

1.2.6 - Debugging workloads

Kf uses Kubernetes under the hood to schedule application workloads onto a cluster. Ultimately, every workload running on a Kubernetes cluster is scheduled as a Pod, but the Pods may have different properties based on the higher level abstractions that schedule them.

Once you understand how to debug Pods, you can debug any running workload on the cluster including the components that make up Kf and Kubernetes.

General debugging

Kubernetes divides most resources into Namespaces. Each Kf Space creates a Namespace with the same name. If you’re debugging a Kf resource, you’ll want to remember to set the -n or --namespace CLI flag on each kubectl command you run.

Finding a Pod

You can list all the Pods in a Namespace using the command:

kubectl get pods -n NAMESPACE

This will list all Pods in the Namespace. You’ll often want to filter these using a label selector like the following:

kubectl get pods -n NAMESPACE -l "app.kubernetes.io/component=app-server,app.kubernetes.io/name=echo"

You can get the definition of a Pod using a command like the following:

kubectl get pods -n NAMESPACE POD_NAME -o yaml

Understand Kubernetes objects

Most Kubernetes objects follow the same general structure. Kubernetes also has extensive help documentation for each type of object (including Kf’s).

If you need to quickly look up what a field is for, use the kubectl explain command with the object type and the path to the field you want documentation on.

$ kubectl explain pod.metadata.labels
KIND:     Pod
VERSION:  v1

FIELD:    labels <map[string]string>

DESCRIPTION:
     Map of string keys and values that can be used to organize and categorize
     (scope and select) objects. May match selectors of replication controllers
     and services. More info: http://kubernetes.io/docs/user-guide/labels

When you run kubectl get RESOURCE_TYPE RESOURCE_NAME -oyaml you’ll see the stored version of the object. An annotated example of an Pod running the instance of an App is below:

apiVersion: v1
kind: Pod
metadata:
    # Annotations hold security information and configuration for the 
    # object.
    annotations:
        kubectl.kubernetes.io/default-container: user-container
        kubectl.kubernetes.io/default-logs-container: user-container
        sidecar.istio.io/inject: "true"
        traffic.sidecar.istio.io/includeOutboundIPRanges: '*'
    # Labels hold information useful for filtering resources.
    labels:
        # Kf sets many of these labels to help find Pods.
        app.kubernetes.io/component: app-server
        app.kubernetes.io/managed-by: kf
        app.kubernetes.io/name: echo
        kf.dev/networkpolicy: app
    name: echo-6b759c978b-zwrt8
    namespace: development
    # Contains the object(s) that "own" this resource, this usually
    # means the ones that were responsible for creating it.
    ownerReferences:
    - apiVersion: apps/v1
        blockOwnerDeletion: true
        controller: true
        kind: ReplicaSet
        name: echo-6b759c978b
        uid: 7f0ee42d-f1e8-4c4f-b0c8-f0c5d7f27a0a
    # The ID of the resource, if deleted and re-created the ID will
    # change.
    uid: 0d49d5d7-afa4-4904-9f69-f98ce1923745
spec:
    # Contains the desired state of the object, written by a human or
    # one of the metadata.ownerReferences.
    # Omitted for brevity.
status:
    # Contains the state of the object as written by the controller.
    # Omitted for brevity.

When you see an object with an metadata.ownerReferences set, you can run kubectl get again to find that object’s information all the way back up until you find the root object responsible for creating it. In this case the chain would look like Pod -> ReplicaSet -> Deployment -> App.

Getting logs

You can get logs for a specific Pod using kubectl logs:

kubectl logs -c user-container -n NAMESPACE POD_NAME

You can find a list of containers on the Pod in the .spec.containers.name field:

kubectl get pods -n NAMESPACE -o jsonpath='{.spec.containers[*].name}' POD_NAME

Port forwarding

You can port-forward to a specific port using kubectl port-forward.

# Bind remote port 8080 to local port 8080.
kubectl port-forward -n NAMESPACE POD_NAME 8080

# Bind remote port 8080 to local port 9000.
kubectl port-forward -n NAMESPACE POD_NAME 9000:8080

Open a shell

You can open a shell to a container using kubectl exec.

kubectl exec --stdin --tty -n NAMESPACE -c user-container POD_NAME -- /bin/bash

Kf Specifics

The following sections show specific information about Kf types that schedule Pods.

App Pods

Label selector

All Apps:

app.kubernetes.io/component=app-server,app.kubernetes.io/managed-by=kf

Specific App (replace APP_NAME):

app.kubernetes.io/component=app-server,app.kubernetes.io/managed-by=kf,app.kubernetes.io/name=APP_NAME

Expected containers

• user-container Container running the application code.
• istio-proxy Connects the application code to the virtual network.

Ownership hierarchy

Each resource will have a metadata.ownerReferences to the resource below it:

1. Kubernetes Pod Runs a single instance of the application code.
2. Kubernetes ReplicaSet Schedules all the instances for one version of an App.
3. Kubernetes Deployment Manages rollouts and scaling of multiple versions of the App.
4. Kf App Orchestrates routing, rollouts, service bindings, builds, etc. for an App.

Build Pods

Label selector

All Builds:

app.kubernetes.io/component=build,app.kubernetes.io/managed-by=kf

Builds for a specific App (replace APP_NAME):

app.kubernetes.io/component=build,app.kubernetes.io/managed-by=kf,app.kubernetes.io/name=APP_NAME

Specific Build for an App (replace APP_NAME and TASKRUN_NAME):

app.kubernetes.io/component=build,app.kubernetes.io/managed-by=kf,app.kubernetes.io/name=APP_NAME,tekton.dev/taskRun=TASKRUN_NAME

Expected containers

• step-.* Container(s) that execute different steps of the build process. Specific steps depend on the type of build.
• istio-proxy (Optional) Connects the application code to the virtual network.

Ownership hierarchy

Each resource will have a metadata.ownerReferences to the resource below it:

1. Kubernetes Pod Runs the steps of the build.
2. Tekton TaskRun Schedules the Pod, ensures it runs to completion once, and cleans it up once done.
3. Kf Build Creates a TaskRun with the proper steps to build an App from source.
4. Kf App Creates Builds with app-specific information like environment variables.

Task Pods

Label selector

All Tasks:

app.kubernetes.io/component=task,app.kubernetes.io/managed-by=kf

Tasks for a specific App (replace APP_NAME):

app.kubernetes.io/component=task,app.kubernetes.io/managed-by=kf,app.kubernetes.io/name=APP_NAME

Specific Task for an App (replace APP_NAME and TASKRUN_NAME):

app.kubernetes.io/component=task,app.kubernetes.io/managed-by=kf,app.kubernetes.io/name=APP_NAME,tekton.dev/taskRun=TASKRUN_NAME

Expected containers

• step-user-container Container running the application code.
• istio-proxy Connects the application code to the virtual network.

Ownership hierarchy

1. Kubernetes Pod Runs an instance of the application code.
2. Tekton TaskRun Schedules the Pod, ensures it runs to completion once, and cleans it up once done.
3. Kf Task Creates a TaskRun with the proper step to run a one-off Task.
4. Kf App Creates Builds with app-specific information like environment variables.

Next steps

• Explore the Kubernetes guide to debugging applications.
• Understand how to write label selectors.

1.2.7 - Configure routes and domains

This page describes how routes and domains work in Kf, and how developers and administrators configure routes and domains for an App deployed on Kf cluster.

You must create domain and routes to give external access to your application.

Internal routing

Kf apps can communicate internally with other apps in the cluster directly using a service mesh without leaving the cluster network. By default, all traffic on the service mesh is encrypted using mutual TLS.

All apps deployed in the Kf cluster come with an internal endpoint configured by default. You can use the address app-name.space-name.svc.cluster.local for internal communication between apps. To use this internal address no extra steps are required. Mutual TLS is enabled by default for internal routes. Note that this internal address is only accessible from the pods running the apps and not accessible from outside the cluster.

App load balancing

Traffic is routed by Istio to healthy instances of an App using a round-robin policy. Currently, this policy can’t be changed.

Route capabilities

Routes tell the cluster’s ingress gateway where to deliver traffic and what to do if no Apps are available on the given address. By default, if no App is available on a Route and the Route receives a request it returns an HTTP 503 status code.

Routes are comprised of three parts: host, domain, and path. For example, in the URI payroll.mydatacenter.example.com/login:

• The host is payroll
• The domain is mydatacenter.example.com
• The path is /login

Routes must contain a domain, but the host and path is optional. Multiple Routes can share the same host and domain if they specify different paths. Multiple Apps can share the same Route and traffic will be split between them. This is useful if you need to support legacy blue/green deployments. If multiple Apps are bound to different paths, the priority is longest to shortest path.

Warning: Kf doesn’t currently support TCP port-based routing. You must use a Kubernetes LoadBalancer if you want to expose a TCP port to the Internet. Ports are available on the cluster internal App address <app-name>.<space>.

Manage routes

The following sections describe how to use the kf CLI to manage Routes.

List routes

Developers can list Routes for the current Space using the kf routes command.

$ kf routes
Getting Routes in Space: my-space
Found 2 Routes in Space my-space

HOST    DOMAIN       PATH    APPS
echo    example.com  /       echo
*       example.com  /login  uaa

Create a route

Developers can create Routes using the kf create-route command.

# Create a Route in the targeted Space to match traffic for myapp.example.com/*
$ kf create-route example.com --hostname myapp

# Create a Route in the Space myspace to match traffic for myapp.example.com/*
$ kf create-route -n myspace example.com --hostname myapp

# Create a Route in the targeted Space to match traffic for myapp.example.com/mypath*
$ kf create-route example.com --hostname myapp --path /mypath

# You can also supply the Space name as the first parameter if you have
# scripts that rely on the old cf style API.
$ kf create-route myspace example.com --hostname myapp # myapp.example.com

After a Route is created, if no Apps are bound to it then an HTTP 503 status code is returned for any matching requests.

Map a route to your app

Developers can make their App accessible on a Route using the kf map-route command.

$ kf map-route MYAPP mycluster.example.com --hostname myapp --path mypath

Unmap a route

Developers can remove their App from being accessible on a Route using the kf unmap-route command.

$ kf unmap-route MYAPP mycluster.example.com --hostname myapp --path mypath

Delete a route

Developers can delete a Route using the kf delete-route command.

$ kf delete-route mycluster.example.com --hostname myapp --path mypath

Deleting a Route will stop traffic from being routed to all Apps listening on the Route.

Manage routes declaratively in your app manifest

Routes can be managed declaratively in your app manifest file. They will be created if they do not yet exist.

---
applications:
- name: my-app
  # ...
  routes:
  - route: example.com
  - route: www.example.com/path

You can read more about the supported route properties in the manifest documentation.

Routing CRDs

There are four types that are relevant to routing:

• VirtualService
• Route
• Service
• App

Each App has a Service, which is an abstract name given to all running instances of your App. The name of the Service is the same as the App. A Route represents a single external URL. Routes constantly watch for changes to Apps, when an App requests to be added to a Route, the Route updates its list of Apps and then the VirtualService. A VirtualService represents a single domain and merges a list of all Routes in a Space that belong to that domain.

Istio reads the configuration on VirtualServices to determine how to route traffic.

1.2.8 - Tasks

1.2.8.1 - Tasks Overview

About Tasks

Unlike Apps which run indefinitely and restart if the process terminates, Tasks run a process until it completes and then stop. Tasks are run in their own containers and are based on the configuration and binary of an existing App.

Tasks are not accessible from routes, and should be used for one-off or scheduled recurring work necessary for the health of an application.

Use cases for Tasks

• Migrating a database
• Running a batch job (scheduled/unscheduled)
• Sending an email
• Transforming data (ETL)
• Processing data (upload/backup/download)

How Tasks work

Tasks are executed asynchronously and run independently from the parent App or other Tasks running on the same App. An App created for running Tasks does not have routes created or assigned, and the Run lifecycle is skipped. The Source code upload and Build lifecycles still proceed and result in a container image used for running Tasks after pushing the App (see App lifecycles at Deploying an Application).

The lifecycle of a Task is as follows:

1. You push an App for running tasks with the kf push APP_NAME --task command.
2. You run a Task on the App with the kf run-task APP_NAME command. Task inherits the environment variables, service bindings, resource allocation, start-up command, and security groups bound to the App.
3. Kf creates a Tekton PipelineRun with values from the App and parameters from the run-task command.
4. The Tekton PipelineRun creates a Kubernetes Pod which launches a container based on the configurations on the App and Task.
5. Task execution stops (Task exits or is terminated manually), the underlying Pod is either stopped or terminated. Pods of stopped Tasks are preserved and thus Task logs are accessible via the kf logs APP_NAME --task command.
6. If you terminate a Task before it stops, the Tekton PipelineRun is cancelled (see Cancelling a PipelineRun), the underlying Pod together with the logs are deleted. The logs of termianted Tasks are delivered to the cluster level logging streams if configured (e.g. Stackdriver, Fluentd).
7. If the number of Tasks run on an App is greater than 500, the oldest Tasks are automatically deleted.

Tasks retention policy

Tasks are created as custom resources in the Kubernetes cluster, therefore, it is important not to exhaust the space of the underlying etcd database. By default, Kf only keeps the latest 500 Tasks per each App. Once the number of Tasks reach 500, the oldest Tasks (together with the underlying Pods and logs) will be automatically deleted.

Task logging and execution history

Any data or messages the Task outputs to STDOUT or STDERR is available by using the kf logs APP_NAME --task command. Cluster level logging mechanism (such as Stackdriver, Fluentd) will deliver the Task logs to the configured logging destination.

Scheduling Tasks

As described above, Tasks can be run asynchronously by using the kf run-task APP_NAME command. Alternatively, you can schedule Tasks for execution by first creating a Job using the kf create-job command, and then scheduling it with the kf schedule-job JOB_NAME command. You can schedule that Job to automatically run Tasks on a specified unix-cron schedule.

How Tasks are scheduled

Create and schedule a Job to run the Task. A Job describes the Task to execute and automatically manages Task creation.

Tasks are created on the schedule even if previous executions of the Task are still running. If any executions are missed for any reason, only the most recently missed execution are executed when the system recovers.

Deleting a Job deletes all associated Tasks. If any associated Tasks were still in progress, they are forcefully deleted without running to completion.

Tasks created by a scheduled Job are still subject to the Task retention policy.

Differences from PCF Scheduler

PCF Scheduler allows multiple schedules for a single Job while Kf only supports a single schedule per Job. You can replicate the PCF Scheduler behavior by creating multiple Jobs, one for each schedule.

1.2.8.2 - Run Tasks

You can execute short-lived workflows by running them as Tasks in Kf. Tasks are run under Apps, meaning that each Task must have an associated App. Each Task execution uses the build artifacts from the parent App. Because Tasks are short-lived, the App is not deployed as a long-running application, and no routes should be created for the App or the Task.

Push an App for running Tasks

1. Clone the test-app repo repo:

   git clone https://github.com/cloudfoundry-samples/test-app test-app
   
   cd test-app
   

2. Push the App.

   Push the App with the kf push APP_NAME --task command. The --task flag indicates that the App is meant to be used for running Tasks, and thus no routes are created on the App, and it is not deployed as a long-running application:

   kf push test-app --task
   

3. Confirm that no App instances or routes were created by listing the App:

   kf apps
   

   Notice that the App is not started and has no URLs:

   Listing Apps in Space: test-space
   Name                     Instances  Memory  Disk  CPU   URLs
   test-app                 stopped    1Gi     1Gi   100m  <nil>
   

Run a Task on the App

When you run a Task on the App, you can optionally specify a start command by using the --command flag. If no start command is specified, it uses the start command specified on the App. If the App doesn’t have a start command specified, it looks up the CMD configuration of the container image. A start command must exist in order to run the Task successfully.

kf run-task test-app --command "printenv"

You see something similar to this, confirming that the Task was submitted:

Task test-app-gd8dv is submitted successfully for execution.

The Task name is automatically generated, prefixed with the App name, and suffixed with an arbitrary string. The Task name is a unique identifier for Tasks within the same cluster.

Specify Task resource limits

Resource limits (such as CPU cores/Memory limit/Disk quota) can be specified in the App (during kf push) or during the kf run-task command. The limits specified in the kf run-task command take prededence over the limits specified on the App.

To specify resource limits in an App, you can use the --cpu-cores, --memory-limit, and --disk-quota flags in the kf push command:

kf push test-app --command "printenv" --cpu-cores=0.5 --memory-limit=2G --disk-quota=5G --task

To override these limits in the App, you can use the --cpu-cores, --memory-limit, and --disk-quota flags in the kf run-task command:

kf run-task test-app --command "printenv" --cpu-cores=0.5 --memory-limit=2G --disk-quota=5G

Specify a custom display name for a Task

You can optionally use the --name flag to specify a custom display name for a Task for easier identification/grouping:

$ kf run-task test-app --command "printenv" --name foo
Task test-app-6swct is submitted successfully for execution.

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-6swct    3   foo                1m     21s       True       <nil>

Manage Tasks

View all Tasks of an App with the kf tasks APP_NAME command:

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-gd8dv    1   test-app-gd8dv     1m     21s       True       <nil>

Cancel a Task

Cancel an active Task by using the kf terminate-task command:

• Cancel a Task by Task name:

  $ kf terminate-task test-app-6w6mz
  Task "test-app-6w6mz" is successfully submitted for termination
  

• Or cancel a Task by APP_NAME + Task ID:

  $ kf terminate-task test-app 2
  Task "test-app-6w6mz" is successfully submitted for termination
  

Cancelled Tasks have PipelineRunCancelled status.

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-gd8dv    1   test-app-gd8dv     1m     21s       True       <nil>
test-app-6w6mz    2   test-app-6w6mz     38s    11s       False      PipelineRunCancelled

View Task logs

View logs of a Task by using the kf logs APP_NAME --task command:

$ kf logs test-app --task

1.2.8.3 - Schedule Tasks

You can execute short-lived workflows by running them as Tasks. Running Tasks describes how to run Tasks under Apps.

You can also schedule Tasks to run at recurring intervals specified using the unix-cron format. With scheduled Tasks, you first push an App running the Task as you do with an unscheduled Task, and then create a Job to schedule the Task.

You can define a schedule so that your Task runs multiple times a day or on specific days and months.

Push an App for running scheduled Tasks

1. Clone the test-app repo:

git clone https://github.com/cloudfoundry-samples/test-app test-app

cd test-app

1. Push the App.

   Push the App with the kf push APP_NAME --task command. The --task flag indicates that the App is meant to be used for running Tasks, and thus no routes will be created on the App and it will not be deployed as a long-running application.

   kf push test-app --task
   

2. Confirm that no App instances or routes were created by listing the App:

   kf apps
   

   Notice that the App is not started and has no URLs:

   Listing Apps in Space: test-space
   Name                     Instances  Memory  Disk  CPU   URLs
   test-app                 stopped    1Gi     1Gi   100m  <nil>
   

Create a Job

To run a Task on a schedule, you must first create a Job that describes the Task:

kf create-job test-app test-job "printenv"

The Job starts suspended or unscheduled, and does not create Tasks until it is manually executed by kf run-job or scheduled by kf schedule-task.

Run a Job manually

Jobs can be run ad hoc similar to running Tasks by kf run-task. This option can be useful for testing the Job before scheduling or running as needed in addition to the schedule.

kf run-job test-job

This command runs the Task defined by the Job a single time immediately.

Schedule a Job

To schedule the Job for execution, you must provide a unix-cron schedule in the kf schedule-job command:

kf schedule-job test-job "* * * * *"

This command triggers the Job to automatically create Tasks on the specified schedule. In this example a Task runs every minute.

You can update a Job’s schedule by running kf schedule-task with a new schedule. Jobs in Kf can only have a single cron schedule. This differs from the PCF Scheduler, which allows multiple schedules for a single Job. If you require multiple cron schedules, then you can achieve that with multiple Jobs.

Manage Jobs and schedules

View all Jobs, both scheduled and unscheduled, in the current Space by using the kf jobs command:

$ kf jobs
Listing Jobs in Space: test space
Name               Schedule    Suspend  LastSchedule  Age  Ready  Reason
test-job           * * * * *   <nil>    16s           2m   True   <nil>
unscheduled-job    0 0 30 2 *  true     16s           2m   True   <nil>

Additionally, you can view only Jobs that are actively scheduled with the kf job-schedules command.

$ kf job-schedules
Listing job schedules in Space: test space
Name           Schedule   Suspend  LastSchedule  Age  Ready  Reason
test-job       * * * * *  <nil>    16s           2m   True   <nil>

Notice how the unscheduled-job is not listed in the kf job-schedules output.

Cancel a Job’s schedule

You can stop a scheduled Job with the kf delete-job-schedule command:

kf delete-job-schedule test-job

This command suspends the Job and stops it from creating Tasks on the previous schedule. The Job is not deleted and can be scheduled again by kf schedule-job to continue execution.

Delete a Job

The entire Job can be deleted with the kf delete-job command:

kf delete-job test-job

This command deletes the Job and all Tasks that were created by the Job, both scheduled and manual executions. If any Tasks are still running, this command forcefully deletes them.

If you want to ensure that running Tasks are not interrupted, then first delete the Jobs schedule with kf delete-job-schedule, wait for all Tasks to complete, and then delete the job by calling kf delete-job.

1.3 - Operate and Maintain Kf

1.3.1 - Service brokers

1.3.1.1 - Service Brokers Overview

Kf supports binding and provisioning apps to Open Service Broker (OSB) services.

Any compatible service broker can be installed using the create-service-broker command, but only the Kf Cloud Service Broker is fully supported.

Special services such as syslog drains, volume services, route services, service keys, and shared services aren’t currently supported.

1.3.1.2 - About Kf Cloud Service Broker

The Kf Cloud Service Broker is a Service Broker bundle that includes the open source Cloud Service Broker and Google Cloud Brokerpak. It is made available as a public Docker image and ready to deploy as a Kubernetes service in Kf clusters. Once the Kf Cloud Service Broker service is deployed in a cluster, developers can provision Google Cloud backing services through the Kf Cloud Service Broker service, and bind the backing services to Kf Apps.

Requirements

• Kf Cloud Service Broker requires a MySQL instance and a service account for accessing the MySQL instance and Google Cloud backing services to be provisioned. Connection from the Kf Cloud Service Broker to the MySQL instance goes through the Cloud SQL Auth Proxy.
• Requests to access Google Cloud services (for example: Cloud SQL for MySQL or Cloud Memorystore for Redis) are authenticated via Workload Identity.

Override Brokerpak defaults

Brokerpaks are essentially a Terraform plan and related dependencies in a tar file. You can inspect the Terraform plans to see what the defaults are, and then you can tell Kf Cloud Service Broker to override them when creating new services.

For example, the Terraform plan for MySQL includes a variable called authorized_network. If not overridden, the default VPC will be used. If you’d like to override the default, you can pass that during service creation. Here are some examples:

1. Override the compute region config.

 kf create-service csb-google-postgres small spring-music-postgres-db -c '{"config":"YOUR_COMPUTE_REGION"}'

1. Override the authorized_network and compute region config.

 kf create-service csb-google-postgres small spring-music-postgres-db -c '{"config":"YOUR_COMPUTE_REGION","authorized_network":"YOUR_CUSTOM_VPC_NAME"}'

You can learn more by reading the MySQL Plans and Configs documentation.

Architecture

The following Kf Cloud Service Broker architecture shows how instances are created.

• The Kf Cloud Service Broker (CSB) is installed in its own namespace.
• On installation, a MySQL instance must be provided to persist business logic used by Kf Cloud Service Broker. Requests are sent securely from the Kf Cloud Service Broker pod to the MySQL instance via the MySQL Auth Proxy.
• On service provisioning, a Kf Service custom resource is created. The reconciler of the Kf Service provisions Google Cloud backing services using the Open Service Broker API.
• When a request to provision/deprovision backing resources is received, Kf Cloud Service Broker sends resource creation/deletion requests to the corresponding Google Cloud service, and these requests are authenticated with Workload Identity. It also persists the business logics (e.g. mapping of Kf services to backing services, service bindings) to the MySQL instance.
• On backing service creation success, the backing service is bound to an App via VCAP_SERVICES.

What’s next?

• Deploy Kf Cloud Service Broker.
• Learn how to list and provision services.

{% endblock %}

1.3.1.3 - Deploy Kf Cloud Service Broker

This page shows you how to deploy Kf Cloud Service Broker and use it to provision or deprovision backing resources. Read about the concepts and architecture to learn more about the Kf Cloud Service Broker.

Create environment variables

• Linux

  export PROJECT_ID=YOUR_PROJECT_ID
  export CLUSTER_PROJECT_ID=YOUR_PROJECT_ID
  export CSB_IMAGE_DESTINATION=YOUR_DOCKER_REPO_CSB_PATH
  export CLUSTER_NAME=kf-cluster
  export INSTANCE_NAME=cloud-service-broker
  export COMPUTE_REGION=us-central1
  

• Windows PowerShell

  Set-Variable -Name PROJECT_ID -Value YOUR_PROJECT_ID
  Set-Variable -Name CLUSTER_PROJECT_ID -Value YOUR_PROJECT_ID
  Set-Variable -Name CSB_IMAGE_DESTINATION=YOUR_DOCKER_REPO_CSB_PATH
  Set-Variable -Name CLUSTER_NAME -Value kf-cluster
  Set-Variable -Name INSTANCE_NAME -Value cloud-service-broker
  Set-Variable -Name COMPUTE_REGION -Value us-central1
  

Build the broker

First you’ll want to download and build the broker and push it to your container registry:

git clone --single-branch --branch main https://github.com/google/kf.git kf
cd kf/samples/cloud-service-broker
docker build --tag ${CSB_IMAGE_DESTINATION} .
docker push ${CSB_IMAGE_DESTINATION}

Set up the Kf Cloud Service Broker database

1. Create a MySQL instance.

   gcloud sql instances create ${INSTANCE_NAME} --cpu=2 --memory=7680MB --require-ssl --region=${COMPUTE_REGION}
   

2. Create a database named servicebroker in the MySQL instance.

   gcloud sql databases create servicebroker -i ${INSTANCE_NAME}

3. Create a username and password to be used by the broker.

   gcloud sql users create csbuser -i ${INSTANCE_NAME} --password=csbpassword

Set up a Google Service Account for the broker

1. Create a Google Service Account.

    gcloud iam service-accounts create csb-${CLUSTER_NAME}-sa \
        --project=${CLUSTER_PROJECT_ID} \
        --description="GSA for CSB at ${CLUSTER_NAME}" \
        --display-name="csb-${CLUSTER_NAME}"

2. Grant roles/cloudsql.client permissions to the Service Account. This is required to connect the service broker pod to the CloudSQL for MySQL instance through the CloudSQL Proxy.

    gcloud projects add-iam-policy-binding ${CLUSTER_PROJECT_ID} \
        --member="serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --role="roles/cloudsql.client"

3. Grant additional Google Cloud permissions to the Service Account.

    gcloud projects add-iam-policy-binding ${CLUSTER_PROJECT_ID} \
        --member="serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --role="roles/compute.networkUser"

    gcloud projects add-iam-policy-binding ${CLUSTER_PROJECT_ID} \
        --member="serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --role="roles/cloudsql.admin"

    gcloud projects add-iam-policy-binding ${CLUSTER_PROJECT_ID} \
        --member="serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --role="roles/redis.admin"

4. Verify the permissions.

    gcloud projects get-iam-policy ${CLUSTER_PROJECT_ID} \
        --filter='bindings.members:serviceAccount:"CSB_SERVICE_ACCOUNT_NAME"' \
        --flatten="bindings[].members"

Set up Workload Identity for the broker

1. Bind the Google Service Account with the Kubernetes Service Account.

    gcloud iam service-accounts add-iam-policy-binding "csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --project=${CLUSTER_PROJECT_ID} \
        --role="roles/iam.workloadIdentityUser" \
        --member="serviceAccount:${CLUSTER_PROJECT_ID}.svc.id.goog[kf-csb/csb-user]"

2. Verify the binding.

    gcloud iam service-accounts get-iam-policy "csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com" \
        --project=${CLUSTER_PROJECT_ID}

Set up a Kubernetes Secret to share configuration with the broker

1. Create a config.yml file.

   cat << EOF >> ./config.yml
   gcp:
     credentials: ""
     project: ${CLUSTER_PROJECT_ID}
   
   db:
     host: 127.0.0.1
     password: csbpassword
     user: csbuser
     tls: false
   api:
     user: servicebroker
     password: password
   EOF
   

2. Create the kf-csb namespace.

   kubectl create ns kf-csb
   

3. Create the Kubernetes Secret.

   kubectl create secret generic csb-secret --from-file=config.yml -n kf-csb
   

Install the Kf Cloud Service Broker

1. Copy the kf-csb-template.yaml into kf-csb.yaml for working:

   cp kf-csb-template.yaml /tmp/kf-csb.yaml
   

2. Edit /tmp/kf-csb.yaml and replace placeholders with final values. In the example below, sed is used.

   sed -i "s|<GSA_NAME>|csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com|g" /tmp/kf-csb.yaml
   
   sed -i "s|<INSTANCE_CONNECTION_NAME>|${CLUSTER_PROJECT_ID}:${COMPUTE_REGION}:${INSTANCE_NAME}|g" /tmp/kf-csb.yaml
   
   sed -i "s|<DB_PORT>|3306|g" /tmp/kf-csb.yaml
   
   sed -i "s|<CSB_IMAGE_DESTINATION>|${CSB_IMAGE_DESTINATION}|g" /tmp/kf-csb.yaml
   

3. Apply yaml for Kf Cloud Service Broker.

   kubectl apply -f /tmp/kf-csb.yaml
   

4. Verify the Cloud Service Broker installation status.

   kubectl get pods -n kf-csb
   

Create a service broker

kf create-service-broker cloud-service-broker servicebroker password http://csb-controller.kf-csb/

Validate installation

Check for available services in the marketplace.

kf marketplace

If everything is installed and configured correctly, you should see the following:

$ kf marketplace

Broker                Name                          Namespace  Description
cloud-service-broker  csb-google-bigquery                      A fast, economical and fully managed data warehouse for large-scale data analytics.
cloud-service-broker  csb-google-dataproc                      Dataproc is a fully-managed service for running Apache Spark and Apache Hadoop clusters in a simpler, more cost-efficient way.
cloud-service-broker  csb-google-mysql                         Mysql is a fully managed service for the Google Cloud Platform.
cloud-service-broker  csb-google-postgres                      PostgreSQL is a fully managed service for the Google Cloud Platform.
cloud-service-broker  csb-google-redis                         Cloud Memorystore for Redis is a fully managed Redis service for the Google Cloud Platform.
cloud-service-broker  csb-google-spanner                       Fully managed, scalable, relational database service for regional and global application data.
cloud-service-broker  csb-google-stackdriver-trace             Distributed tracing service
cloud-service-broker  csb-google-storage-bucket                Google Cloud Storage that uses the Terraform back-end and grants service accounts IAM permissions directly on the bucket.

Clean up

1. Delete cloud-service-broker.

   kf delete-service-broker cloud-service-broker
   

2. Delete CSB components.

   kubectl delete ns kf-csb
   

3. Delete the broker’s database instance.

    gcloud sql instances delete ${INSTANCE_NAME} --project=${CLUSTER_PROJECT_ID}

4. Remove the IAM policy bindings.

    gcloud projects remove-iam-policy-binding ${CLUSTER_PROJECT_ID} \
    --member='serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com' \
    --role=roles/cloudsql.client

    gcloud projects remove-iam-policy-binding ${CLUSTER_PROJECT_ID} \
    --member='serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com' \
    --role=roles/compute.networkUser

    gcloud projects remove-iam-policy-binding ${CLUSTER_PROJECT_ID} \
    --member='serviceAccount:csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com' \
    --role=roles/redis.admin

5. Remove the GSA.

    gcloud iam service-accounts delete csb-${CLUSTER_NAME}-sa@${CLUSTER_PROJECT_ID}.iam.gserviceaccount.com \
      --project=${CLUSTER_PROJECT_ID}

1.3.1.4 - Set up NFS platform

Kf supports Kubernetes native NFS, and exposes them with a dedicated nfsvolumebroker service broker for developers to consume. This broker has an nfs service offering which has a service plan named existing.

Use kf marketplace to see the service offering:

$ kf marketplace
...
Broker           Name  Namespace  Description
nfsvolumebroker  nfs              mout nfs shares
...

Use kf marketplace -s nfs to see the service plan:

$ kf marketplace -s nfs
...
Broker           Name      Free  Description
nfsvolumebroker  existing  true  mount existing nfs server
...

Requirements

You need an NFS volume that can be accessed by your Kubernetes cluster. For example Cloud Filestore which Google’s managed NFS solution that provides access to clusters in the same gcloud project.

Prepare NFS

If you have an existing NFS service, you can use that. If you want a Google managed NFS service, create a Filestore instance and Kf will automaticlaly configure the cluster to use it.

Warning: You only need to create the NFS instance. Kf will create relevant Kubernetes objects, including PersistentVolume and PersistentVolumeClaims. Do not manually mount the volume.

What’s next?

• Configure NFS volumes

1.3.2 - Customizing Kf

1.3.2.1 - Customizing Overview

Kf supports customizing some cluster wide configurations by manipulating the kfsystem custom rsource.

1.3.2.2 - Cluster Configuration

Kf uses a Kubernetes configmap named config-defaults in the kf namespace to store cluster wide configuration settings. This document explains its structure and fields.

Structure of the config-defaults configmap

The configmap contains three types of key/value pairs in the .data field:

• Comment keys prefixed by _ contain examples, notes, and warnings.
• String keys contain plain text values.
• Object keys contain a JSON or YAML value that has been encoded as a string.

Example:

_note: "This is some note"
stringKey: "This is a string key that's not encoded as JSON or YAML."
objectKey: |
  - "These keys contain nested YAML or JSON."
  - true
  - 123.45  

Example section

The example section under the _example key contains explanations for other fields and examples. Changes to this section have no effect.

Space container registry

The spaceContainerRegistry property is a plain text value that specifies the default container registry each space uses to store built images.

Example:

spaceContainerRegistry: gcr.io/my-project

Space cluster domains

The spaceClusterDomains property is a string encoded YAML array of domain objects.

Each space in the cluster adds all items in the array to its list of domains that developers can bind their apps to.

Example:

spaceClusterDomains: |
  # Support canonical and vanity domains
  - domain: $(SPACE_NAME).prod.example.com
  - domain: $(SPACE_NAME).kf.us-east1.prod.example.com

  # Using a dynamic DNS resolver
  - domain: $(SPACE_NAME).$(CLUSTER_INGRESS_IP).nip.io

  # Creating an internal domain only visible within the cluster
  - domain: $(SPACE_NAME)-apps.internal
    gatewayName: kf/internal-gateway  

Buildpacks V2 lifecycle builder

The buildpacksV2LifecycleBuilder property contains the version of the Cloud Foundry builder binary used execute buildpack v2 builds.

The value is a Git reference. To use a specific version, append an @ symbol followed by a Git SHA to the end.

Example:

buildpacksV2LifecycleBuilder: "code.cloudfoundry.org/buildpackapplifecycle/builder@GIT_SHA"

Buildpacks V2 lifecycle launcher

The buildpacksV2LifecycleLauncher property contains the version of the Cloud Foundry launcher binary built into every buildpack V2 application.

The value is a Git reference. To use a specific version, append an @ symbol followed by a Git SHA to the end.

Example:

buildpacksV2LifecycleLauncher: "code.cloudfoundry.org/buildpackapplifecycle/launcher@GIT_SHA"

Buildpacks V2 list

The spaceBuildpacksV2 property is a string encoded YAML array that holds an ordered list of default buildpacks that are used to build applications compatible with the V2 buildpacks process.

Stacks V2 list

The spaceBuildpacksV2 property is a string encoded YAML array that holds an ordered list of stacks that can be used with Cloud Foundry compatible builds.

Stacks V3 list

The spaceStacksV3 property is a string encoded YAML array that holds an ordered list of stacks that can be used with Cloud Native Buildpack builds.

Example:

spaceStacksV3: |
  - name: heroku-18
    description: The official Heroku stack based on Ubuntu 18.04
    buildImage: heroku/pack:18-build
    runImage: heroku/pack:18
    nodeSelector:
       kubernetes.io/os: windows  

Default to V3 Stack

The spaceDefaultToV3Stack property contains a quoted value true or false indicating whether spaces should use V3 stacks if a user doesn’t specify one.

Feature flags

The featureFlags property contains a string encoded YAML map of feature flags that can enable and disable features of Kf.

Flag names that aren’t supported by Kf will be ignored.

Example:

featureFlags: |
  disable_custom_builds: false
  enable_dockerfile_builds: true
  enable_some_feature: true  

ProgressDeadlineSeconds

ProgressDeadlineSeconds contains a configurable quoted integer indicating the maximum allowed time between state transition and reaching a stable state before provisioning or deprovisioning when pushing an application. The default value is 600 seconds.

TerminationGracePeriodSeconds

The TerminationGracePeriodSeconds contains a configurable quoted integer indicating the time between when the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. The default value is 30 seconds.

1.3.2.3 - Customizing Kf Features

Build Retention

You can control how many Kf Builds are kept before being garbage collected.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildRetentionCount', 'value': 1}]"

Task Retention

You can control how many Kf Tasks are kept before being garbage collected.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/taskRetentionCount', 'value': 1}]"

Enable or Disable the Istio Sidecar

If you do not require the Istio sidecar for the Build pods, then they can be disabled by setting the value to true. Enable by setting the value to false.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildDisableIstioSidecar', 'value': true}]"

Enable or Disable Routing Retries

Allows enabling/disbaling retries in the VirtualServices that route traffic to Apps. Kf leaves this value unset by default and it’s inherited from Istio.

Istio’s default retry mechanism attempts to make up for instability inherent in service meshes, however allowing retries requires the contents of the payload to be buffered within Envoy. This may fail for large payloads and the buffering will need to be disabled at the expense of some stability.

Values for routeDisableRetries:

• false Inherit Istio’s retry settings. (Default)
• true Set retries to 0.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/routeDisableRetries','value':true}]"

Enable or Disable Routing Hosts Ignoring Any Port Numbers

Allows enabling/disabling routing hosts ignoring any specified port number. By default hosts are matched using the exact value specified in the route Host (e.g a request with a Host header value of example.com:443 does not match with preconfigured route Host example.com). By enabling, ports are ignored and only hosts are used (e.g example.com:443 matches example.com)

Note: Feature will only work in clusters with istio 1.15+. In older versions it will function as though it where disabled.

Values for routeHostIgnoringPort:

• false Will match the Host header in request to the exact configured route Host. (Default)
• true Will use regexp to match to the configured route Host ignoring any port specified in the Host header of the request.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/routeHostIgnoringPort','value':true}]"

Build Pod Resource Limits

The default pod resource size can be increased from the default to accommodate very large builds. The units for the value are in Mi or Gi.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildPodResources', 'value': {'limits': {'memory': '234Mi'}}}]"

Read Kubernetes container resource docs for more information about container resource management.

Robust Build Snapshots

Kf uses Kaniko to build the final application containers in the v2 buildpack lifecycle. To produce image layers, Kaniko needs to take “snapsohts” of the image which requires iterating over all files in the image and checking if they’ve changed.

The fast mode checks for file attribute changes (like timestamps and size) and the slow mode checks full file hashes to determine if a file changed.

Kf apps aren’t expected to overwrite operating system files in their build, so fast mode should be used to reduce disk pressure.

Values for buildKanikoRobustSnapshot:

• false Will use a fast snapshot mode for v2 builds. (Default)
• true Will use a robust snapshot mode for v2 builds to catch uncommon cases.

kubectl patch \
kfsystem kfsystem \
--type='json' \
-p="[{'op': 'replace', 'path': '/spec/kf/config/buildKanikoRobustSnapshot', 'value': true}]"

Self Signed Certificates for Service Brokers

If you want to use self signed certificates for TLS (https instead of http) for the service broker URL, the Kf controller requires the CA certificate. To configure Kf for this scenario, create an immutable Kubernetes secret in the kf namespace and update the kfsystem.spec.kf.config.secrets.controllerCACerts.name object to point to it.

1. Create a secret to store the self-signed certificate.

    kubectl create secret generic cacerts -nkf --from-file /path/to/cert/certs.pem
    

2. Make the secret immutable.

    kubectl patch -nkf secret cacerts \
        --type='json' \
        -p="[{'op':'add','path':'/immutable','value':true}]"
    

3. Update kfsystem to point to the secret.

   kubectl patch \
     kfsystem kfsystem \
     --type='json' \
     -p="[{'op':'add','path':'/spec/kf/config/secrets','value':{'controllerCACerts':{'name':'<var>cacerts</var>'}}}]"
   

Set CPU minimums and ratios

Application default CPU ratios and minimums can be set in the operator.

Values are set in CPU units. Units are typically expressed in millicpus (m), or thousandths of a CPU.

The spec.kf.config.appCPUMin property specifies a minimum amount of CPU per application, even if the developer has specified less.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appCPUMin','value':'<var>200m</var>'}]"

The spec.kf.config.appCPUPerGBOfRAM property specifies a default amount of CPU to give each app per GB or RAM requested.

You can choose different approaches based on the desired outcome:

• Choose the ratio of CPU to RAM for the cluster’s nodes if you want to maximize utilization.
• Choose a ratio of 1 CPU to 4 GB of RAM which typically works well for I/0 or memory bound web applications.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appCPUPerGBOfRAM','value':'<var>250m</var>'}]"

Set buildpacks using git tags

Buildpacks can support pinning by using git tags instead of automatically sourcing the latest buildpack from a git repository.

Add a new buildpack as follows and use a git tag to specify which version of the buildpack the app should use. Otherwise the buildpack will default to the latest version.

For example, to pin Golang buildpack version 1.9.49 do:

kubectl patch \
kfsystem kfsystem \
--type='json' \
 -p='[{"op":"add","path":"data/spec/kf/config/spaceBuildpacksV2","value":[{"name":"go_buildpack_v1.9.49","url":"https://github.com/cloudfoundry/go-buildpack.git#v1.9.49"}]}]'

This command will add the following to the config-defaults configmaps resource:

data:
  SpaceBuildpacksV2: |
    - name: go_buildpack_v1.9.49
      url: https://github.com/cloudfoundry/go-buildpack.git#v1.9.49

The kubectl patch command will replace all the existing buildpacks in the config-defaults configmaps. If the user would like the existing buildpacks to remain, these too need to be included in the command.

To get the list of existing buildpacks in the configmaps run the following command:

kubectl describe configmaps config-defaults -n kf

Set ProgressDeadlineSeconds

ProgressDeadlineSeconds can be set in the kfsystem operator.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/progressDeadlineSeconds','value':100}]"

Set TerminationGracePeriodSeconds

TerminationGracePeriodSeconds can be set in the kfsystem operator.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/terminationGracePeriodSeconds','value':200}]"

Enable/Disable App Start Command Lookup

Allows enabling/disbaling start command lookup in the App reconciler. This behavior requires the reconciler to fetch container configuration for every app from the container registry and enables displaying the start command on kf push and in kf app.

Enabling this behavior on a large cluster may make the reconcilation times for Apps slow.

Values for appDisableStartCommandLookup:

• false Enable start command lookup. (Default)
• true Disable start command lookup.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appDisableStartCommandLookup','value':true}]"

Enable/Disable Kubernetes service Account (KSA) overrides

Allows enabling/disbaling the ability to override the Kubernetes Service Account for Apps via annotation.

Values for appEnableServiceAccountOverride:

• false Don’t allow overriding service account. (Default)
• true Allow developers to override KSAs for their Apps.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/appEnableServiceAccountOverride','value':true}]"

Set default Kf Task timeout

Kf uses Tekton TaskRuns as its mechanism to run Kf Tasks. Tekton may impose a default timeout on TaskRuns that differs depending on the version of Tekton you have installed.

You can override this setting either on the Tekton configmap – which will set the value for both Kf Tasks and Builds or on the Kf operator to apply the value only to Tasks.

The following values are supported:

• null - Inherit the value from Tekton (Default).
• Value <= 0 - Tasks get an infinite timeout.
• Value >= 0 - Tasks get the given timeout.

Consider setting a long, but non-infinite timeout to prevent improperly programmed tasks from consuming resources.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/taskDefaultTimeoutMinutes','value':-1}]"

Enable/Disbale NFS in Kf Tasks

You can enable/disable the ability for Tasks run from Apps to mount NFS volumes.

Mounting NFS volumes requires FUSE which grants Task Pods with NFS additional system privileges on the node and may be a security concern.

kubectl patch \
    kfsystem kfsystem \
    --type='json' \
    -p="[{'op':'add','path':'/spec/kf/config/taskDisableVolumeMounts','value':true}]"

1.3.3 - Testing Kf Components

This page describes the tests that Kf runs and notable gaps you may want to cover when using Kf as your platform.

Components

Unit tests

Kf uses Go’s unit testing functionality to perform extensive validation of the business logic in the CLI, the control plane components, and “golden” files which validate the structure of I/O that Kf expects to be deterministic (e.g. certain CLI output).

Unit tests can be executed using the ./hack/unit-test.sh shell script in the Kf repository. This script skips exercising the end to end tests.

Unit tests are naturally limited because they have to mock interactions with external services e.g. the Kubernetes control plane and container registries.

End to end tests

Kf uses specially marked Go tests for acceptance and end to end testing.

These can be executed using the ./hack/integration-test.sh and ./hack/acceptance-test.sh shell scripts. These scripts will test against the currently targeted Kf cluster.

These tests check for behavior between Kf and other components, but are still limited because they tend to test one specific thing at a time and are built for speed.

Load tests

Kf can run load tests against the platform using the ./ci/cloudbuild/test.yaml Cloud Build template with the _STRESS_TEST_BASELOAD_FACTOR environment variable set.

This will deploy the Diego stresds test workloads to Kf to check how the platform behaves with failing Apps.

Gaps

Kf runs unit tests and end to end tests for each release. You may want to augment with additional qualificatoin tests on your own cluster to check for:

• Compatibility with your service brokers.
• Compatibility with all buildpacks you normally use.
• Compatibility with represenative workloads.
• Compatibility with intricate combinations of features (e.g. service bindings, docker containers, buildpacks, routing).
• Compatibility with non-Kf Kubernetes components.

1.3.4 - Kf dependencies and architecture

Kf requires Kubernetes and several other OSS projects to run. Some of the dependencies are satisfied with Google-managed services—for example, GKE provides Kubernetes.

Dependencies

• GKE
• Anthos Service Mesh
• Tekton Pipelines

Get CRD details

Kf supports the kubectl subcommand explain. It allows you to list the fields in Kf CRDs to understand how to create Kf objects via automation instead of manually via the CLI. This command is designed to be used with ConfigSync to automate creation and management of resources like Spaces across many clusters. You can use this against any of the component kinds below.

In this example, we examine the kind called space in the spaces CRD:

kubectl explain space.spec

The output looks similar to this:

$ kubectl explain space.spec
KIND:     Space
VERSION:  kf.dev/v1alpha1

RESOURCE: spec <Object>

DESCRIPTION:
     SpaceSpec contains the specification for a space.

FIELDS:
   buildConfig  <Object>
     BuildConfig contains config for the build pipelines.

   networkConfig        <Object>
     NetworkConfig contains settings for the space's networking environment.

   runtimeConfig        <Object>
     RuntimeConfig contains settings for the app runtime environment.

Kf components

Kf installs several of its own Kubernetes custom resources and controllers. The custom resources effectively serve as the Kf API and are used by the kf CLI to interact with the system. The controllers use Kf’s CRDs to orchestrate the other components in the system.

You can view the CRDs installed and used by Kf by running the following command:

kubectl api-resources --api-group=kf.dev

The output of that command is as follows:

NAME                      SHORTNAMES   APIGROUP   NAMESPACED   KIND
apps                                   kf.dev     true         App
builds                                 kf.dev     true         Build
clusterservicebrokers                  kf.dev     false        ClusterServiceBroker
routes                                 kf.dev     true         Route
servicebrokers                         kf.dev     true         ServiceBroker
serviceinstancebindings                kf.dev     true         ServiceInstanceBinding
serviceinstances                       kf.dev     true         ServiceInstance
spaces                                 kf.dev     false        Space

Apps

Apps represent a twelve-factor application deployed to Kubernetes. They encompass source code, configuration, and the current state of the application. Apps are responsible for reconciling:

• Kf Builds
• Kf Routes
• Kubernetes Deployments
• Kubernetes Services
• Kubernetes ServiceAccounts
• Kubernetes Secrets

You can list Apps using Kf or kubectl:

kf apps

kubectl get apps -n space-name

Builds

Builds combine the source code and build configuration for Apps. They provision Tekton TaskRuns with the correct steps to actuate a Buildpack V2, Buildpack V3, or Dockerfile build.

You can list Builds using Kf or kubectl:

kf builds

kubectl get builds -n space-name

ClusterServiceBrokers

ClusterServiceBrokers hold the connection information necessary to extend Kf with a service broker. They are responsible for fetching the catalog of services the broker provides and displaying them in the output of kf marketplace.

You can list ClusterServiceBrokers using kubectl:

kubectl get clusterservicebrokers

Routes

Routes are a high level structure that contain HTTP routing rules. They are responsible for reconciling Istio VirtualServices.

You can list Routes using Kf or kubectl:

kf routes

kubectl get routes -n space-name

ServiceBrokers

ServiceBrokers hold the connection information necessary to extend Kf with a service broker. They are responsible for fetching the catalog of services the broker provides and displaying them in the output of kf marketplace.

You can list ServiceBrokers using kubectl:

kubectl get servicebrokers -n space-name

ServiceInstanceBinding

ServiceInstanceBindings hold the parameters to create a binding on a service broker and the credentials the broker returns for the binding. They are responsible for calling the bind API on the broker to bind the service.

You can list ServiceInstanceBindings using Kf or kubectl:

kf bindings

kubectl get serviceinstancebindings -n space-name

ServiceInstance

ServiceInstances hold the parameters to create a service on a service broker. They are responsible for calling the provision API on the broker to create the service.

You can list ServiceInstances using Kf or kubectl:

kf services

kubectl get serviceinstances -n space-name

Spaces

Spaces hold configuration information similar to Cloud Foundry organizations and spaces. They are responsible for:

• Creating the Kubernetes Namespace that other Kf resources are provisioned into.
• Creating Kubernetes NetworkPolicies to enforce network connection policies.
• Holding configuration and policy for Builds, Apps, and Routes.

You can list Spaces using Kf or kubectl:

kf spaces

kubectl get spaces

Kf RBAC / Permissions

The following sections list permissions for Kf and its components to have correct access at the cluster level. These permissions are required and enabled by default in Kf; do not attempt to disable them.

Note that the appdevexperience-operator service account has the same set of permissions as controller. The operator is what deploys all Kf components, including custom resource definitions and controllers.

RBAC for Kf service accounts

The following apiGroup definitions detail which access control permissions components in {{product_name}} have on which API groups and resources for both the controller and appdevexperience-operator service accounts.

- apiGroups:
  - "authentication.k8s.io"
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - "authorization.k8s.io"
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - persistentvolumeclaims
  - persistentvolumes
  - endpoints
  - events
  - configmaps
  - secrets
  verbs: *
- apiGroups:
  - ""
  resources:
  - services
  - services/status
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - "apps"
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs: *
- apiGroups:
  - "apps"
  resources:
  - deployments/finalizers
  verbs:
  - get
  - list
  - create
  - update
  - delete
  - patch
  - watch
- apiGroups:
  - "rbac.authorization.k8s.io"
  resources:
  - clusterroles
  - roles
  - clusterrolebindings
  - rolebindings
  verbs:
  - create
  - delete
  - update
  - patch
  - escalate
  - get
  - list
  - deletecollection
  - bind
- apiGroups:
  - "apiregistration.k8s.io"
  resources:
  - apiservices
  verbs:
  - update
  - patch
  - create
  - delete
  - get
  - list
- apiGroups:
  - "pubsub.cloud.google.com"
  resources:
  - topics 
  - topics/status
  verbs: *
- apiGroups:
  - ""
  resources:
  - namespaces
  - namespaces/finalizers
  - serviceaccounts
  verbs: 
  - get
  - list
  - create
  - update
  - watch
  - delete
  - patch
  - watch
- apiGroups:
  - "autoscaling"
  resources:
  - horizontalpodautoscalers
  verbs: 
  - create
  - delete
  - get
  - list
  - update
  - patch
  - watch
- apiGroups:
  - "coordination.k8s.io"
  resources:
  - leases
  verbs: *
- apiGroups:
  - "batch"
  resources:
  - jobs
  - cronjobs
  verbs: 
  - get
  - list
  - create
  - update
  - patch
  - delete
  - deletecollection
  - watch
- apiGroups:
  - "messaging.cloud.google.com"
  resources:
  - channels
  verbs: 
  - delete
- apiGroups:
  - "pubsub.cloud.google.com"
  resources:
  - pullsubscriptions
  verbs: 
  - delete
  - get
  - list
  - watch
  - create
  - update
  - patch
- apiGroups:
  - "pubsub.cloud.google.com"
  resources:
  - [pullsubscriptions/status
  verbs: 
  - get
  - update
  - patch
- apiGroups:
  - "events.cloud.google.com"
  resources: *
  verbs: *
- apiGroups:
  - "keda.k8s.io"
  resources: *
  verbs: *
- apiGroups:
  - "admissionregistration.k8s.io"
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - get
  - list
  - create
  - update
  - patch
  - delete
  - watch
- apiGroups:
  - "extensions"
  resources:
  - ingresses
  - ingresses/status
  verbs: *
- apiGroups:
  - ""
  resources: 
  - endpoints/restricted
  verbs:
  - create
- apiGroups:
  - "certificates.k8s.io"
  resources: 
  - certificatesigningrequests
  - certificatesigningrequests/approval
  - certificatesigningrequests/status
  verbs: 
  - update
  - create
  - get
  - delete
- apiGroups:
  - "apiextensions.k8s.io"
  resources:
  - customresourcedefinitions
  verbs:   
  - get
  - list
  - create
  - update
  - patch
  - delete
  - watch
- apiGroups:
  - "networking.k8s.io"
  resources: 
  - networkpolicies
  verbs: 
  - get
  - list
  - create
  - update
  - patch
  - delete
  - deletecollection
  - watch
- apiGroups:
  - ""
  resources: 
  - nodes
  verbs: 
  - get
  - list
  - watch
  - update
  - patch
- apiGroups:
  - ""
  resources: 
  - nodes/status
  verbs: 
  - patch

The following table lists how the RBAC permissions are used in Kf, where:

• view includes the verbs: get, list, watch
• modify includes the verbs: create, update, delete, patch

Third-party libraries

Third-party library source code and licenses can be found in the /third_party directory of any Kf container image.

You can also run kf third-party-licenses to view the third-party licenses for the version of the Kf CLI that you downloaded.

1.3.5 - Kf reference architecture diagrams

Kf can benefit from the rich ecosystem of Anthos and GKE, including automation, managed backing services, and development tools.

GKE reference architecture

Kf clusters are managed just like any other GKE cluster, and access resources in the same way.

ConfigSync

Kf can work with ConfigSync to automate Space creation across any number of clusters. Create new namespaces on non-Kf clusters, and Spaces on Kf clusters. You can also manage Spaces by removing the Kf CLI from Space management, if desired. Use kubectl explain to get Kf CRD specs to fully manage your product configuration via GitOps.

1.3.6 - Logging and Monitoring

1.3.6.1 - Create and user monitoring dashboards.

You can use Google Cloud Monitoring dashboards to create custom dashboards and charts. Kf comes with a default template which can be used to create dashboards to monitor the performance of your applications.

Application performance dashboard

Run the following commands to deploy a dashboard in your monitoring workspace in Cloud monitoring dashboards to monitor performance of your apps. This dashboard has application performance metrics like requests/sec, round trip latency, HTTP error codes, and more.

git clone https://github.com/google/kf
cd ./kf/dashboards
./create-dashboard.py my-dashboard my-cluster my-space

System resources and performance dashboard

You can view all the system resources and performance metrics such as list of nodes, pods, containers and much more using a built-in dashboard. Click the link below to access the system dashboard.

More details about this dashboard can be found here.

Create SLO and alerts

You can create SLOs and Alerts on available metrics to monitor performance and availability of both system and applications. For example, you can use the metrics istio.io/service/server/response_latencies to setup an alert on the application roundtrip latency.

Configure dashboard access control

Follow these instructions to provide dashboard access to developers and other members on the team. The role roles/monitoring.dashboardViewer provides read-only access to dashboards.

1.3.6.2 - Logging and monitoring

Kf can use GKE’s Google Cloud integrations to send a log of events to your Cloud Monitoring and Cloud Logging project for observability. For more information, see Overview of GKE operations.

Kf deploys two server side components:

1. Controller
2. Webhook

To view the logs for these components, use the following Cloud Logging query:

resource.type="k8s_container"
resource.labels.project_id=<PROJECT ID>
resource.labels.location=<GCP ZONE>
resource.labels.cluster_name=<CLUSTER NAME>
resource.labels.namespace_name="kf"
labels.k8s-pod/app=<controller OR webhook>

1.3.6.3 - Logging and monitoring overview

By default, Kf includes native integration with Cloud Monitoring and Cloud Logging. When you create a cluster, both Monitoring and Cloud Logging are enabled by default. This integration lets you monitor your running clusters and help analyze your system and application performance using advanced profiling and tracing capabilities.

Application level performance metrics is provided by Istio sidecar injection which is injected alongside applications deployed via Kf. You can also create SLO and Alerts using this default integration to monitor performance and availability of both system and applications.

Ensure the following are setup on your cluster:

• Cloud Monitoring and Cloud Logging are enabled on the Kf cluster by default unless you disabled them explicitly, so no extra step is required.

• Istio sidecar injection is enabled. Sidecar proxy will inject application level performance metrics.

1.3.6.4 - View logs

Kf provides you with several types of logs. This document describes these logs and how to access them.

Application logs

All logs written to standard output stdout and standard error stderr, are uploaded to Cloud Logging and stored under the log name user-container.

Open Cloud Logging and run the following query:

resource.type="k8s_container"
log_name="projects/YOUR_PROJECT_ID/logs/user-container"
resource.labels.project_id=YOUR_PROJECT_ID
resource.labels.location=GCP_COMPUTE_ZONE (e.g. us-central1-a)
resource.labels.cluster_name=YOUR_CLUSTER_NAME
resource.labels.namespace_name=YOUR_KF_SPACE_NAME
resource.labels.pod_name:YOUR_KF_APP_NAME

You should see all your application logs written on standard stdout and standard error stderr.

Access logs for your applications

Kf provides access logs using Istio sidecar injection. Access logs are stored under the log name server-accesslog-stackdriver.

Open Cloud Logging and run the following query:

resource.type="k8s_container"
log_name="projects/YOUR_PROJECT_ID/logs/server-accesslog-stackdriver"
resource.labels.project_id=YOUR_PROJECT_ID
resource.labels.location=GCP_COMPUTE_ZONE (e.g. us-central1-a)
resource.labels.cluster_name=YOUR_CLUSTER_NAME
resource.labels.namespace_name=YOUR_KF_SPACE_NAME
resource.labels.pod_name:YOUR_KF_APP_NAME

You should see access logs for your application. Sample access log:

{
  "insertId": "166tsrsg273q5mf",
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "http://test-app-38n6dgwh9kx7h-c72edc13nkcm.***. ***.nip.io/",
    "requestSize": "738",
    "status": 200,
    "responseSize": "3353",
    "remoteIp": "10.128.0.54:0",
    "serverIp": "10.48.0.18:8080",
    "latency": "0.000723777s",
    "protocol": "http"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "user-container",
      "project_id": ***,
      "namespace_name": ***,
      "pod_name": "test-app-85888b9796-bqg7b",
      "location": "us-central1-a",
      "cluster_name": ***
    }
  },
  "timestamp": "2020-11-19T20:09:21.721815Z",
  "severity": "INFO",
  "labels": {
    "source_canonical_service": "istio-ingressgateway",
    "source_principal": "spiffe://***.svc.id.goog/ns/istio-system/sa/istio-ingressgateway-service-account",
    "request_id": "0e3bac08-ab68-408f-9b14-0aec671845bf",
    "source_app": "istio-ingressgateway",
    "response_flag": "-",
    "route_name": "default",
    "upstream_cluster": "inbound|80|http-user-port|test-app.***.svc.cluster.local",
    "destination_name": "test-app-85888b9796-bqg7b",
    "destination_canonical_revision": "latest",
    "destination_principal": "spiffe://***.svc.id.goog/ns/***/sa/sa-test-app",
    "connection_id": "82261",
    "destination_workload": "test-app",
    "destination_namespace": ***,
    "destination_canonical_service": "test-app",
    "upstream_host": "127.0.0.1:8080",
    "log_sampled": "false",
    "mesh_uid": "proj-228179605852",
    "source_namespace": "istio-system",
    "requested_server_name": "outbound_.80_._.test-app.***.svc.cluster.local",
    "source_canonical_revision": "asm-173-6",
    "x-envoy-original-dst-host": "",
    "destination_service_host": "test-app.***.svc.cluster.local",
    "source_name": "istio-ingressgateway-5469f77856-4n2pw",
    "source_workload": "istio-ingressgateway",
    "x-envoy-original-path": "",
    "service_authentication_policy": "MUTUAL_TLS",
    "protocol": "http"
  },
  "logName": "projects/*/logs/server-accesslog-stackdriver",
  "receiveTimestamp": "2020-11-19T20:09:24.627065813Z"
}

Audit logs

Audit Logs provides a chronological record of calls that have been made to the Kubernetes API Server. Kubernetes audit log entries are useful for investigating suspicious API requests, for collecting statistics, or for creating monitoring alerts for unwanted API calls.

Open Cloud Logging and run the following query:

resource.type="k8s_container"
log_name="projects/YOUR_PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
resource.labels.project_id=YOUR_PROJECT_ID
resource.labels.location=GCP_COMPUTE_ZONE (e.g. us-central1-a)
resource.labels.cluster_name=YOUR_CLUSTER_NAME
protoPayload.request.metadata.name=YOUR_APP_NAME
protoPayload.methodName:"deployments."

You should see a trace of calls being made to the Kubernetes API server.

Configure logging access control

Follow these instructions to provide logs access to developers and other members on the team. The role roles/logging.viewer provides read-only access to logs.

Use Logs Router

You can also use Logs Router to route the logs to supported destinations.

1.3.7 - Networking

1.3.7.1 - Set up a custom domain

All Kf Apps that serve HTTP traffic to users or applications outside of the cluster must be associated with a domain name.

Kf has three locations where domains can be configured. Ordered by precedence, they are:

1. Apps
2. Spaces
3. The config-defaults ConfigMap in the kf Namespace

Edit the config-defaults ConfigMap

The config-defaults ConfigMap holds cluster-wide settings for Kf and can be edited by cluster administrators. The values in the ConfigMap are read by the Spaces controller and modify their configuration. Domain values are reflected in the Space’s status.networkConfig.domains field.

To modify Kf cluster’s domain, edit the kfsystem, the operator will then popluate the change to config-defaults configmap under kf namespace:

kubectl edit kfsystem

Add or update the entry for the spaceClusterDomains key under spec.kf.config like the following:

spaceClusterDomains: my-domain.com

To validate the configuration was updated correctly, check the domain value in a Space:

kf space SPACE_NAME -o "jsonpath={.status.networkConfig.domains[]['domain']}"

The output will look similar to:

Getting Space some-space
some-space.my-domain.com

Each Space prefixes the cluster domains with its own name. This prevents conflicts between Apps.

Assign Space domains

Spaces are the authoritative location for domain configuration. You can assign domains and sub-domains to each Space for developers to use. The field for configuring domains is spec.networkConfig.domains.

Use kf space to view the domains assigned to a Space:

kf space SPACE_NAME

In the output, the Spec field contains specific configuration for the Space and the Status field reflects configuration for the Space with cluster-wide defaults appended to the end:

...
Spec:
  Network Config:
    Domains:
      Domain: my-space.mycompany.com
...
Status:
  Network Config:
    Domains:
      Domain: my-space.mycompany.com
      Domain: my-space.prod.us-east1.kf.mycompany.com

Add or remove domains using the CLI

The kf CLI supports mutations on Space domains. Each command outputs a diff between the old and new configurations.

Add a new domain with kf configure-space append-domain:

kf configure-space append-domain SPACE_NAME myspace.mycompany.com

Add or make an existing domain the default with kf configure-space set-default-domain:

kf configure-space set-default-domain SPACE_NAME myspace.mycompany.com

And finally, remove a domain:

kf configure-space remove-domain SPACE_NAME myspace.mycompany.com

Use Apps to specify domains

Apps can specify domains as part of their configuration. Routes are mapped to Apps during kf push using the following logic:

let current_routes  = The set of routes already on the app
let manifest_routes = The set of routes defined by the manifest
let flag_routes     = The set of routes supplied by the --route flag(s)
let no_route        = Whether the manifest has no-route:true or --no-route is set
let random_route    = Whether the manifest has random-route:true or --random-route is set

let new_routes = Union(current_routes, manifest_routes, flag_routes)

if new_routes.IsEmpty() then
  if random_route then
    new_routes.Add(CreateRandomRoute())
  else
    new_routes.Add(CreateDefaultRoute())
  end
end

if no_route then
  new_routes.RemoveAll()
end

return new_routes

If an App doesn’t specify a Route, or requests a random Route, the first domain on the Space is used. If the first domain on a Space changes, all Apps in the Space using the default domain are updated to reflect it.

Customize domain templates

Kf supports variable substitution in domains. Substitution allows a single cluster-wide domain to be customized per-Space and to react to changes to the ingress IP. Substitution is performed on variables with the syntax $(VARIABLE_NAME) that occur in a domain.

Examples

The following examples demonstrate how domain variables can be used to support a variety of different organizational structures and cluster patterns.

• Using a wildcard DNS service like nip.io:

  $(SPACE_NAME).$(CLUSTER_INGRESS_IP).nip.io
  

• Domain for an organization with centrally managed DNS:

  $(SPACE_NAME).cluster-name.example.com
  

• Domain for teams who manage their own DNS:

  cluster-name.$(SPACE_NAME).example.com
  

• Domain for a cluster with warm failover and external circuit breaker:

  $(SPACE_NAME)-failover.cluster-name.example.com
  

Differences between Kf and CF

• Kf Spaces prefix the cluster-wide domain with the Space name.
• Kf does not check for domain conflicts on user-specified routes.

1.3.7.2 - Set up HTTPS ingress

You can secure the ingress gateway with HTTPS by using simple TLS, and enable HTTPS connections to specific webpages. In addition, you can redirect HTTP connections to HTTPS.

HTTPS creates a secure channel over an insecure network, protecting against man-in-the-middle attacks and encrypting traffic between the client and server. To prepare a web server to accept HTTPS connections, an administrator must create a public key certificate for the server. This certificate must be signed by a trusted certificate authority for a web browser to accept it without warning.

Edit the gateway named external-gateway in the kf namespace using the built-in Kubernetes editor:

kubectl edit gateway -n kf external-gateway

1. Assuming you have a certificate and key for your service, create a Kubernetes secret for the ingress gateway. Make sure the secret name does not begin with istio or prometheus. For this example, the secret is named myapp-https-credential.
2. Under servers:
3. Add a section for port 443.
4. Under tls:, set the credentialName to the name of the secret you just created.
5. Under hosts:, add the host name of the service you want to secure with HTTPS. This can be set to an entire domain using a wildcard (e.g. *.example.com) or scoped to just one hostname (e.g. myapp.example.com).
6. There should already be a section under servers: for port 80 HTTP. Keep this section in the Gateway definition if you would like all traffic to come in as HTTP.
7. To redirect HTTP to HTTPS, add the value httpsRedirect: true under tls in the HTTP server section. See the Istio Gateway documentation for reference. Note that adding this in the section where hosts is set to * means that all traffic is redirected to HTTPS. If you only want to redirect HTTP to HTTPS for a single app/domain, add a separate HTTP section specifying the redirect.

Shown below is an example of a Gateway spec that sets up HTTPS for myapp.example.com and redirects HTTP to HTTPS for that host:

spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - myapp.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: myapp-https-credential
      mode: SIMPLE
  - hosts:
    - myapp.example.com
    port:
      name: http-my-app
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

1.3.7.3 - Set up network policies

Kf integrates tightly with Kubernetes and Istio to provide robust network policy enforcement.

By default, Kf workloads are run in the Kubernetes cluster and resolve addresses using Kubernetes DNS. This DNS resolver will first attempt to resolve addresses within the cluster, and only if none are found will attempt external resolution.

Each Kf App gets run with an Envoy sidecar injected by Istio or the Anthos Service Mesh (ASM). This sidecar proxies all network traffic in and out of the Kubernetes Pod.

Each Kubernetes Pod is executed on a Node, a physical or virtual machine responsible for managing the container images that make up a Pod. Nodes exist on a physical or virtual network.

Together, these form a hierarchy of systems you can apply network policies. These are listed below from least to most granular.

Network level policies

Workload protection starts with the network your GKE cluster is installed on.

If you’re running Kf on a GKE cluster on GCP, Kf recommends:

• Placing your GKE cluster on a Virtual Private Cloud (VPC) network.
  • With Private Google Access enabled.
• Using Cloud NAT to control egress.

Node level policies

You can set up policies for containers running on the Node using Kubernetes NetworkPolicies. These are the closest mapping to Cloud Foundry network policies that exist in Kubernetes.

NetworkPolicies are backed by a Kubernetes add-on. If you set up your own GKE cluster, you will need to enable NetworkPolicy enforcement.

Kf labels Apps with kf.dev/networkpolicy=app and builds with kf.dev/networkpolicy=build. This allows you to target NetworkPolicies directly at Pods running Apps or Builds.

Each Kf Space creates two NetworkPolicies to start with, one targeting Apps and one targeting Builds. You can change the configuration on the Space’s spec.networkConfig.(app|build)NetworkPolicy.(in|e)gress fields. These fields can be set to one of the following values:

By default Kf uses a permissive network policy. This allows the following functionality that Kf uses:

• North/South routing to the cluster ingress gateway
• Egress to the Internet e.g. to fetch Buildpacks
• East/West routing between Apps
• Access to the Kubernetes DNS server
• Access to container registries
• Direct access to the VPC network
• Access to Google services like Cloud Logging
• Access to the Workload Identity server for automatic rotating credentials

Service mesh policies

If you need fine-grained networking control, authentication, authorization, and observability you can apply policies using Anthos Service Mesh.

A service mesh is an infrastructure layer that enables managed, observable and secure communication across your services, letting you create robust enterprise applications made up of many microservices on your chosen infrastructure.

You can see the list of supported features here.

1.3.8 - Security

1.3.8.1 - Security Overview

Kf aims to provide a similar developer experience to Cloud Foundry, replicating the build, push, and deploy lifecycle. It does this by building a developer UX layer on top of widely-known, broadly used and adopted technologies like Kubernetes, Istio, and container registries rather than by implementing all the pieces from the ground up.

When making security decisions, Kf aims to provide complete solutions that are native to their respective components and can be augmented with other mechanisms. Breaking that down:

• Complete solutions means that Kf tries not to provide partial solutions that can lead to a false sense of security.
• Native means that the solutions should be a part of the component rather than a Kf construct to prevent breaking changes.
• Can be augmented means the approach Kf takes should work seamlessly with other Kubernetes and Google Cloud tooling for defense in depth.

Important considerations

In addition to the Current limitations described below, it is important that you read through and understand the items outlined in this section.

Workload Identity

By default, Kf uses Workload Identity to provide secure delivery and rotation of the Service Account credentials used by Kf to interact with your Google Cloud Project. Workload Identity achieves this by mapping a Kubernetes Service Account (KSA) to a Google Service Account (GSA). The Kf controller runs in the kf namespace and uses a KSA named controller mapped to your GSA to do the following things:

1. Write metrics to Stackdriver
2. When a new Kf space is created (kf create-space), the Kf controller creates a new KSA named kf-builder in the new space and maps it to the same GSA.
3. The kf-builder KSA is used by Tekton to push and pull container images to Google Container Registry (gcr.io)

This diagram illustrates those interactions:

NFS

In order to mimic Cloud Foundry’s UID/GID mapping, containers in Kf that mount NFS volumes need the ability to run as root and the ability to access the FUSE device of the kernel running the Node.

Current limitations

• A developer pushing an app with Kf can also create Pods (with kubectl) that can use the kf-builder KSA with the permissions of its associated GSA.

• Kf uses the same Pod to fetch, build, and store images. Assume that any credentials that you provide can be known by the authors and publishers of the buildpacks you use.

• Kf doesn’t support quotas to protect against noisy neighbors. Use Kubernetes resource quotas.

Other resources

Google Cloud

General

• GKE security overview
• GKE cluster multi-tenancy
• Workload Identity
• GKE and Cloud IAM policies

Recommended protections

• Protecting cluster metadata
• Role-based access control

Advanced protections

• GKE Sandbox
• Network policies

1.3.8.2 - Role-based access control

Kf provides a set of Kubernetes roles that allow multiple teams to share a Kf cluster. This page describes the roles and best practices to follow when using them.

When to use Kf roles

Kf roles allow multiple teams to share a Kubernetes cluster with Kf installed. The roles provide access to individual Kf Spaces.

Use Kf roles to share access to a cluster if the following are true:

• The cluster is used by trusted teams.
• Workloads on the cluster share the same assumptions about the level of security provided by the environment.
• The cluster exists in a Google Cloud project that is tightly controlled.

Kf roles will not:

• Protect your cluster from untrusted developers or workloads. See the GKE shared responsibility model for more information.
• Provide isolation for your workloads. See the guide to harden your cluster for more information.
• Prevent additional Kubernetes roles from being defined that interact with Kf.
• Prevent access from administrators who have access to the Google Cloud project or cluster.

Kf roles

The following sections describe the Kubernetes RBAC Roles provided by Kf and how they interact with GKE IAM.

Predefined roles

Kf provides several predefined Kubernetes roles to help you provide access to different subjects that perform different roles. Each predefined role can be bound to a subject within a Kubernetes Namespace managed by a Kf Space.

When a subject is bound to a role within a Kubernetes Namespace, their access is limited to objects that exist in the Namespace that match the grants listed in the role. In Kf, some resources are defined at the cluster scope. Kf watches for changes to subjects in the Namespace and grants additional, limited, roles at the cluster scope.