Networking

• 1: Set up a custom domain
• 2: Set up HTTPS ingress
• 3: Set up network policies

1 - Set up a custom domain

All Kf Apps that serve HTTP traffic to users or applications outside of the cluster must be associated with a domain name.

Kf has three locations where domains can be configured. Ordered by precedence, they are:

1. Apps
2. Spaces
3. The config-defaults ConfigMap in the kf Namespace

Edit the config-defaults ConfigMap

The config-defaults ConfigMap holds cluster-wide settings for Kf and can be edited by cluster administrators. The values in the ConfigMap are read by the Spaces controller and modify their configuration. Domain values are reflected in the Space’s status.networkConfig.domains field.

To modify Kf cluster’s domain, edit the kfsystem, the operator will then popluate the change to config-defaults configmap under kf namespace:

kubectl edit kfsystem

Add or update the entry for the spaceClusterDomains key under spec.kf.config like the following:

spaceClusterDomains: my-domain.com

To validate the configuration was updated correctly, check the domain value in a Space:

kf space SPACE_NAME -o "jsonpath={.status.networkConfig.domains[]['domain']}"

The output will look similar to:

Getting Space some-space
some-space.my-domain.com

Each Space prefixes the cluster domains with its own name. This prevents conflicts between Apps.

Assign Space domains

Spaces are the authoritative location for domain configuration. You can assign domains and sub-domains to each Space for developers to use. The field for configuring domains is spec.networkConfig.domains.

Use kf space to view the domains assigned to a Space:

kf space SPACE_NAME

In the output, the Spec field contains specific configuration for the Space and the Status field reflects configuration for the Space with cluster-wide defaults appended to the end:

...
Spec:
  Network Config:
    Domains:
      Domain: my-space.mycompany.com
...
Status:
  Network Config:
    Domains:
      Domain: my-space.mycompany.com
      Domain: my-space.prod.us-east1.kf.mycompany.com

Add or remove domains using the CLI

The kf CLI supports mutations on Space domains. Each command outputs a diff between the old and new configurations.

Add a new domain with kf configure-space append-domain:

kf configure-space append-domain SPACE_NAME myspace.mycompany.com

Add or make an existing domain the default with kf configure-space set-default-domain:

kf configure-space set-default-domain SPACE_NAME myspace.mycompany.com

And finally, remove a domain:

kf configure-space remove-domain SPACE_NAME myspace.mycompany.com

Use Apps to specify domains

Apps can specify domains as part of their configuration. Routes are mapped to Apps during kf push using the following logic:

let current_routes  = The set of routes already on the app
let manifest_routes = The set of routes defined by the manifest
let flag_routes     = The set of routes supplied by the --route flag(s)
let no_route        = Whether the manifest has no-route:true or --no-route is set
let random_route    = Whether the manifest has random-route:true or --random-route is set

let new_routes = Union(current_routes, manifest_routes, flag_routes)

if new_routes.IsEmpty() then
  if random_route then
    new_routes.Add(CreateRandomRoute())
  else
    new_routes.Add(CreateDefaultRoute())
  end
end

if no_route then
  new_routes.RemoveAll()
end

return new_routes

If an App doesn’t specify a Route, or requests a random Route, the first domain on the Space is used. If the first domain on a Space changes, all Apps in the Space using the default domain are updated to reflect it.

Customize domain templates

Kf supports variable substitution in domains. Substitution allows a single cluster-wide domain to be customized per-Space and to react to changes to the ingress IP. Substitution is performed on variables with the syntax $(VARIABLE_NAME) that occur in a domain.

Examples

The following examples demonstrate how domain variables can be used to support a variety of different organizational structures and cluster patterns.

• Using a wildcard DNS service like nip.io:

  $(SPACE_NAME).$(CLUSTER_INGRESS_IP).nip.io
  

• Domain for an organization with centrally managed DNS:

  $(SPACE_NAME).cluster-name.example.com
  

• Domain for teams who manage their own DNS:

  cluster-name.$(SPACE_NAME).example.com
  

• Domain for a cluster with warm failover and external circuit breaker:

  $(SPACE_NAME)-failover.cluster-name.example.com
  

Differences between Kf and CF

• Kf Spaces prefix the cluster-wide domain with the Space name.
• Kf does not check for domain conflicts on user-specified routes.

2 - Set up HTTPS ingress

You can secure the ingress gateway with HTTPS by using simple TLS, and enable HTTPS connections to specific webpages. In addition, you can redirect HTTP connections to HTTPS.

HTTPS creates a secure channel over an insecure network, protecting against man-in-the-middle attacks and encrypting traffic between the client and server. To prepare a web server to accept HTTPS connections, an administrator must create a public key certificate for the server. This certificate must be signed by a trusted certificate authority for a web browser to accept it without warning.

Edit the gateway named external-gateway in the kf namespace using the built-in Kubernetes editor:

kubectl edit gateway -n kf external-gateway

1. Assuming you have a certificate and key for your service, create a Kubernetes secret for the ingress gateway. Make sure the secret name does not begin with istio or prometheus. For this example, the secret is named myapp-https-credential.
2. Under servers:
3. Add a section for port 443.
4. Under tls:, set the credentialName to the name of the secret you just created.
5. Under hosts:, add the host name of the service you want to secure with HTTPS. This can be set to an entire domain using a wildcard (e.g. *.example.com) or scoped to just one hostname (e.g. myapp.example.com).
6. There should already be a section under servers: for port 80 HTTP. Keep this section in the Gateway definition if you would like all traffic to come in as HTTP.
7. To redirect HTTP to HTTPS, add the value httpsRedirect: true under tls in the HTTP server section. See the Istio Gateway documentation for reference. Note that adding this in the section where hosts is set to * means that all traffic is redirected to HTTPS. If you only want to redirect HTTP to HTTPS for a single app/domain, add a separate HTTP section specifying the redirect.

Shown below is an example of a Gateway spec that sets up HTTPS for myapp.example.com and redirects HTTP to HTTPS for that host:

spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - myapp.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: myapp-https-credential
      mode: SIMPLE
  - hosts:
    - myapp.example.com
    port:
      name: http-my-app
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

3 - Set up network policies

Kf integrates tightly with Kubernetes and Istio to provide robust network policy enforcement.

By default, Kf workloads are run in the Kubernetes cluster and resolve addresses using Kubernetes DNS. This DNS resolver will first attempt to resolve addresses within the cluster, and only if none are found will attempt external resolution.

Each Kf App gets run with an Envoy sidecar injected by Istio or the Anthos Service Mesh (ASM). This sidecar proxies all network traffic in and out of the Kubernetes Pod.

Each Kubernetes Pod is executed on a Node, a physical or virtual machine responsible for managing the container images that make up a Pod. Nodes exist on a physical or virtual network.

Together, these form a hierarchy of systems you can apply network policies. These are listed below from least to most granular.

Network level policies

Workload protection starts with the network your GKE cluster is installed on.

If you’re running Kf on a GKE cluster on GCP, Kf recommends:

• Placing your GKE cluster on a Virtual Private Cloud (VPC) network.
  • With Private Google Access enabled.
• Using Cloud NAT to control egress.

Node level policies

You can set up policies for containers running on the Node using Kubernetes NetworkPolicies. These are the closest mapping to Cloud Foundry network policies that exist in Kubernetes.

NetworkPolicies are backed by a Kubernetes add-on. If you set up your own GKE cluster, you will need to enable NetworkPolicy enforcement.

Kf labels Apps with kf.dev/networkpolicy=app and builds with kf.dev/networkpolicy=build. This allows you to target NetworkPolicies directly at Pods running Apps or Builds.

Each Kf Space creates two NetworkPolicies to start with, one targeting Apps and one targeting Builds. You can change the configuration on the Space’s spec.networkConfig.(app|build)NetworkPolicy.(in|e)gress fields. These fields can be set to one of the following values:

By default Kf uses a permissive network policy. This allows the following functionality that Kf uses:

• North/South routing to the cluster ingress gateway
• Egress to the Internet e.g. to fetch Buildpacks
• East/West routing between Apps
• Access to the Kubernetes DNS server
• Access to container registries
• Direct access to the VPC network
• Access to Google services like Cloud Logging
• Access to the Workload Identity server for automatic rotating credentials

Service mesh policies

If you need fine-grained networking control, authentication, authorization, and observability you can apply policies using Anthos Service Mesh.

A service mesh is an infrastructure layer that enables managed, observable and secure communication across your services, letting you create robust enterprise applications made up of many microservices on your chosen infrastructure.

You can see the list of supported features here.