Tasks

• 1: Tasks Overview
• 2: Run Tasks
• 3: Schedule Tasks

1 - Tasks Overview

About Tasks

Unlike Apps which run indefinitely and restart if the process terminates, Tasks run a process until it completes and then stop. Tasks are run in their own containers and are based on the configuration and binary of an existing App.

Tasks are not accessible from routes, and should be used for one-off or scheduled recurring work necessary for the health of an application.

Use cases for Tasks

• Migrating a database
• Running a batch job (scheduled/unscheduled)
• Sending an email
• Transforming data (ETL)
• Processing data (upload/backup/download)

How Tasks work

Tasks are executed asynchronously and run independently from the parent App or other Tasks running on the same App. An App created for running Tasks does not have routes created or assigned, and the Run lifecycle is skipped. The Source code upload and Build lifecycles still proceed and result in a container image used for running Tasks after pushing the App (see App lifecycles at Deploying an Application).

The lifecycle of a Task is as follows:

1. You push an App for running tasks with the kf push APP_NAME --task command.
2. You run a Task on the App with the kf run-task APP_NAME command. Task inherits the environment variables, service bindings, resource allocation, start-up command, and security groups bound to the App.
3. Kf creates a Tekton PipelineRun with values from the App and parameters from the run-task command.
4. The Tekton PipelineRun creates a Kubernetes Pod which launches a container based on the configurations on the App and Task.
5. Task execution stops (Task exits or is terminated manually), the underlying Pod is either stopped or terminated. Pods of stopped Tasks are preserved and thus Task logs are accessible via the kf logs APP_NAME --task command.
6. If you terminate a Task before it stops, the Tekton PipelineRun is cancelled (see Cancelling a PipelineRun), the underlying Pod together with the logs are deleted. The logs of termianted Tasks are delivered to the cluster level logging streams if configured (e.g. Stackdriver, Fluentd).
7. If the number of Tasks run on an App is greater than 500, the oldest Tasks are automatically deleted.

Tasks retention policy

Tasks are created as custom resources in the Kubernetes cluster, therefore, it is important not to exhaust the space of the underlying etcd database. By default, Kf only keeps the latest 500 Tasks per each App. Once the number of Tasks reach 500, the oldest Tasks (together with the underlying Pods and logs) will be automatically deleted.

Task logging and execution history

Any data or messages the Task outputs to STDOUT or STDERR is available by using the kf logs APP_NAME --task command. Cluster level logging mechanism (such as Stackdriver, Fluentd) will deliver the Task logs to the configured logging destination.

Scheduling Tasks

As described above, Tasks can be run asynchronously by using the kf run-task APP_NAME command. Alternatively, you can schedule Tasks for execution by first creating a Job using the kf create-job command, and then scheduling it with the kf schedule-job JOB_NAME command. You can schedule that Job to automatically run Tasks on a specified unix-cron schedule.

How Tasks are scheduled

Create and schedule a Job to run the Task. A Job describes the Task to execute and automatically manages Task creation.

Tasks are created on the schedule even if previous executions of the Task are still running. If any executions are missed for any reason, only the most recently missed execution are executed when the system recovers.

Deleting a Job deletes all associated Tasks. If any associated Tasks were still in progress, they are forcefully deleted without running to completion.

Tasks created by a scheduled Job are still subject to the Task retention policy.

Differences from PCF Scheduler

PCF Scheduler allows multiple schedules for a single Job while Kf only supports a single schedule per Job. You can replicate the PCF Scheduler behavior by creating multiple Jobs, one for each schedule.

2 - Run Tasks

You can execute short-lived workflows by running them as Tasks in Kf. Tasks are run under Apps, meaning that each Task must have an associated App. Each Task execution uses the build artifacts from the parent App. Because Tasks are short-lived, the App is not deployed as a long-running application, and no routes should be created for the App or the Task.

Push an App for running Tasks

1. Clone the test-app repo repo:

   git clone https://github.com/cloudfoundry-samples/test-app test-app
   
   cd test-app
   

2. Push the App.

   Push the App with the kf push APP_NAME --task command. The --task flag indicates that the App is meant to be used for running Tasks, and thus no routes are created on the App, and it is not deployed as a long-running application:

   kf push test-app --task
   

3. Confirm that no App instances or routes were created by listing the App:

   kf apps
   

   Notice that the App is not started and has no URLs:

   Listing Apps in Space: test-space
   Name                     Instances  Memory  Disk  CPU   URLs
   test-app                 stopped    1Gi     1Gi   100m  <nil>
   

Run a Task on the App

When you run a Task on the App, you can optionally specify a start command by using the --command flag. If no start command is specified, it uses the start command specified on the App. If the App doesn’t have a start command specified, it looks up the CMD configuration of the container image. A start command must exist in order to run the Task successfully.

kf run-task test-app --command "printenv"

You see something similar to this, confirming that the Task was submitted:

Task test-app-gd8dv is submitted successfully for execution.

The Task name is automatically generated, prefixed with the App name, and suffixed with an arbitrary string. The Task name is a unique identifier for Tasks within the same cluster.

Specify Task resource limits

Resource limits (such as CPU cores/Memory limit/Disk quota) can be specified in the App (during kf push) or during the kf run-task command. The limits specified in the kf run-task command take prededence over the limits specified on the App.

To specify resource limits in an App, you can use the --cpu-cores, --memory-limit, and --disk-quota flags in the kf push command:

kf push test-app --command "printenv" --cpu-cores=0.5 --memory-limit=2G --disk-quota=5G --task

To override these limits in the App, you can use the --cpu-cores, --memory-limit, and --disk-quota flags in the kf run-task command:

kf run-task test-app --command "printenv" --cpu-cores=0.5 --memory-limit=2G --disk-quota=5G

Specify a custom display name for a Task

You can optionally use the --name flag to specify a custom display name for a Task for easier identification/grouping:

$ kf run-task test-app --command "printenv" --name foo
Task test-app-6swct is submitted successfully for execution.

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-6swct    3   foo                1m     21s       True       <nil>

Manage Tasks

View all Tasks of an App with the kf tasks APP_NAME command:

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-gd8dv    1   test-app-gd8dv     1m     21s       True       <nil>

Cancel a Task

Cancel an active Task by using the kf terminate-task command:

• Cancel a Task by Task name:

  $ kf terminate-task test-app-6w6mz
  Task "test-app-6w6mz" is successfully submitted for termination
  

• Or cancel a Task by APP_NAME + Task ID:

  $ kf terminate-task test-app 2
  Task "test-app-6w6mz" is successfully submitted for termination
  

Cancelled Tasks have PipelineRunCancelled status.

$ kf tasks test-app
Listing Tasks in Space: test space
Name              ID  DisplayName        Age    Duration  Succeeded  Reason
test-app-gd8dv    1   test-app-gd8dv     1m     21s       True       <nil>
test-app-6w6mz    2   test-app-6w6mz     38s    11s       False      PipelineRunCancelled

View Task logs

View logs of a Task by using the kf logs APP_NAME --task command:

$ kf logs test-app --task

3 - Schedule Tasks

You can execute short-lived workflows by running them as Tasks. Running Tasks describes how to run Tasks under Apps.

You can also schedule Tasks to run at recurring intervals specified using the unix-cron format. With scheduled Tasks, you first push an App running the Task as you do with an unscheduled Task, and then create a Job to schedule the Task.

You can define a schedule so that your Task runs multiple times a day or on specific days and months.

Push an App for running scheduled Tasks

1. Clone the test-app repo:

git clone https://github.com/cloudfoundry-samples/test-app test-app

cd test-app

1. Push the App.

   Push the App with the kf push APP_NAME --task command. The --task flag indicates that the App is meant to be used for running Tasks, and thus no routes will be created on the App and it will not be deployed as a long-running application.

   kf push test-app --task
   

2. Confirm that no App instances or routes were created by listing the App:

   kf apps
   

   Notice that the App is not started and has no URLs:

   Listing Apps in Space: test-space
   Name                     Instances  Memory  Disk  CPU   URLs
   test-app                 stopped    1Gi     1Gi   100m  <nil>
   

Create a Job

To run a Task on a schedule, you must first create a Job that describes the Task:

kf create-job test-app test-job "printenv"

The Job starts suspended or unscheduled, and does not create Tasks until it is manually executed by kf run-job or scheduled by kf schedule-task.

Run a Job manually

Jobs can be run ad hoc similar to running Tasks by kf run-task. This option can be useful for testing the Job before scheduling or running as needed in addition to the schedule.

kf run-job test-job

This command runs the Task defined by the Job a single time immediately.

Schedule a Job

To schedule the Job for execution, you must provide a unix-cron schedule in the kf schedule-job command:

kf schedule-job test-job "* * * * *"

This command triggers the Job to automatically create Tasks on the specified schedule. In this example a Task runs every minute.

You can update a Job’s schedule by running kf schedule-task with a new schedule. Jobs in Kf can only have a single cron schedule. This differs from the PCF Scheduler, which allows multiple schedules for a single Job. If you require multiple cron schedules, then you can achieve that with multiple Jobs.

Manage Jobs and schedules

View all Jobs, both scheduled and unscheduled, in the current Space by using the kf jobs command:

$ kf jobs
Listing Jobs in Space: test space
Name               Schedule    Suspend  LastSchedule  Age  Ready  Reason
test-job           * * * * *   <nil>    16s           2m   True   <nil>
unscheduled-job    0 0 30 2 *  true     16s           2m   True   <nil>

Additionally, you can view only Jobs that are actively scheduled with the kf job-schedules command.

$ kf job-schedules
Listing job schedules in Space: test space
Name           Schedule   Suspend  LastSchedule  Age  Ready  Reason
test-job       * * * * *  <nil>    16s           2m   True   <nil>

Notice how the unscheduled-job is not listed in the kf job-schedules output.

Cancel a Job’s schedule

You can stop a scheduled Job with the kf delete-job-schedule command:

kf delete-job-schedule test-job

This command suspends the Job and stops it from creating Tasks on the previous schedule. The Job is not deleted and can be scheduled again by kf schedule-job to continue execution.

Delete a Job

The entire Job can be deleted with the kf delete-job command:

kf delete-job test-job

This command deletes the Job and all Tasks that were created by the Job, both scheduled and manual executions. If any Tasks are still running, this command forcefully deletes them.

If you want to ensure that running Tasks are not interrupted, then first delete the Jobs schedule with kf delete-job-schedule, wait for all Tasks to complete, and then delete the job by calling kf delete-job.