Skip to content

googleDataflowFlexTemplateJob

Creates a Flex Template job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.

Example Usage

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
new google.dataflowFlexTemplateJob.DataflowFlexTemplateJob(
  this,
  "big_data_job",
  {
    container_spec_gcs_path: "gs://my-bucket/templates/template.json",
    name: "dataflow-flextemplates-job",
    parameters: [
      {
        inputSubscription: "messages",
      },
    ],
    provider: "${google-beta}",
  }
);

Note on "destroy" / "apply"

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other Terraform / Google resources.

The Dataflow resource is considered 'existing' while it is in a nonterminal state. If it reaches a terminal state (e.g. 'FAILED', 'COMPLETE', 'CANCELLED'), it will be recreated on the next 'apply'. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is 'destroyed' may be "cancelled" or "drained". If "cancelled", the job terminates - any data written remains where it is, but no new data will be processed. If "drained", no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is "cancelled", but if a user sets onDelete to "drain" in the configuration, you may experience a long wait for your terraformDestroy to complete.

You can potentially short-circuit the wait by setting skipWaitOnJobTermination to true, but beware that unless you take active steps to ensure that the job name parameter changes between instances, the name will conflict and the launch of the new job will fail. One way to do this is with a random_id resource, for example:

import * as cdktf from "cdktf";
/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
import * as random from "./.gen/providers/random";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google, random.
For a more precise conversion please use the --provider flag in convert.*/
/*Terraform Variables are not always the best fit for getting inputs in the context of Terraform CDK.
You can read more about this at https://cdk.tf/variables*/
const bigDataJobSubscriptionId = new cdktf.TerraformVariable(
  this,
  "big_data_job_subscription_id",
  {
    default: "projects/myproject/subscriptions/messages",
  }
);
const randomIdBigDataJobNameSuffix = new random.id.Id(
  this,
  "big_data_job_name_suffix",
  {
    byte_length: 4,
    keepers: [
      {
        region: "${var.region}",
        subscription_id: bigDataJobSubscriptionId.value,
      },
    ],
  }
);
new google.dataflowFlexTemplateJob.DataflowFlexTemplateJob(
  this,
  "big_data_job",
  {
    container_spec_gcs_path: "gs://my-bucket/templates/template.json",
    name: `dataflow-flextemplates-job-\${${randomIdBigDataJobNameSuffix.dec}}`,
    parameters: [
      {
        inputSubscription: bigDataJobSubscriptionId.value,
      },
    ],
    provider: "${google-beta}",
    region: "${var.region}",
    skip_wait_on_job_termination: true,
  }
);

Argument Reference

The following arguments are supported:

  • name - (Required) A unique name for the resource, required by Dataflow.

  • containerSpecGcsPath - (Required) The GCS path to the Dataflow job Flex Template.

  • parameters - (Optional) Key/Value pairs to be passed to the Dataflow job (as used in the template). Additional pipeline options such as serviceAccount, workerMachineType, etc can be specified here.

  • labels - (Optional) User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page. NOTE: Google-provided Dataflow templates often provide default labels that begin with googDataflowProvided. Unless explicitly set in config, these labels will be ignored to prevent diffs on re-apply.

  • onDelete - (Optional) One of "drain" or "cancel". Specifies behavior of deletion during terraformDestroy. See above note.

  • skipWaitOnJobTermination - (Optional) If set to true, terraform will treat draining and cancelling as terminal states when deleting the resource, and will remove the resource from terraform state and move on. See above note.

  • project - (Optional) The project in which the resource belongs. If it is not provided, the provider project is used.

  • region - (Optional) The region in which the created job should run.

Attributes Reference

In addition to the arguments listed above, the following computed attributes are exported:

  • jobId - The unique ID of this job.

  • state - The current state of the resource, selected from the JobState enum

Import

This resource does not support import.