Skip to content

googleComposerEnvironment

An environment for running orchestration tasks.

Environments run Apache Airflow software on Google infrastructure.

To get more information about Environments, see:

\~> Warning: We STRONGLY recommend you read the GCP guides as the Environment resource requires a long deployment process and involves several layers of GCP infrastructure, including a Kubernetes Engine cluster, Cloud Storage, and Compute networking resources. Due to limitations of the API, Terraform will not be able to automatically find or manage many of these underlying resources. In particular:

  • It can take up to one hour to create or update an environment resource. In addition, GCP may only detect some errors in configuration when they are used (e.g. ~40-50 minutes into the creation process), and is prone to limited error reporting. If you encounter confusing or uninformative errors, please verify your configuration is valid against GCP Cloud Composer before filing bugs against the Terraform provider. * Environments create Google Cloud Storage buckets that do not get cleaned up automatically on environment deletion. More about Composer's use of Cloud Storage. * Please review the known issues for Composer if you are having problems.

Example Usage

Basic Usage (Cloud Composer 1)

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
new google.composerEnvironment.ComposerEnvironment(this, "test", {
  name: "example-composer-env",
  region: "us-central1",
});

Basic Usage (Cloud Composer 2)

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
new google.composerEnvironment.ComposerEnvironment(this, "test", {
  config: [
    {
      software_config: [
        {
          image_version: "composer-2-airflow-2",
        },
      ],
    },
  ],
  name: "example-composer-env",
  region: "us-central1",
});

With GKE and Compute Resource Dependencies

NOTE To use custom service accounts, you need to give at least role/composerWorker to the service account being used by the GKE Nodes on the Composer project. For more information, see the Access Control page in the Cloud Composer documentation. You may need to assign additional roles depending on what the Airflow DAGs will be running.

NOTE We STRONGLY recommend you read the Cloud Composer guides as the Environment resource requires a long deployment process and involves several layers of Google Cloud infrastructure, including a Kubernetes Engine cluster, Cloud Storage, and Compute networking resources. Composer manages most of these resources fully and as a result, Terraform may not be able to automatically find or manage the underlying resources. In particular:

  • It can take up to 50 minutes to create or update an environment resource and some errors may be detected later in the process. Also, some error messages may not be clear at first sight because they involve issues with the underlying resources. If you encounter such errors, please review Composer logs and verify if your configuration is valid against Cloud Composer before filing bugs against the Terraform provider.
  • Environments create Google Cloud Storage buckets that contain your DAGs and other work files. These buckets do not get deleted automatically on environment deletion. This is by design; it ensures that DAGs source code and other valuable data don’t get lost when an environment is deleted. More about Composer's use of Cloud Storage.
  • Please review the known issues for Cloud Composer if you are having problems.

GKE and Compute Resource Dependencies (Cloud Composer 1)

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
const googleComputeNetworkTest = new google.computeNetwork.ComputeNetwork(
  this,
  "test",
  {
    auto_create_subnetworks: false,
    name: "composer-test-network",
  }
);
const googleComputeSubnetworkTest =
  new google.computeSubnetwork.ComputeSubnetwork(this, "test_1", {
    ip_cidr_range: "10.2.0.0/16",
    name: "composer-test-subnetwork",
    network: googleComputeNetworkTest.id,
    region: "us-central1",
  });
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleComputeSubnetworkTest.overrideLogicalId("test");
const googleServiceAccountTest = new google.serviceAccount.ServiceAccount(
  this,
  "test_2",
  {
    account_id: "composer-env-account",
    display_name: "Test Service Account for Composer Environment",
  }
);
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleServiceAccountTest.overrideLogicalId("test");
const googleComposerEnvironmentTest =
  new google.composerEnvironment.ComposerEnvironment(this, "test_3", {
    config: [
      {
        database_config: [
          {
            machine_type: "db-n1-standard-2",
          },
        ],
        node_config: [
          {
            machine_type: "n1-standard-1",
            network: googleComputeNetworkTest.id,
            service_account: googleServiceAccountTest.name,
            subnetwork: googleComputeSubnetworkTest.id,
            zone: "us-central1-a",
          },
        ],
        node_count: 4,
        web_server_config: [
          {
            machine_type: "composer-n1-webserver-2",
          },
        ],
      },
    ],
    name: "example-composer-env",
    region: "us-central1",
  });
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleComposerEnvironmentTest.overrideLogicalId("test");
new google.projectIamMember.ProjectIamMember(this, "composer-worker", {
  member: `serviceAccount:\${${googleServiceAccountTest.email}}`,
  role: "roles/composer.worker",
});

GKE and Compute Resource Dependencies (Cloud Composer 2)

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
new google.provider.GoogleProvider(this, "google", {
  project: "bigdata-writers",
});
const googleComputeNetworkTest = new google.computeNetwork.ComputeNetwork(
  this,
  "test",
  {
    auto_create_subnetworks: false,
    name: "composer-test-network3",
  }
);
const googleComputeSubnetworkTest =
  new google.computeSubnetwork.ComputeSubnetwork(this, "test_2", {
    ip_cidr_range: "10.2.0.0/16",
    name: "composer-test-subnetwork",
    network: googleComputeNetworkTest.id,
    region: "us-central1",
  });
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleComputeSubnetworkTest.overrideLogicalId("test");
const googleServiceAccountTest = new google.serviceAccount.ServiceAccount(
  this,
  "test_3",
  {
    account_id: "composer-env-account",
    display_name: "Test Service Account for Composer Environment",
  }
);
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleServiceAccountTest.overrideLogicalId("test");
const googleComposerEnvironmentTest =
  new google.composerEnvironment.ComposerEnvironment(this, "test_4", {
    config: [
      {
        environment_size: "ENVIRONMENT_SIZE_SMALL",
        node_config: [
          {
            network: googleComputeNetworkTest.id,
            service_account: googleServiceAccountTest.name,
            subnetwork: googleComputeSubnetworkTest.id,
          },
        ],
        software_config: [
          {
            image_version: "composer-2-airflow-2",
          },
        ],
        workloads_config: [
          {
            scheduler: [
              {
                count: 1,
                cpu: 0.5,
                memory_gb: 1.875,
                storage_gb: 1,
              },
            ],
            web_server: [
              {
                cpu: 0.5,
                memory_gb: 1.875,
                storage_gb: 1,
              },
            ],
            worker: [
              {
                cpu: 0.5,
                max_count: 3,
                memory_gb: 1.875,
                min_count: 1,
                storage_gb: 1,
              },
            ],
          },
        ],
      },
    ],
    name: "example-composer-env-tf-c2",
    region: "us-central1",
  });
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
googleComposerEnvironmentTest.overrideLogicalId("test");
new google.projectIamMember.ProjectIamMember(this, "composer-worker", {
  member: `serviceAccount:\${${googleServiceAccountTest.email}}`,
  project: "your-project-id",
  role: "roles/composer.worker",
});

With Software (Airflow) Config

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as google from "./.gen/providers/google";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: google.
For a more precise conversion please use the --provider flag in convert.*/
new google.composerEnvironment.ComposerEnvironment(this, "test", {
  config: [
    {
      software_config: [
        {
          airflow_config_overrides: [
            {
              "core-dags_are_paused_at_creation": "True",
            },
          ],
          env_variables: [
            {
              FOO: "bar",
            },
          ],
          pypi_packages: [
            {
              numpy: "",
              scipy: "==1.1.0",
            },
          ],
          scheduler_count: 2,
        },
      ],
    },
  ],
  name: "mycomposer",
  region: "us-central1",
});

Argument Reference - Cloud Composer 1

The following arguments are supported:

  • name - (Required) Name of the environment

  • config - (Optional) Configuration parameters for this environment Structure is documented below.

  • labels - (Optional) User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions: Label keys must be between 1 and 63 characters long and must conform to the following regular expression: [aZ]([AZ09]*[aZ09])?. Label values must be between 0 and 63 characters long and must conform to the regular expression ([aZ]([AZ09]*[aZ09])?)?. No more than 64 labels can be associated with a given environment. Both keys and values must be <= 128 bytes in size.

  • region - (Optional) The location or Compute Engine region for the environment.

  • project - (Optional) The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

The config block supports:

  • nodeCount - (Optional, Cloud Composer 1 only) The number of nodes in the Kubernetes Engine cluster of the environment.

  • nodeConfig - (Optional) The configuration used for the Kubernetes Engine cluster. Structure is documented below.

  • recoveryConfig - (Optional, Cloud Composer 2 only) The configuration settings for recovery. Structure is documented below.

  • softwareConfig - (Optional) The configuration settings for software inside the environment. Structure is documented below.

  • privateEnvironmentConfig - (Optional) The configuration used for the Private IP Cloud Composer environment. Structure is documented below.

  • webServerNetworkAccessControl - (Optional, Cloud Composer 1 only) The network-level access control policy for the Airflow web server. If unspecified, no network-level access restrictions are applied.

  • databaseConfig - (Optional, Cloud Composer 1 only) The configuration settings for Cloud SQL instance used internally by Apache Airflow software.

  • webServerConfig - (Optional, Cloud Composer 1 only) The configuration settings for the Airflow web server App Engine instance.

  • encryptionConfig - (Optional) The encryption options for the Cloud Composer environment and its dependencies.

  • maintenanceWindow - (Optional, Beta) The configuration settings for Cloud Composer maintenance windows.

  • masterAuthorizedNetworksConfig - (Optional) Configuration options for the master authorized networks feature. Enabled master authorized networks will disallow all external traffic to access Kubernetes master through HTTPS except traffic from the given CIDR blocks, Google Compute Engine Public IPs and Google Prod IPs. Structure is documented below.

The nodeConfig block supports:

  • zone - (Optional, Cloud Composer 1 only) The Compute Engine zone in which to deploy the VMs running the Apache Airflow software, specified as the zone name or relative resource name (e.g. "projects/{project}/zones/{zone}"). Must belong to the enclosing environment's project and region.

  • machineType - (Optional, Cloud Composer 1 only) The Compute Engine machine type used for cluster instances, specified as a name or relative resource name. For example: "projects/{project}/zones/{zone}/machineTypes/{machineType}". Must belong to the enclosing environment's project and region/zone.

  • network - (Optional) The Compute Engine network to be used for machine communications, specified as a self-link, relative resource name (for example "projects/{project}/global/networks/{network}"), by name.

    The network must belong to the environment's project. If unspecified, the "default" network ID in the environment's project is used. If a Custom Subnet Network is provided, subnetwork must also be provided.

  • subnetwork - (Optional) The Compute Engine subnetwork to be used for machine communications, specified as a self-link, relative resource name (for example, "projects/{project}/regions/{region}/subnetworks/{subnetwork}"), or by name. If subnetwork is provided, network must also be provided and the subnetwork must belong to the enclosing environment's project and region.

  • diskSizeGb - (Optional, Cloud Composer 1 only) The disk size in GB used for node VMs. Minimum size is 20GB. If unspecified, defaults to 100GB. Cannot be updated.

  • oauthScopes - (Optional, Cloud Composer 1 only) The set of Google API scopes to be made available on all node VMs. Cannot be updated. If empty, defaults to ["https://wwwGoogleapisCom/auth/cloudPlatform"].

  • serviceAccount - (Optional) The Google Cloud Platform Service Account to be used by the node VMs. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated. If given, note that the service account must have roles/composerWorker for any GCP resources created under the Cloud Composer Environment.

  • tags - (Optional) The list of instance tags applied to all node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC1035. Cannot be updated.

  • ipAllocationPolicy - (Optional) Configuration for controlling how IPs are allocated in the GKE cluster. Structure is documented below. Cannot be updated.

  • maxPodsPerNode - (Optional, Beta, Cloud Composer 1 only) The maximum pods per node in the GKE cluster allocated during environment creation. Lowering this value reduces IP address consumption by the Cloud Composer Kubernetes cluster. This value can only be set if the environment is VPC-Native. The range of possible values is 8-110, and the default is 32. Cannot be updated.

  • enableIpMasqAgent - (Optional) Deploys 'ip-masq-agent' daemon set in the GKE cluster and defines nonMasqueradeCIDRs equals to pod IP range so IP masquerading is used for all destination addresses, except between pods traffic. See the documentation.

The softwareConfig block supports:

  • airflowConfigOverrides - (Optional) Apache Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation".

    Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and cannot contain "=" or ";". Section and property names cannot contain characters: "." Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format. Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

  • pypiPackages - (Optional) Custom Python Package Index (PyPI) packages to be installed in the environment. Keys refer to the lowercase package name (e.g. "numpy"). Values are the lowercase extras and version specifier (e.g. "==1.12.0", "[devel,gcp_api]", "[devel]>=1.8.2, <1.9.2"). To specify a package without pinning it to a version specifier, use the empty string as the value.

  • envVariables - (Optional) Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes. Environment variable names must match the regular expression [aZAZ_][aZAZ09_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression airflow_[aZ09_]+_[aZ09_]+), and they cannot match any of the following reserved names: 

    AIRFLOW_HOME
C_FORCE_ROOT
CONTAINER_NAME
DAGS_FOLDER
GCP_PROJECT
GCS_BUCKET
GKE_CLUSTER_NAME
SQL_DATABASE
SQL_INSTANCE
SQL_PASSWORD
SQL_PROJECT
SQL_REGION
SQL_USER

  • imageVersion - (Optional in Cloud Composer 1, required in Cloud Composer 2)

    The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-([09]+(\.[09]+\.[09]+(Preview\.[09]+)?)?|latest)Airflow-([09]+(\.[09]+(\.[09]+)?)?). The Cloud Composer portion of the image version is a full semantic version, or an alias in the form of major version number or 'latest'. The Apache Airflow portion of the image version is a full semantic version that points to one of the supported Apache Airflow versions, or an alias in the form of only major or major.minor versions specified. For more information about Cloud Composer images, see Cloud Composer version list.

  • pythonVersion - (Optional, Cloud Composer 1 only) The major version of Python used to run the Apache Airflow scheduler, worker, and webserver processes. Can be set to '2' or '3'. If not specified, the default is '3'.

  • schedulerCount - (Optional, Cloud Composer 1 with Airflow 2 only) The number of schedulers for Airflow.

See documentation for setting up private environments. The privateEnvironmentConfig block supports:

  • enablePrivateEndpoint - If true, access to the public endpoint of the GKE cluster is denied. If this field is set to true, the ipAllocationPolicyUseIpAliases field must also be set to true for Cloud Composer 1 environments.

  • masterIpv4CidrBlock - (Optional) The IP range in CIDR notation to use for the hosted master network. This range is used for assigning internal IP addresses to the cluster master or set of masters and to the internal load balancer virtual IP. This range must not overlap with any other ranges in use within the cluster's network. If left blank, the default value of is used. See documentation for default values per region.

  • cloudSqlIpv4CidrBlock - (Optional) The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. Needs to be disjoint from webServerIpv4CidrBlock

  • webServerIpv4CidrBlock - (Optional, Cloud Composer 1 only) The CIDR block from which IP range for web server will be reserved. Needs to be disjoint from masterIpv4CidrBlock and cloudSqlIpv4CidrBlock.

  • enablePrivatelyUsedPublicIps - (Optional) When enabled, IPs from public (non-RFC1918) ranges can be used for ipAllocationPolicyClusterIpv4CidrBlock and ipAllocationPolicyServiceIpv4CidrBlock.

The webServerNetworkAccessControl supports:

  • allowedIpRange - A collection of allowed IP ranges with descriptions. Structure is documented below.

The allowedIpRange supports:

  • value - (Required) IP address or range, defined using CIDR notation, of requests that this rule applies to. Examples: 19216811 or 19216800/16 or 2001:db8::/32 or 2001:0Db8:0000:0042:0000:8A2E:0370:7334. IP range prefixes should be properly truncated. For example, 1234/24 should be truncated to 1230/24. Similarly, for IPv6, 2001:db8::1/32 should be truncated to 2001:db8::/32.

  • description - (Optional) A description of this ip range.

The ipAllocationPolicy block supports:

  • useIpAliases - (Optional, Cloud Composer 1 only) Whether or not to enable Alias IPs in the GKE cluster. If true, a VPC-native cluster is created. Defaults to true if the ipAllocationPolicy block is present in config.

  • clusterSecondaryRangeName - (Optional) The name of the cluster's secondary range used to allocate IP addresses to pods. Specify either clusterSecondaryRangeName or clusterIpv4CidrBlock but not both. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true.

  • servicesSecondaryRangeName - (Optional) The name of the services' secondary range used to allocate IP addresses to the cluster. Specify either servicesSecondaryRangeName or servicesIpv4CidrBlock but not both. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true.

  • clusterIpv4CidrBlock - (Optional) The IP address range used to allocate IP addresses to pods in the cluster. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either clusterSecondaryRangeName or clusterIpv4CidrBlock but not both.

  • servicesIpv4CidrBlock - (Optional) The IP address range used to allocate IP addresses in this cluster. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either servicesSecondaryRangeName or servicesIpv4CidrBlock but not both.

The databaseConfig block supports:

  • machineType - (Required) Cloud SQL machine type used by Airflow database. It has to be one of: db-n1-standard-2, db-n1-standard-4, db-n1-standard-8 or db-n1-standard-16.

The webServerConfig block supports:

  • machineType - (Required) Machine type on which Airflow web server is running. It has to be one of: composer-n1-webserver-2, composer-n1-webserver-4 or composer-n1-webserver-8. Value custom is returned only in response, if Airflow web server parameters were manually changed to a non-standard values.

The encryptionConfig block supports:

  • kmsKeyName - (Required) Customer-managed Encryption Key available through Google's Key Management Service. It must be the fully qualified resource name, i.e. projects/project-id/locations/location/keyRings/keyring/cryptoKeys/key. Cannot be updated.

The maintenanceWindow block supports:

  • startTime - (Required) Start time of the first recurrence of the maintenance window.

  • endTime - (Required) Maintenance window end time. It is used only to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to 'start_time'.

  • recurrence - (Required) Maintenance window recurrence. Format is a subset of RFC-5545 (https://tools.ietf.org/html/rfc5545) 'RRULE'. The only allowed values for 'FREQ' field are 'FREQ=DAILY' and 'FREQ=WEEKLY;BYDAY=...'. Example values: 'FREQ=WEEKLY;BYDAY=TU,WE', 'FREQ=DAILY'.

The masterAuthorizedNetworksConfig block supports:

  • enabled - (Required) Whether or not master authorized networks is enabled.

  • cidrBlocks - cidrBlocksdefine up to 50 external networks that could access Kubernetes master through HTTPS. Structure is documented below.

The cidrBlocks supports:

  • displayName - (Optional) displayName is a field for users to identify CIDR blocks.

  • cidrBlock - (Required) `cidr_block< must be specified in CIDR notation.

Argument Reference - Cloud Composer 2

The following arguments are supported:

  • name - (Required) Name of the environment

  • config - (Optional) Configuration parameters for this environment. Structure is documented below.

  • labels - (Optional) User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions: Label keys must be between 1 and 63 characters long and must conform to the following regular expression: [aZ]([AZ09]*[aZ09])?. Label values must be between 0 and 63 characters long and must conform to the regular expression ([aZ]([AZ09]*[aZ09])?)?. No more than 64 labels can be associated with a given environment. Both keys and values must be <= 128 bytes in size.

  • region - (Optional) The location or Compute Engine region for the environment.

  • project - (Optional) The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

The config block supports:

  • nodeConfig - (Optional) The configuration used for the Kubernetes Engine cluster. Structure is documented below.

  • softwareConfig - (Optional) The configuration settings for software (Airflow) inside the environment. Structure is documented below.

  • privateEnvironmentConfig - (Optional) The configuration used for the Private IP Cloud Composer environment. Structure is documented below.

  • encryptionConfig - (Optional) The encryption options for the Cloud Composer environment and its dependencies.

  • maintenanceWindow - (Optional) The configuration settings for Cloud Composer maintenance windows.

  • workloadsConfig - (Optional, Cloud Composer 2 only) The Kubernetes workloads configuration for GKE cluster associated with the Cloud Composer environment.

  • environmentSize - (Optional, Cloud Composer 2 only) The environment size controls the performance parameters of the managed Cloud Composer infrastructure that includes the Airflow database. Values for environment size are environmentSizeSmall, environmentSizeMedium, and environmentSizeLarge.

  • masterAuthorizedNetworksConfig - (Optional) Configuration options for the master authorized networks feature. Enabled master authorized networks will disallow all external traffic to access Kubernetes master through HTTPS except traffic from the given CIDR blocks, Google Compute Engine Public IPs and Google Prod IPs. Structure is documented below.

The nodeConfig block supports:

  • network - (Optional) The Compute Engine network to be used for machine communications, specified as a self-link, relative resource name (for example "projects/{project}/global/networks/{network}"), by name.

    The network must belong to the environment's project. If unspecified, the "default" network ID in the environment's project is used. If a Custom Subnet Network is provided, subnetwork must also be provided.

  • subnetwork - (Optional) The Compute Engine subnetwork to be used for machine communications, specified as a self-link, relative resource name (for example, "projects/{project}/regions/{region}/subnetworks/{subnetwork}"), or by name. If subnetwork is provided, network must also be provided and the subnetwork must belong to the enclosing environment's project and region.

  • serviceAccount - (Optional) The Google Cloud Platform Service Account to be used by the node VMs. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated. If given, note that the service account must have roles/composerWorker for any GCP resources created under the Cloud Composer Environment.

  • ipAllocationPolicy - (Optional) Configuration for controlling how IPs are allocated in the GKE cluster. Structure is documented below. Cannot be updated.

  • enableIpMasqAgent - (Optional) IP Masq Agent translates Pod IP addresses to node IP addresses, so that destinations and services targeted from Airflow DAGs and tasks only receive packets from node IP addresses instead of Pod IP addresses See the documentation.

The softwareConfig block supports:

  • airflowConfigOverrides - (Optional) Apache Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation".

    Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and cannot contain "=" or ";". Section and property names cannot contain characters: "." Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format. Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

  • pypiPackages - (Optional) Custom Python Package Index (PyPI) packages to be installed in the environment. Keys refer to the lowercase package name (e.g. "numpy"). Values are the lowercase extras and version specifier (e.g. "==1.12.0", "[devel,gcp_api]", "[devel]>=1.8.2, <1.9.2"). To specify a package without pinning it to a version specifier, use the empty string as the value.

  • envVariables - (Optional) Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes. Environment variable names must match the regular expression [aZAZ_][aZAZ09_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression airflow_[aZ09_]+_[aZ09_]+), and they cannot match any of the following reserved names: 

    AIRFLOW_HOME
C_FORCE_ROOT
CONTAINER_NAME
DAGS_FOLDER
GCP_PROJECT
GCS_BUCKET
GKE_CLUSTER_NAME
SQL_DATABASE
SQL_INSTANCE
SQL_PASSWORD
SQL_PROJECT
SQL_REGION
SQL_USER

  • imageVersion - (Required in Cloud Composer 2, optional in Cloud Composer 1)

    In Cloud Composer 2, you must specify an image with Cloud Composer 2. Otherwise, the default image for Cloud Composer 1 is used. For more information about Cloud Composer images, see Cloud Composer version list.

    The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-([09]+(\.[09]+\.[09]+(Preview\.[09]+)?)?|latest)Airflow-([09]+(\.[09]+(\.[09]+)?)?). The Cloud Composer portion of the image version is a full semantic version, or an alias in the form of major version number or 'latest'. The Apache Airflow portion of the image version is a full semantic version that points to one of the supported Apache Airflow versions, or an alias in the form of only major or major.minor versions specified. Important: In-place upgrade is only available using googleBeta provider. It's because updating the imageVersion is still in beta. Using googleBeta provider, you can upgrade in-place between minor or patch versions of Cloud Composer or Apache Airflow. For example, you can upgrade your environment from composer116X to composer117X, or from airflow21X to airflow22X. You cannot upgrade between major Cloud Composer or Apache Airflow versions (from 1XX to 2XX). To do so, create a new environment.

  • cloudDataLineageIntegration - (Optional, Beta, Cloud Composer environments in versions composer-2.1.2-airflow-..* and newer) The configuration for Cloud Data Lineage integration. Structure is documented below.

The cloudDataLineageIntegration block supports:

  • enabled - (Required) Whether or not Cloud Data Lineage integration is enabled.

See documentation for setting up private environments. The privateEnvironmentConfig block supports:

  • enablePrivateEndpoint - If true, access to the public endpoint of the GKE cluster is denied.

  • masterIpv4CidrBlock - (Optional) The IP range in CIDR notation to use for the hosted master network. This range is used for assigning internal IP addresses to the cluster master or set of masters and to the internal load balancer virtual IP. This range must not overlap with any other ranges in use within the cluster's network. If left blank, the default value of is used. See documentation for default values per region.

  • cloudSqlIpv4CidrBlock - (Optional) The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. Needs to be disjoint from webServerIpv4CidrBlock

  • cloudComposerNetworkIpv4CidrBlock" - (Optional, Cloud Composer 2 only) The CIDR block from which IP range for Cloud Composer Network in tenant project will be reserved. Needs to be disjoint from private_cluster_config.master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block.

  • enablePrivatelyUsedPublicIps - (Optional) When enabled, IPs from public (non-RFC1918) ranges can be used for ipAllocationPolicyClusterIpv4CidrBlock and ipAllocationPolicyServiceIpv4CidrBlock.

  • cloudComposerConnectionSubnetwork - (Optional) When specified, the environment will use Private Service Connect instead of VPC peerings to connect to Cloud SQL in the Tenant Project, and the PSC endpoint in the Customer Project will use an IP address from this subnetwork. This field is supported for Cloud Composer environments in versions composer2.*.*Airflow-*.*.* and newer.

The ipAllocationPolicy block supports:

  • clusterSecondaryRangeName - (Optional) The name of the cluster's secondary range used to allocate IP addresses to pods. Specify either clusterSecondaryRangeName or clusterIpv4CidrBlock but not both.

  • servicesSecondaryRangeName - (Optional) The name of the services' secondary range used to allocate IP addresses to the cluster. Specify either servicesSecondaryRangeName or servicesIpv4CidrBlock but not both.

  • clusterIpv4CidrBlock - (Optional) The IP address range used to allocate IP addresses to pods in the cluster. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either clusterSecondaryRangeName or clusterIpv4CidrBlock but not both.

  • servicesIpv4CidrBlock - (Optional) The IP address range used to allocate IP addresses in this cluster. For Cloud Composer 1 environments, this field is applicable only when useIpAliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either servicesSecondaryRangeName or servicesIpv4CidrBlock but not both.

The encryptionConfig block supports:

  • kmsKeyName - (Required) Customer-managed Encryption Key available through Google's Key Management Service. It must be the fully qualified resource name, i.e. projects/project-id/locations/location/keyRings/keyring/cryptoKeys/key. Cannot be updated.

The maintenanceWindow block supports:

  • startTime - (Required) Start time of the first recurrence of the maintenance window.

  • endTime - (Required) Maintenance window end time. It is used only to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to 'start_time'.

  • recurrence - (Required) Maintenance window recurrence. Format is a subset of RFC-5545 (https://tools.ietf.org/html/rfc5545) 'RRULE'. The only allowed values for 'FREQ' field are 'FREQ=DAILY' and 'FREQ=WEEKLY;BYDAY=...'. Example values: 'FREQ=WEEKLY;BYDAY=TU,WE', 'FREQ=DAILY'.

The recoveryConfig block supports:

  • scheduledSnapshotsConfig - (Optional) The recovery configuration settings for the Cloud Composer environment.

The scheduledSnapshotsConfig block supports:

  • enabled - (Optional) When enabled, Cloud Composer periodically saves snapshots of your environment to a Cloud Storage bucket.

  • snapshotLocation - (Optional) The URI of a bucket folder where to save the snapshot.

  • snapshotCreationSchedule - (Optional) Snapshot schedule, in the unix-cron format.

  • timeZone - (Optional) A time zone for the schedule. This value is a time offset and does not take into account daylight saving time changes. Valid values are from UTC-12 to UTC+12. Examples: UTC, UTC-01, UTC+03.

The workloadsConfig block supports:

  • scheduler - (Optional) Configuration for resources used by Airflow schedulers.

  • triggerer - (Optional, Beta) Configuration for resources used by Airflow triggerer.

  • webServer - (Optional) Configuration for resources used by Airflow web server.

  • worker - (Optional) Configuration for resources used by Airflow workers.

The scheduler block supports:

  • cpu - (Optional) The number of CPUs for a single Airflow scheduler.

  • memoryGb - (Optional) The amount of memory (GB) for a single Airflow scheduler.

  • storageGb - (Optional) The amount of storage (GB) for a single Airflow scheduler.

  • count - (Optional) The number of schedulers.

The triggerer block supports:

  • cpu - (Required) The number of CPUs for a single Airflow triggerer.

  • memoryGb - (Required) The amount of memory (GB) for a single Airflow triggerer.

  • count - (Required) The number of Airflow triggerers.

The webServer block supports:

  • cpu - (Optional) The number of CPUs for the Airflow web server.

  • memoryGb - (Optional) The amount of memory (GB) for the Airflow web server.

  • storageGb - (Optional) The amount of storage (GB) for the Airflow web server.

The worker block supports:

  • cpu - (Optional) The number of CPUs for a single Airflow worker.

  • memoryGb - (Optional) The amount of memory (GB) for a single Airflow worker.

  • floatStorageGb (Optional) The amount of storage (GB) for a single Airflow worker.

  • minCount - (Optional) The minimum number of Airflow workers that the environment can run. The number of workers in the environment does not go above this number, even if a lower number of workers can handle the load.

  • maxCount - (Optional) The maximum number of Airflow workers that the environment can run. The number of workers in the environment does not go above this number, even if a higher number of workers is required to handle the load.

Attributes Reference

In addition to the arguments listed above, the following computed attributes are exported:

  • id - an identifier for the resource with format projects/{{project}}/locations/{{region}}/environments/{{name}}

  • config0GkeCluster - The Kubernetes Engine cluster used to run this environment.

  • config0DagGcsPrefix - The Cloud Storage prefix of the DAGs for this environment. Although Cloud Storage objects reside in a flat namespace, a hierarchical file tree can be simulated using '/'-delimited object name prefixes. DAG objects for this environment reside in a simulated directory with this prefix.

  • config0AirflowUri - The URI of the Apache Airflow Web UI hosted within this environment.

Timeouts

This resource provides the following Timeouts configuration options: configuration options:

  • create - Default is 60 minutes.
  • update - Default is 60 minutes.
  • delete - Default is 6 minutes.

Import

Environment can be imported using any of these accepted formats:

$ terraform import google_composer_environment.default projects/{{project}}/locations/{{region}}/environments/{{name}}
$ terraform import google_composer_environment.default {{project}}/{{region}}/{{name}}
$ terraform import google_composer_environment.default {{name}}