azurermHdinsightSparkCluster
Manages a HDInsight Spark Cluster.
Example Usage
/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as azurerm from "./.gen/providers/azurerm";
/*The following providers are missing schema information and might need manual adjustments to synthesize correctly: azurerm.
For a more precise conversion please use the --provider flag in convert.*/
const azurermResourceGroupExample = new azurerm.resourceGroup.ResourceGroup(
this,
"example",
{
location: "West Europe",
name: "example-resources",
}
);
const azurermStorageAccountExample = new azurerm.storageAccount.StorageAccount(
this,
"example_1",
{
account_replication_type: "LRS",
account_tier: "Standard",
location: azurermResourceGroupExample.location,
name: "hdinsightstor",
resource_group_name: azurermResourceGroupExample.name,
}
);
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
azurermStorageAccountExample.overrideLogicalId("example");
const azurermStorageContainerExample =
new azurerm.storageContainer.StorageContainer(this, "example_2", {
container_access_type: "private",
name: "hdinsight",
storage_account_name: azurermStorageAccountExample.name,
});
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
azurermStorageContainerExample.overrideLogicalId("example");
const azurermHdinsightSparkClusterExample =
new azurerm.hdinsightSparkCluster.HdinsightSparkCluster(this, "example_3", {
cluster_version: "3.6",
component_version: [
{
spark: "2.3",
},
],
gateway: [
{
password: "TerrAform123!",
username: "acctestusrgw",
},
],
location: azurermResourceGroupExample.location,
name: "example-hdicluster",
resource_group_name: azurermResourceGroupExample.name,
roles: [
{
head_node: [
{
password: "AccTestvdSC4daf986!",
username: "acctestusrvm",
vm_size: "Standard_A3",
},
],
worker_node: [
{
password: "AccTestvdSC4daf986!",
target_instance_count: 3,
username: "acctestusrvm",
vm_size: "Standard_A3",
},
],
zookeeper_node: [
{
password: "AccTestvdSC4daf986!",
username: "acctestusrvm",
vm_size: "Medium",
},
],
},
],
storage_account: [
{
is_default: true,
storage_account_key: azurermStorageAccountExample.primaryAccessKey,
storage_container_id: azurermStorageContainerExample.id,
},
],
tier: "Standard",
});
/*This allows the Terraform resource name to match the original name. You can remove the call if you don't need them to match.*/
azurermHdinsightSparkClusterExample.overrideLogicalId("example");
Argument Reference
The following arguments are supported:
-
name
- (Required) Specifies the name for this HDInsight Spark Cluster. Changing this forces a new resource to be created. -
resourceGroupName
- (Required) Specifies the name of the Resource Group in which this HDInsight Spark Cluster should exist. Changing this forces a new resource to be created. -
location
- (Required) Specifies the Azure Region which this HDInsight Spark Cluster should exist. Changing this forces a new resource to be created. -
clusterVersion
- (Required) Specifies the Version of HDInsights which should be used for this Cluster. Changing this forces a new resource to be created. -
componentVersion
- (Required) AcomponentVersion
block as defined below. -
encryptionInTransitEnabled
- (Optional) Whether encryption in transit is enabled for this Cluster. Changing this forces a new resource to be created. -
diskEncryption
- (Optional) One or morediskEncryption
block as defined below. -
gateway
- (Required) Agateway
block as defined below. -
roles
- (Required) Aroles
block as defined below. -
network
- (Optional) Anetwork
block as defined below. -
computeIsolation
- (Optional) AcomputeIsolation
block as defined below. -
storageAccount
- (Optional) One or morestorageAccount
block as defined below. -
storageAccountGen2
- (Optional) AstorageAccountGen2
block as defined below. -
tier
- (Required) Specifies the Tier which should be used for this HDInsight Spark Cluster. Possible values arestandard
orpremium
. Changing this forces a new resource to be created. -
tlsMinVersion
- (Optional) The minimal supported TLS version. Possible values are 1.0, 1.1 or 1.2. Changing this forces a new resource to be created.
\~> NOTE: Starting on June 30, 2020, Azure HDInsight will enforce TLS 1.2 or later versions for all HTTPS connections. For more information, see Azure HDInsight TLS 1.2 Enforcement.
-
tags
- (Optional) A map of Tags which should be assigned to this HDInsight Spark Cluster. -
metastores
- (Optional) Ametastores
block as defined below. -
monitor
- (Optional) Amonitor
block as defined below. -
extension
- (Optional) Anextension
block as defined below. -
securityProfile
- (Optional) AsecurityProfile
block as defined below. Changing this forces a new resource to be created.
A componentVersion
block supports the following:
spark
- (Required) The version of Spark which should be used for this HDInsight Spark Cluster. Changing this forces a new resource to be created.
A gateway
block supports the following:
password
- (Required) The password used for the Ambari Portal.
-> NOTE: This password must be different from the one used for the headNode
, workerNode
and zookeeperNode
roles.
username
- (Required) The username used for the Ambari Portal. Changing this forces a new resource to be created.
A headNode
block supports the following:
-
username
- (Required) The Username of the local administrator for the Head Nodes. Changing this forces a new resource to be created. -
vmSize
- (Required) The Size of the Virtual Machine which should be used as the Head Nodes. Possible values areextraSmall
,small
,medium
,large
,extraLarge
,a5
,a6
,a7
,a8
,a9
,a10
,a11
,standardA1V2
,standardA2V2
,standardA2MV2
,standardA3
,standardA4V2
,standardA4MV2
,standardA8V2
,standardA8MV2
,standardD1
,standardD2
,standardD3
,standardD4
,standardD11
,standardD12
,standardD13
,standardD14
,standardD1V2
,standardD2V2
,standardD3V2
,standardD4V2
,standardD5V2
,standardD11V2
,standardD12V2
,standardD13V2
,standardD14V2
,standardDs1V2
,standardDs2V2
,standardDs3V2
,standardDs4V2
,standardDs5V2
,standardDs11V2
,standardDs12V2
,standardDs13V2
,standardDs14V2
,standardE2V3
,standardE4V3
,standardE8V3
,standardE16V3
,standardE20V3
,standardE32V3
,standardE64V3
,standardE64IV3
,standardE2SV3
,standardE4SV3
,standardE8SV3
,standardE16SV3
,standardE20SV3
,standardE32SV3
,standardE64SV3
,standardE64IsV3
,standardD2AV4
,standardD4AV4
,standardD8AV4
,standardD16AV4
,standardD32AV4
,standardD48AV4
,standardD64AV4
,standardD96AV4
,standardE2AV4
,standardE4AV4
,standardE8AV4
,standardE16AV4
,standardE20AV4
,standardE32AV4
,standardE48AV4
,standardE64AV4
,standardE96AV4
,standardG1
,standardG2
,standardG3
,standardG4
,standardG5
,standardF2SV2
,standardF4SV2
,standardF8SV2
,standardF16SV2
,standardF32SV2
,standardF64SV2
,standardF72SV2
,standardGs1
,standardGs2
,standardGs3
,standardGs4
,standardGs5
andstandardNc24
. Changing this forces a new resource to be created. -
password
- (Optional) The Password associated with the local administrator for the Head Nodes. Changing this forces a new resource to be created.
-> NOTE: If specified, this password must be at least 10 characters in length and must contain at least one digit, one uppercase and one lower case letter, one non-alphanumeric character (except characters ' " ` ).
sshKeys
- (Optional) A list of SSH Keys which should be used for the local administrator on the Head Nodes. Changing this forces a new resource to be created.
-> NOTE: Either a password
or one or more sshKeys
must be specified - but not both.
-
subnetId
- (Optional) The ID of the Subnet within the Virtual Network where the Head Nodes should be provisioned within. Changing this forces a new resource to be created. -
virtualNetworkId
- (Optional) The ID of the Virtual Network where the Head Nodes should be provisioned within. Changing this forces a new resource to be created. -
scriptActions
- (Optional) The script action which will run on the cluster. Changing this forces a new resource to be created.
A scriptActions
block supports the following:
-
name
- (Required) The name of the script action. -
uri
- (Required) The URI to the script. -
parameters
- (Optional) The parameters for the script provided.
A roles
block supports the following:
-
headNode
- (Required) AheadNode
block as defined above. -
workerNode
- (Required) AworkerNode
block as defined below. -
zookeeperNode
- (Required) AzookeeperNode
block as defined below.
A network
block supports the following:
connectionDirection
- (Optional) The direction of the resource provider connection. Possible values includeinbound
oroutbound
. Defaults toinbound
. Changing this forces a new resource to be created.
-> NOTE: To enabled the private link the connectionDirection
must be set to outbound
.
privateLinkEnabled
- (Optional) Is the private link enabled? Possible values includetrue
orfalse
. Defaults tofalse
. Changing this forces a new resource to be created.
A computeIsolation
block supports the following:
-
computeIsolationEnabled
- (Optional) This field indicates whether enable compute isolation or not. Possible values aretrue
orfalse
. -
hostSku
- (Optional) The name of the host SKU.
A storageAccount
block supports the following:
isDefault
- (Required) Is this the Default Storage Account for the HDInsight Hadoop Cluster? Changing this forces a new resource to be created.
-> NOTE: One of the storageAccount
or storageAccountGen2
blocks must be marked as the default.
-
storageAccountKey
- (Required) The Access Key which should be used to connect to the Storage Account. Changing this forces a new resource to be created. -
storageContainerId
- (Required) The ID of the Storage Container. Changing this forces a new resource to be created.
-> NOTE: This can be obtained from the id
of the azurermStorageContainer
resource.
storageResourceId
- (Optional) The ID of the Storage Account. Changing this forces a new resource to be created.
A storageAccountGen2
block supports the following:
isDefault
- (Required) Is this the Default Storage Account for the HDInsight Hadoop Cluster? Changing this forces a new resource to be created.
-> NOTE: One of the storageAccount
or storageAccountGen2
blocks must be marked as the default.
-
storageResourceId
- (Required) The ID of the Storage Account. Changing this forces a new resource to be created. -
filesystemId
- (Required) The ID of the Gen2 Filesystem. Changing this forces a new resource to be created. -
managedIdentityResourceId
- (Required) The ID of Managed Identity to use for accessing the Gen2 filesystem. Changing this forces a new resource to be created.
-> NOTE: This can be obtained from the id
of the azurermStorageContainer
resource.
A workerNode
block supports the following:
-
scriptActions
- (Optional) The script action which will run on the cluster. Changing this forces a new resource to be created. -
username
- (Required) The Username of the local administrator for the Worker Nodes. Changing this forces a new resource to be created. -
vmSize
- (Required) The Size of the Virtual Machine which should be used as the Worker Nodes. Possible values areextraSmall
,small
,medium
,large
,extraLarge
,a5
,a6
,a7
,a8
,a9
,a10
,a11
,standardA1V2
,standardA2V2
,standardA2MV2
,standardA3
,standardA4V2
,standardA4MV2
,standardA8V2
,standardA8MV2
,standardD1
,standardD2
,standardD3
,standardD4
,standardD11
,standardD12
,standardD13
,standardD14
,standardD1V2
,standardD2V2
,standardD3V2
,standardD4V2
,standardD5V2
,standardD11V2
,standardD12V2
,standardD13V2
,standardD14V2
,standardDs1V2
,standardDs2V2
,standardDs3V2
,standardDs4V2
,standardDs5V2
,standardDs11V2
,standardDs12V2
,standardDs13V2
,standardDs14V2
,standardE2V3
,standardE4V3
,standardE8V3
,standardE16V3
,standardE20V3
,standardE32V3
,standardE64V3
,standardE64IV3
,standardE2SV3
,standardE4SV3
,standardE8SV3
,standardE16SV3
,standardE20SV3
,standardE32SV3
,standardE64SV3
,standardE64IsV3
,standardD2AV4
,standardD4AV4
,standardD8AV4
,standardD16AV4
,standardD32AV4
,standardD48AV4
,standardD64AV4
,standardD96AV4
,standardE2AV4
,standardE4AV4
,standardE8AV4
,standardE16AV4
,standardE20AV4
,standardE32AV4
,standardE48AV4
,standardE64AV4
,standardE96AV4
,standardG1
,standardG2
,standardG3
,standardG4
,standardG5
,standardF2SV2
,standardF4SV2
,standardF8SV2
,standardF16SV2
,standardF32SV2
,standardF64SV2
,standardF72SV2
,standardGs1
,standardGs2
,standardGs3
,standardGs4
,standardGs5
andstandardNc24
. Changing this forces a new resource to be created. -
password
- (Optional) The Password associated with the local administrator for the Worker Nodes. Changing this forces a new resource to be created.
-> NOTE: If specified, this password must be at least 10 characters in length and must contain at least one digit, one uppercase and one lower case letter, one non-alphanumeric character (except characters ' " ` ).
sshKeys
- (Optional) A list of SSH Keys which should be used for the local administrator on the Worker Nodes. Changing this forces a new resource to be created.
-> NOTE: Either a password
or one or more sshKeys
must be specified - but not both.
-
subnetId
- (Optional) The ID of the Subnet within the Virtual Network where the Worker Nodes should be provisioned within. Changing this forces a new resource to be created. -
targetInstanceCount
- (Required) The number of instances which should be run for the Worker Nodes. -
virtualNetworkId
- (Optional) The ID of the Virtual Network where the Worker Nodes should be provisioned within. Changing this forces a new resource to be created. -
autoscale
- (Optional) Aautoscale
block as defined below.
A zookeeperNode
block supports the following:
-
scriptActions
- (Optional) The script action which will run on the cluster. Changing this forces a new resource to be created. -
username
- (Required) The Username of the local administrator for the Zookeeper Nodes. Changing this forces a new resource to be created. -
vmSize
- (Required) The Size of the Virtual Machine which should be used as the Zookeeper Nodes. Possible values areextraSmall
,small
,medium
,large
,extraLarge
,a5
,a6
,a7
,a8
,a9
,a10
,a11
,standardA1V2
,standardA2V2
,standardA2MV2
,standardA3
,standardA4V2
,standardA4MV2
,standardA8V2
,standardA8MV2
,standardD1
,standardD2
,standardD3
,standardD4
,standardD11
,standardD12
,standardD13
,standardD14
,standardD1V2
,standardD2V2
,standardD3V2
,standardD4V2
,standardD5V2
,standardD11V2
,standardD12V2
,standardD13V2
,standardD14V2
,standardDs1V2
,standardDs2V2
,standardDs3V2
,standardDs4V2
,standardDs5V2
,standardDs11V2
,standardDs12V2
,standardDs13V2
,standardDs14V2
,standardE2V3
,standardE4V3
,standardE8V3
,standardE16V3
,standardE20V3
,standardE32V3
,standardE64V3
,standardE64IV3
,standardE2SV3
,standardE4SV3
,standardE8SV3
,standardE16SV3
,standardE20SV3
,standardE32SV3
,standardE64SV3
,standardE64IsV3
,standardD2AV4
,standardD4AV4
,standardD8AV4
,standardD16AV4
,standardD32AV4
,standardD48AV4
,standardD64AV4
,standardD96AV4
,standardE2AV4
,standardE4AV4
,standardE8AV4
,standardE16AV4
,standardE20AV4
,standardE32AV4
,standardE48AV4
,standardE64AV4
,standardE96AV4
,standardG1
,standardG2
,standardG3
,standardG4
,standardG5
,standardF2SV2
,standardF4SV2
,standardF8SV2
,standardF16SV2
,standardF32SV2
,standardF64SV2
,standardF72SV2
,standardGs1
,standardGs2
,standardGs3
,standardGs4
,standardGs5
andstandardNc24
. Changing this forces a new resource to be created. -
password
- (Optional) The Password associated with the local administrator for the Zookeeper Nodes. Changing this forces a new resource to be created.
-> NOTE: If specified, this password must be at least 10 characters in length and must contain at least one digit, one uppercase and one lower case letter, one non-alphanumeric character (except characters ' " ` ).
sshKeys
- (Optional) A list of SSH Keys which should be used for the local administrator on the Zookeeper Nodes. Changing this forces a new resource to be created.
-> NOTE: Either a password
or one or more sshKeys
must be specified - but not both.
-
subnetId
- (Optional) The ID of the Subnet within the Virtual Network where the Zookeeper Nodes should be provisioned within. Changing this forces a new resource to be created. -
virtualNetworkId
- (Optional) The ID of the Virtual Network where the Zookeeper Nodes should be provisioned within. Changing this forces a new resource to be created.
A diskEncryption
block supports the following:
-
encryptionAlgorithm
- (Optional) This is an algorithm identifier for encryption. Possible values arersa15
,rsaOaep
,rsaOaep256
. -
encryptionAtHostEnabled
- (Optional) This is indicator to show whether resource disk encryption is enabled. -
keyVaultKeyId
- (Optional) The ID of the key vault key. -
keyVaultManagedIdentityId
- (Optional) This is the resource ID of Managed Identity used to access the key vault.
A metastores
block supports the following:
-
hive
- (Optional) Ahive
block as defined below. -
oozie
- (Optional) Anoozie
block as defined below. -
ambari
- (Optional) Anambari
block as defined below.
A hive
block supports the following:
-
server
- (Required) The fully-qualified domain name (FQDN) of the SQL server to use for the external Hive metastore. Changing this forces a new resource to be created. -
databaseName
- (Required) The external Hive metastore's existing SQL database. Changing this forces a new resource to be created. -
username
- (Required) The external Hive metastore's existing SQL server admin username. Changing this forces a new resource to be created. -
password
- (Required) The external Hive metastore's existing SQL server admin password. Changing this forces a new resource to be created.
An oozie
block supports the following:
-
server
- (Required) The fully-qualified domain name (FQDN) of the SQL server to use for the external Oozie metastore. Changing this forces a new resource to be created. -
databaseName
- (Required) The external Oozie metastore's existing SQL database. Changing this forces a new resource to be created. -
username
- (Required) The external Oozie metastore's existing SQL server admin username. Changing this forces a new resource to be created. -
password
- (Required) The external Oozie metastore's existing SQL server admin password. Changing this forces a new resource to be created.
An ambari
block supports the following:
-
server
- (Required) The fully-qualified domain name (FQDN) of the SQL server to use for the external Ambari metastore. Changing this forces a new resource to be created. -
databaseName
- (Required) The external Hive metastore's existing SQL database. Changing this forces a new resource to be created. -
username
- (Required) The external Ambari metastore's existing SQL server admin username. Changing this forces a new resource to be created. -
password
- (Required) The external Ambari metastore's existing SQL server admin password. Changing this forces a new resource to be created.
A monitor
block supports the following:
-
logAnalyticsWorkspaceId
- (Required) The Operations Management Suite (OMS) workspace ID. -
primaryKey
- (Required) The Operations Management Suite (OMS) workspace key.
A extension
block supports the following:
-
logAnalyticsWorkspaceId
- (Required) The workspace ID of the log analytics extension. -
primaryKey
- (Required) The workspace key of the log analytics extension.
An autoscale
block supports the following:
-
capacity
- (Optional) Acapacity
block as defined below. -
recurrence
- (Optional) Arecurrence
block as defined below.
-> NOTE: Either a capacity
or recurrence
block must be specified - but not both.
A capacity
block supports the following:
-
maxInstanceCount
- (Required) The maximum number of worker nodes to autoscale to based on the cluster's activity. -
minInstanceCount
- (Required) The minimum number of worker nodes to autoscale to based on the cluster's activity.
A recurrence
block supports the following:
-
schedule
- (Required) A list ofschedule
blocks as defined below. -
timezone
- (Required) The time zone for the autoscale schedule times.
A schedule
block supports the following:
-
days
- (Required) The days of the week to perform autoscale. Possible values aremonday
,tuesday
,wednesday
,thursday
,friday
,saturday
andsunday
. -
targetInstanceCount
- (Required) The number of worker nodes to autoscale at the specified time. -
time
- (Required) The time of day to perform the autoscale in 24hour format.
A securityProfile
block supports the following:
-
aaddsResourceId
- (Required) The resource ID of the Azure Active Directory Domain Service. Changing this forces a new resource to be created. -
domainName
- (Required) The name of the Azure Active Directory Domain. Changing this forces a new resource to be created. -
domainUsername
- (Required) The username of the Azure Active Directory Domain. Changing this forces a new resource to be created. -
domainUserPassword
- (Required) The user password of the Azure Active Directory Domain. Changing this forces a new resource to be created. -
ldapsUrls
- (Required) A list of the LDAPS URLs to communicate with the Azure Active Directory. Changing this forces a new resource to be created. -
msiResourceId
- (Required) The User Assigned Identity for the HDInsight Cluster. Changing this forces a new resource to be created. -
clusterUsersGroupDns
- (Optional) A list of the distinguished names for the cluster user groups. Changing this forces a new resource to be created.
Attributes Reference
The following attributes are exported:
-
id
- The ID of the HDInsight Spark Cluster. -
httpsEndpoint
- The HTTPS Connectivity Endpoint for this HDInsight Spark Cluster. -
sshEndpoint
- The SSH Connectivity Endpoint for this HDInsight Spark Cluster.
Timeouts
The timeouts
block allows you to specify timeouts for certain actions:
create
- (Defaults to 60 minutes) Used when creating the Spark HDInsight Cluster.update
- (Defaults to 60 minutes) Used when updating the Spark HDInsight Cluster.read
- (Defaults to 5 minutes) Used when retrieving the Spark HDInsight Cluster.delete
- (Defaults to 60 minutes) Used when deleting the Spark HDInsight Cluster.
Import
HDInsight Spark Clusters can be imported using the resourceId
, e.g.