Skip to content

Resource: awsDmsS3Endpoint

Provides a DMS (Data Migration Service) S3 endpoint resource. DMS S3 endpoints can be created, updated, deleted, and imported.

\~> Note: AWS is deprecating extraConnectionAttributes, such as used with awsDmsEndpoint. This resource is an alternative to awsDmsEndpoint and does not use extraConnectionAttributes. (AWS currently includes extraConnectionAttributes in the raw responses to the AWS Provider requests and so they may be visible in Terraform logs.)

\~> Note: Some of this resource's arguments have default values that come from the AWS Provider. Other default values are provided by AWS and subject to change without notice. When relying on AWS defaults, the Terraform state will often have a zero value. For example, the AWS Provider does not provide a default for cdcMaxBatchInterval but the AWS default is 60 (seconds). However, the Terraform state will show 0 since this is the value return by AWS when no value is present. Below, we aim to flag the defaults that come from AWS (e.g., "AWS default...").

Example Usage

Minimal Configuration

This is the minimal configuration for an awsDmsS3Endpoint. This endpoint will rely on the AWS Provider and AWS defaults.

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as aws from "./.gen/providers/aws";
new aws.dmsS3Endpoint.DmsS3Endpoint(this, "example", {
  bucketName: "beckut_name",
  depends_on: ["${aws_iam_role_policy.example}"],
  endpointId: "donnedtipi",
  endpointType: "target",
  serviceAccessRoleArn: "${aws_iam_role.example.arn}",
});

Complete Configuration

/*Provider bindings are generated by running cdktf get.
See https://cdk.tf/provider-generation for more details.*/
import * as aws from "./.gen/providers/aws";
new aws.dmsS3Endpoint.DmsS3Endpoint(this, "example", {
  addColumnName: true,
  addTrailingPaddingCharacter: false,
  bucketFolder: "folder",
  bucketName: "bucket_name",
  cannedAclForObjects: "private",
  cdcInsertsAndUpdates: true,
  cdcInsertsOnly: false,
  cdcMaxBatchInterval: 100,
  cdcMinFileSize: 16,
  cdcPath: "cdc/path",
  compressionType: "GZIP",
  csvDelimiter: ";",
  csvNoSupValue: "x",
  csvNullValue: "?",
  csvRowDelimiter: "\\r\\n",
  dataFormat: "parquet",
  dataPageSize: 1100000,
  datePartitionDelimiter: "UNDERSCORE",
  datePartitionEnabled: true,
  datePartitionSequence: "yyyymmddhh",
  datePartitionTimezone: "Asia/Seoul",
  depends_on: ["${aws_iam_role_policy.example}"],
  dictPageSizeLimit: 1000000,
  enableStatistics: false,
  encodingType: "plain",
  encryptionMode: "SSE_S3",
  endpointId: "donnedtipi",
  endpointType: "target",
  expectedBucketOwner: "${data.aws_caller_identity.current.account_id}",
  externalTableDefinition: "etd",
  ignoreHeaderRows: 1,
  includeOpForFullLoad: true,
  maxFileSize: 1000000,
  parquetTimestampInMillisecond: true,
  parquetVersion: "parquet-2-0",
  preserveTransactions: false,
  rfc4180: false,
  rowGroupLength: 11000,
  serverSideEncryptionKmsKeyId: "${aws_kms_key.example.arn}",
  serviceAccessRoleArn: "${aws_iam_role.example.arn}",
  sslMode: "none",
  tags: {
    Name: "donnedtipi",
    Remove: "to-remove",
    Update: "to-update",
  },
  timestampColumnName: "tx_commit_time",
  useCsvNoSupValue: false,
  useTaskStartTimeForFullLoadTimestamp: true,
});

Argument Reference

The following arguments are required:

  • bucketName - (Required) S3 bucket name.
  • cdcPath - (Required for CDC; otherwise, Optional) Folder path of CDC files. If cdcPath is set, AWS DMS reads CDC files from this path and replicates the data changes to the target endpoint. Supported in AWS DMS versions 3.4.2 and later.
  • endpointId - (Required) Database endpoint identifier. Identifiers must contain from 1 to 255 alphanumeric characters or hyphens, begin with a letter, contain only ASCII letters, digits, and hyphens, not end with a hyphen, and not contain two consecutive hyphens.
  • endpointType - (Required) Type of endpoint. Valid values are source, target.
  • externalTableDefinition - (Required for source endpoints; otherwise, Optional) JSON document that describes how AWS DMS should interpret the data.
  • serviceAccessRoleArn - (Required) ARN of the IAM role with permissions to the S3 Bucket.

The following arguments are optional:

  • addColumnName - (Optional) Whether to add column name information to the .csv output file. Default is false.
  • addTrailingPaddingCharacter - (Optional) Whether to add padding. Default is false. (Ignored for source endpoints.)
  • bucketFolder - (Optional) S3 object prefix.
  • cannedAclForObjects - (Optional) Predefined (canned) access control list for objects created in an S3 bucket. Valid values include none, private, PUBLIC_READ, PUBLIC_READ_WRITE, AUTHENTICATED_READ, AWS_EXEC_READ, BUCKET_OWNER_READ, and BUCKET_OWNER_FULL_CONTROL. (AWS default is none.)
  • cdcInsertsAndUpdates - (Optional) Whether to write insert and update operations to .csv or .parquet output files. Default is false.
  • cdcInsertsOnly - (Optional) Whether to write insert operations to .csv or .parquet output files. Default is false.
  • cdcMaxBatchInterval - (Optional) Maximum length of the interval, defined in seconds, after which to output a file to Amazon S3. (AWS default is 60.)
  • cdcMinFileSize - (Optional) Minimum file size condition as defined in kilobytes to output a file to Amazon S3. (AWS default is 32000 KB.)
  • certificateArn - (Optional, Default: empty string) ARN for the certificate.
  • compressionType - (Optional) Set to compress target files. Valid values are gzip and none. Default is none. (Ignored for source endpoints.)
  • csvDelimiter - (Optional) Delimiter used to separate columns in the source files. Default is ,.
  • csvNoSupValue - (Optional) Only applies if output files for a CDC load are written in .csv format. If useCsvNoSupValue is set to true, string to use for all columns not included in the supplemental log. If you do not specify a string value, DMS uses the null value for these columns regardless of useCsvNoSupValue. (Ignored for source endpoints.)
  • csvNullValue - (Optional) String to as null when writing to the target. (AWS default is null.)
  • csvRowDelimiter - (Optional) Delimiter used to separate rows in the source files. Default is newline (i.e., \n).
  • dataFormat - (Optional) Output format for the files that AWS DMS uses to create S3 objects. Valid values are csv and parquet. (Ignored for source endpoints -- only csv is valid.)
  • dataPageSize - (Optional) Size of one data page in bytes. (AWS default is 1 MiB, i.e., 1048576.)
  • datePartitionDelimiter - (Optional) Date separating delimiter to use during folder partitioning. Valid values are slash, underscore, dash, and none. (AWS default is slash.) (Ignored for source endpoints.)
  • datePartitionEnabled - (Optional) Partition S3 bucket folders based on transaction commit dates. Default is false. (Ignored for source endpoints.)
  • datePartitionSequence - (Optional) Date format to use during folder partitioning. Use this parameter when datePartitionEnabled is set to true. Valid values are yyyymmdd, yyyymmddhh, yyyymm, mmyyyydd, and ddmmyyyy. (AWS default is yyyymmdd.) (Ignored for source endpoints.)
  • datePartitionTimezone - (Optional) Convert the current UTC time to a timezone. The conversion occurs when a date partition folder is created and a CDC filename is generated. The timezone format is Area/Location (e.g., europe/paris). Use this when datePartitionEnabled is true. (Ignored for source endpoints.)
  • detachTargetOnLobLookupFailureParquet - (Optional) Undocumented argument for use as directed by AWS Support.
  • dictPageSizeLimit - (Optional) Maximum size in bytes of an encoded dictionary page of a column. (AWS default is 1 MiB, i.e., 1048576.)
  • enableStatistics - (Optional) Whether to enable statistics for Parquet pages and row groups. Default is true.
  • encodingType - (Optional) Type of encoding to use. Value values are rleDictionary, plain, and plainDictionary. (AWS default is rleDictionary.)
  • encryptionMode - (Optional) Server-side encryption mode that you want to encrypt your .csv or .parquet object files copied to S3. Valid values are SSE_S3 and SSE_KMS. (AWS default is SSE_S3.) (Ignored for source endpoints -- only SSE_S3 is valid.)
  • expectedBucketOwner - (Optional) Bucket owner to prevent sniping. Value is an AWS account ID.
  • ignoreHeaderRows - (Optional, Force New) When this value is set to 1, DMS ignores the first row header in a .csv file. (AWS default is 0.)
  • includeOpForFullLoad - (Optional) Whether to enable a full load to write INSERT operations to the .csv output files only to indicate how the rows were added to the source database. Default is false.
  • kmsKeyArn - (Optional) ARN for the KMS key that will be used to encrypt the connection parameters. If you do not specify a value for kmsKeyArn, then AWS DMS will use your default encryption key. AWS KMS creates the default encryption key for your AWS account. Your AWS account has a different default encryption key for each AWS region.
  • maxFileSize - (Optional) Maximum size (in KB) of any .csv file to be created while migrating to an S3 target during full load. Valid values are from 1 to 1048576. (AWS default is 1 GB, i.e., 1048576.)
  • parquetTimestampInMillisecond - (Optional) - Specifies the precision of any TIMESTAMP column values written to an S3 object file in .parquet format. Default is false. (Ignored for source endpoints.)
  • parquetVersion - (Optional) Version of the .parquet file format. Valid values are parquet10 and parquet20. (AWS default is parquet10.) (Ignored for source endpoints.)
  • preserveTransactions - (Optional) Whether DMS saves the transaction order for a CDC load on the S3 target specified by cdcPath. Default is false. (Ignored for source endpoints.)
  • rfc4180 - (Optional) For an S3 source, whether each leading double quotation mark has to be followed by an ending double quotation mark. Default is true.
  • rowGroupLength - (Optional) Number of rows in a row group. (AWS default is 10000.)
  • serverSideEncryptionKmsKeyId - (Optional) When encryptionMode is SSE_KMS, ARN for the AWS KMS key. (Ignored for source endpoints -- only SSE_S3 encryptionMode is valid.)
  • sslMode - (Optional) SSL mode to use for the connection. Valid values are none, require, verifyCa, verifyFull. (AWS default is none.)
  • tags - (Optional) Map of tags to assign to the resource. If configured with a provider defaultTags configuration block present, tags with matching keys will overwrite those defined at the provider-level.
  • timestampColumnName - (Optional) Column to add with timestamp information to the endpoint data for an Amazon S3 target.
  • useCsvNoSupValue - (Optional) Whether to use csvNoSupValue for columns not included in the supplemental log. (Ignored for source endpoints.)
  • useTaskStartTimeForFullLoadTimestamp - (Optional) When set to true, uses the task start time as the timestamp column value instead of the time data is written to target. For full load, when set to true, each row of the timestamp column contains the task start time. For CDC loads, each row of the timestamp column contains the transaction commit time.When set to false, the full load timestamp in the timestamp column increments with the time data arrives at the target. Default is false.

Attributes Reference

In addition to all arguments above, the following attributes are exported:

  • endpointArn - ARN for the endpoint.
  • engineDisplayName - Expanded name for the engine name.
  • externalId - Can be used for cross-account validation. Use it in another account with awsDmsS3Endpoint to create the endpoint cross-account.
  • status - Status of the endpoint.
  • tagsAll - Map of tags assigned to the resource, including those inherited from the provider defaultTags configuration block.

Timeouts

Configuration options:

  • create - (Default 5M)
  • delete - (Default 5M)

Import

Endpoints can be imported using the endpointId, e.g.,

$ terraform import aws_dms_s3_endpoint.example example-dms-endpoint-tf