use-aws-privatelink.html 5.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128
  1. 1. In the [SQL Shell](https://console.materialize.com/), or your preferred SQL
  2. client connected to Materialize, use the [`CREATE CONNECTION`](/sql/create-connection/#aws-privatelink)
  3. command to create an AWS PrivateLink connection:
  4. ↕️ **In-region connections**
  5. To connect to an AWS PrivateLink endpoint service in the **same region** as your
  6. Materialize environment:
  7. ```mzsql
  8. CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
  9. SERVICE NAME 'com.amazonaws.vpce.<region_id>.vpce-svc-<endpoint_service_id>',
  10. AVAILABILITY ZONES ('use1-az1', 'use1-az2', 'use1-az4')
  11. );
  12. ```
  13. - Replace the `SERVICE NAME` value with the service name you noted [earlier](#b-optional-configure-network-security).
  14. - Replace the `AVAILABILITY ZONES` list with the IDs of the availability
  15. zones in your AWS account. For in-region connections the availability
  16. zones of the NLB and the consumer VPC **must match**.
  17. To find your availability zone IDs, select your database in the RDS
  18. Console and click the subnets under **Connectivity & security**. For each
  19. subnet, look for **Availability Zone ID** (e.g., `use1-az6`),
  20. not **Availability Zone** (e.g., `us-east-1d`).
  21. ↔️ **Cross-region connections**
  22. To connect to an AWS PrivateLink endpoint service in a **different region** to
  23. the one where your Materialize environment is deployed:
  24. ```mzsql
  25. CREATE CONNECTION privatelink_svc TO AWS PRIVATELINK (
  26. SERVICE NAME 'com.amazonaws.vpce.us-west-1.vpce-svc-<endpoint_service_id>',
  27. -- For now, the AVAILABILITY ZONES clause **is** required, but will be
  28. -- made optional in a future release.
  29. AVAILABILITY ZONES ()
  30. );
  31. ```
  32. - Replace the `SERVICE NAME` value with the service name you noted [earlier](#b-optional-configure-network-security).
  33. - The service name region refers to where the endpoint service was created.
  34. You **do not need** to specify `AVAILABILITY ZONES` manually — these will
  35. be optimally auto-assigned when none are provided.
  36. 1. Retrieve the AWS principal for the AWS PrivateLink connection you just
  37. created:
  38. ```mzsql
  39. SELECT principal
  40. FROM mz_aws_privatelink_connections plc
  41. JOIN mz_connections c ON plc.id = c.id
  42. WHERE c.name = 'privatelink_svc';
  43. ```
  44. <p></p>
  45. ```
  46. principal
  47. ---------------------------------------------------------------------------
  48. arn:aws:iam::664411391173:role/mz_20273b7c-2bbe-42b8-8c36-8cc179e9bbc3_u1
  49. ```
  50. 1. Update your VPC endpoint service to [accept connections from the AWS
  51. principal](https://docs.aws.amazon.com/vpc/latest/privatelink/add-endpoint-service-permissions.html).
  52. 1. If your AWS PrivateLink service is configured to require acceptance of
  53. connection requests, [manually approve the connection request from
  54. Materialize](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).
  55. **Note:** It can take some time for the connection request to show up. Do
  56. not move on to the next step until you've approved the connection.
  57. 1. Validate the AWS PrivateLink connection you created using the
  58. [`VALIDATE CONNECTION`](/sql/validate-connection) command:
  59. ```mzsql
  60. VALIDATE CONNECTION privatelink_svc;
  61. ```
  62. If no validation error is returned, move to the next step.
  63. 1. Use the [`CREATE SECRET`](/sql/create-secret/) command to securely store the
  64. password for the `materialize` MySQL user you created [earlier](#2-create-a-user-for-replication):
  65. ```mzsql
  66. CREATE SECRET mysqlpass AS '<PASSWORD>';
  67. ```
  68. For AWS IAM authentication, you must create a connection to AWS. See the [`CREATE CONNECTION`](/sql/create-connection/#aws) command for details.
  69. 1. Use the [`CREATE CONNECTION`](/sql/create-connection/) command to create
  70. another connection object, this time with database access and authentication
  71. details for Materialize to use:
  72. ```mzsql
  73. CREATE CONNECTION mysql_connection TO MYSQL (
  74. HOST <host>,
  75. PORT 3306,
  76. USER 'materialize',
  77. PASSWORD SECRET mysqlpass,
  78. SSL MODE REQUIRED,
  79. AWS PRIVATELINK privatelink_svc
  80. );
  81. ```
  82. - Replace `<host>` with your RDS endpoint. To find your RDS endpoint, select
  83. your database in the RDS Console, and look under **Connectivity &
  84. security**.
  85. - Replace `<database>` with the name of the database containing the tables
  86. you want to replicate to Materialize.
  87. 1. Use the [`CREATE SOURCE`](/sql/create-source/) command to connect Materialize
  88. to your RDS instance via AWS PrivateLink and start ingesting data:
  89. ```mzsql
  90. CREATE SOURCE mz_source
  91. FROM mysql CONNECTION mysql_connection
  92. FOR ALL TABLES;
  93. ```
  94. By default, the source will be created in the active cluster; to use a
  95. different cluster, use the `IN CLUSTER` clause. To ingest data from
  96. specific schemas or tables, use `FOR SCHEMAS(<schema1>,<schema2>)` or `FOR
  97. TABLES (<table1>, <table2>)` instead of `FOR ALL TABLES`.
  98. AWS IAM authentication is also available, see the [`CREATE CONNECTION`](/sql/create-connection/#mysql) command for details.