Claravine + AWS S3 Integration

Templates: Inbound Configuration

Administrators can configure a bucket in their organization’s AWS S3 account specific for Claravine file import, allowing users to drop a file in the bucket and automatically import the data to Claravine as a dataset. Claravine can be configured to recognize a File Prefix and import the file data that matches the Template Name (i.e. us/submissions/Email will only bring in the file data dropped into the US folder that starts with Email as a submission into the Email template).

The organization must make sure the User has the following permission in their Policy:

• Action:s3:PutObject to designate the bucket manually on the template
• Action:s3:GetObject to ensure we can grab files in the desired bucket
• Action:s3:ListBucket to select the bucket from a dropdown menu on the template.

For more information about how to assign the s3:ListBucket permission, refer to AWS S3 Actions documentation.

Below is a sample of the JSON policy allowing all files sent to a specific folder within the designated S3 bucket:

        {
        "Version": "2012-10-17",
        "Statement": [
        {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
        "s3:PutObject",
        "s3:ListBucket"
        ],
        "Resource": [
        "arn:aws:s3:::this-is-your-bucket/this-is-your-folder/*",
        "arn:aws:s3:::this-is-your-bucket/this-is-your-folder/",
        "arn:aws:s3:::this-is-your-bucket"
        ]
        }
        ]
        }

AWS S3 Inbound Configuration

To configure a template for automated, inbound dataset:

1. Configure the AWS S3 integration with Claravine in the connected Account menu if you have not already.
2. Open the correct Template in the Templates Sub-Navigation menu.
3. Click Step 3 - Connect.
   
   
   
4. Click the plus (+) sign to add a new Inbound integration.
   
   
   
5. Click S3 Inbound Integration.
   The S3 Inbound configuration section displays.
   
   
   If you see the Permission warning and you followed the policy steps above, switch the radio button to "Manual Bucket Entry".
6. Enter the Integration Name. If no value is entered, Claravine will default the Integration Name to the Template name.
7. Select the connected AWS Account.
8. Select the Region. This is the region where your bucket is stored.
9. Select either Manual Bucket Entry or List Buckets based on the S3 user permissions.
10. If Manual Bucket Entry is selected and you are mapping to a Folder in your S3 bucket, enter the S3 Bucket (required) and File Prefix (optional).
    1. Enter the S3 bucket name exactly how it appears in AWS.
    2. If applicable, enter the filename path with the trailing slash (e.g. ProductListingv1/), you can append Claravine dynamic values for the Template Name, Submitted By, and Submitted At elements.

    

11. If List Bucket Entry is selected, select the S3 Bucket (required) from the drop-down menu and enter the File Prefix (optional).
    1. If applicable, enter the filename path with the trailing slash (e.g. ProductListingv1/), you can append Claravine dynamic values for the Template Name, Submitted By, and Submitted At elements.

12. Click Save to save the template.

Things to Know

• Work closely with your Claravine Customer Success Manager to test a small file to confirm configuration is correct.
• Users must use the downloaded Template File from Claravine and update it with any changes to the template configuration. See the user instructions to Import File Via AWS S3 or SFTP Knowledge Base article.
• Imported files must be in CSV UTF-8 format.
• If the Administrator has designated a File Prefix for the file name, ensure the users know the correct file name requirements.
• All governance and restrictions applied to the template fields will apply to the data being imported.
• Currently Claravine cannot provide alerts of failed imports because the file is being imported from a third party. Ensure users check the Submissions page in Claravine to confirm their data has been received and processed. Any inaccuracies to the data, file, or format will result in a failed submission.

Templates: Outbound Configuration

Claravine can automatically generate a AWS S3 file with the data from every submission. Administrators can design the file, what data to include in the file, and where to send the data per template.

To add AWS S3 Outbound to a template:

1. If you have not already, configure the integration with Claravine and AWS S3 as a Connect Account.
2. Open the correct Template in the Templates topical menu.
3. Click Step 3 - Connect.
   
   
   
4. Click to add a new Outbound integration.
   
   
   
5. Click S3.
   The S3 Outbound configuration section displays.
   
   
   
6. Enter the Integration Name. If no value is entered, Claravine will default the Integration Name to the Template name.
7. Select the connected AWS Account.
   An Administrator must first connect the AWS S3 Account in the Accounts Sub-Navigation menu of Integrations.
8. Select the Region.
9. Enter or select the Bucket using Manual Bucket Entry or List Buckets.  If entering the bucket manually, use just the bucket name with no leading or trailing slash. 
10. Check the box to use Server Side Encryption if required. SSE-S3 is the only encryption option. Amazon will encrypt the file before saving it in your select Bucket. The file will then be decrypted when it is downloaded. SSE-KMS is not available at this time.
11. Define the Filename. It will default to {{template_name}}{{submitted_at). You can add text, remove text or click +Insert on the right to add additional dynamic values Submitted By and Submission Name, to the generated Filename. (Submitted at will return YYYY-MM-DD_HH-MM-SS in UTC).
    - To push the file into a specific folder within the Bucket, enter the file path with no leading slash.(e.g. inbox/us/claravine/{submitted_at})
    - The file in AWS will appear with the file path with underscore delimiters (e.g. inbox_us_claravine_2020-09-09_12:37:52).
12. Select the File Type. Options are CSV, XLSX, JSON.
13. Select the Delimiter. Options are comma (,), semicolon (;), tab, and space.
14. Check the box to Include Summary Header in the file output, or exclude it to being the file with the field headers. The Summary Header includes:
    1. Date
    2. User Name
    3. Targets
    4. Template

13. File configuration options are Custom or Default Summary.
    
    1. Default Summary matches the schema in Step 2 Define and Govern schema.
       
       
       If the visible box is unchecked in the schema, it will not be included in the file.
       
       
    2. Custom allows field-specific changes:
       
       
       
       1. Include or exclude the field from the file
       2. Change the Output Label/column header
       3. Manually drag the tile to change the order
       4. Transform menu: Update all Output Labels to:
          • Replace spaces with underscores
          • Replace underscores with spaces
          • Lowercase
          • Uppercase
       5. Meta Field menu: Add a new meta field to match file output requirements:
          • Template Name
          • Submitter Name
          • Submitter Email
          • Created Date
          • Updated Date
          • Empty

14. Click Save to save the template.

Custom Permissions for AWS Accounts

In the event you need to limit permissioning to S3, this article describes how to isolate AWS User permissions using Custom Policies within your AWS IAM.

Claravine requires two actions to read/write to your S3 Bucket:

• s3:PutObject

• s3:ListBucket

You can limit these actions and limit the Resources within a custom Policy in IAM. Follow the steps below to complete this process.

Step 1

Retrieve ARN and Region. For later steps, we need to note the ARN (Amazon Resource Name) of the buckets and/or folders we want to limit access to in Claravine and the region of our bucket.

Go to S3 -> Buckets to see the list of buckets available. Click a bucket (or folder), then select ‘Properties’.

Bucket:

Folder:

Take note of the region, we will need that in Step 4, and copy/paste the ARN in a text file - we will use those values for Step 2.

Step 2

Go to IAM -> Access Management -> Policies. Click ‘Create Policy’ in the upper right-hand corner.

Select the JSON tab.

Copy/Paste the script below into the JSON code block area. Under “Resource”, substitute each ARN with your ARN values collected at Step 1. If you have a folder ARN, make sure to write in a corresponding wildcard value as denoted with a * . In this example, the ARN values are:

JSON Script:

        {
        "Version": "2012-10-17",
        "Statement": [
        {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
        "s3:PutObject",
        "s3:ListBucket"
        ],
        "Resource": [
        "arn:aws:s3:::we.retail.claravine.demo",
        "arn:aws:s3:::we.retail.claravine.demo/demo-folder/",
        "arn:aws:s3:::we.retail.claravine.demo/demo-folder/*"
        ]
        }
        ]
        }
      

Click through the next optional steps, and make changes as required for your company. When you get to ‘Review policy’ give your policy a Name, Description, and click ‘Create policy’

After Policy is created, you can search for your Policy and make edits at any time in IAM -> Access Management -> Policies.

Step 3

Create User with newly created policy. Go to IAM -> Access Management -> Users and click ‘Add Users’.

Give your User a name and select Programmatic access

Click Next.

On Set permissions, select ‘Attach Existing policies directly’ and search for your newly created policy and select.

Edit the permissions boundary if needed. Click through the next option (Tags) and add if necessary for your company needs. Click Next and finally ‘Create user’

After creating a User, copy the Access Key and Secret Key to a secure location. You will need these values to connect in Claravine.

Step 4

Connect your AWS User to Claravine.

Go to Claravine and add Account (Settings -> Integrations -> Accounts) by selecting the blue + sign, give your Account a name for Claravine usage, give description (optional), select the AWS S3 tile, and paste in your Access and Secret Key values. Click Save.

Once your AWS user is saved in Claravine, go to your Template to configure the integration.

For the Integration, Settings → Template → Inbound/Outbound Integration, select your AWS Account, Region (noted from Step 1), you will see an error message because we have limited permissioning on this account, select ‘Manual Bucket Entry’ from the radio button option.

Under S3 Bucket, type your bucket name:

If you are using a folder, under Filename, the start of the Filename looks like this:

If use Manaul entry with file directories:

The screenshot below illustrates the final integration.